Performance investigation of geometric constellation shaping-based coherent WDM optical fiber communication system supported by deep-learning autoencoder

Abstract

Geometric constellation shaping (GCS) has been proposed to enhance the performance of wavelength-division multiplexing (WDM) coherent optical fiber communication (OFC) systems. This paper presents a comprehensive simulation platform to design and simulate these systems' performance using an advanced autoencoder (AE) technique. The design of the AE is based on end-to-end deep learning and takes into account the system parameters, especially those related to the OFC channel. The system design uses identical AEs, one in each WDM channel, and only the AE of the central channel is trained to achieve the required performance target. The developed AE simulation platform is capable of assessing the bit error rate characteristics, signal-to-noise ratio, and mutual information for long-haul coherent OFC-WDM systems operating with various modulation formats. The simulation results indicate that using a multilayer perceptron neural network to design AE requires four hidden layers, sixteen nodes per hidden layer, and a “16x modulation order” batch size to achieve optimum system performance. The AE's behaviour is investigated to identify the allowed ranges of the optimal launch power using two different procedures that achieve better system performance for 2D and 4D constellations.

Keywords

• End-to-end deep learning
• Coherent optical communication
• Autoencoder-based deep learning