In this paper, a high-security probabilistic constellation shaping transmission scheme based on recurrent neural networks (RNNs) is proposed, in which the constellation point probabilistic distribution is generated based on recurrent neural network training. A 4D biplane fractional-order chaotic system is introduced to ensure the security performance of the system. The performance of the proposed scheme is verified in a 2 km seven-core optical transmission system. The RNN-trained probabilistic shaping scheme achieves a transmission gain of 1.23 dB compared to the standard 16QAM signal, 0.39 dB compared to the standard Maxwell-Boltzmann (M-B) distribution signal, and a higher net bit rate. The proposed encryption scheme has higher randomness and security than the conventional integer-order chaotic system, with a key space of 10,163. This scheme will have a promising future fiber optic transmission scheme because it combines the efficient transmission and security of fiber optic transmission systems.
