Optimization of Probabilistic Shaping for Nonlinear Fiber Channels with Non-Gaussian Noise

Probabilistic constellation shaping is investigated in the context of nonlinear fiber optic communication channels. Based on a general framework, different link types are considered—1. dispersion-managed channels, 2. unrepeatered transmission channels and 3. ideal distributed Raman amplified channels. These channels exhibit nonlinear effects to a degree that conventional probabilistic constellation shaping strategies for the additive white Gaussian (AWGN) noise channel are suboptimal. A channel-agnostic optimization strategy is used to optimize the constellation probability mass functions (PMFs) for the channels in use. Optimized PMFs are obtained, which balance the effects of additive amplified spontaneous emission noise and nonlinear interference. The obtained PMFs cannot be modeled by the conventional Maxwell-Boltzmann PMFs and outperform optimal choices of these in all the investigated channels. Suboptimal choices of constellation shapes are associated with increased nonlinear effects in the form of non-Gaussian noise. For dispersion-managed channels, a reach gain in 2 spans is seen and across the three channel types, gains of >0.1 bits/symbol over unshaped quadrature-amplitude modulation (QAM) are seen using channel-optimized probablistic shaping.