Digital nonlinearity compensation in high-capacity optical communication systems considering signal spectral broadening effect

Nyquist-spaced transmission and digital signal processing have proved effective in maximising the spectral efficiency and reach of optical communication systems. In these systems, Kerr nonlinearity determines the performance limits, and leads to spectral broadening of the signals propagating in the fibre. Although digital nonlinearity compensation was validated to be promising for mitigating Kerr nonlinearities, the impact of spectral broadening on nonlinearity compensation has never been quantified. In this paper, the performance of multi-channel digital back-propagation (MC-DBP) for compensating fibre nonlinearities in Nyquist-spaced optical communication systems is investigated, when the effect of signal spectral broadening is considered. It is found that accounting for the spectral broadening effect is crucial for achieving the best performance of DBP in both single-channel and multi-channel communication systems, independent of modulation formats used. For multi-channel systems, the degradation of DBP performance due to neglecting the spectral broadening effect in the compensation is more significant for outer channels. Our work also quantified the minimum bandwidths of optical receivers and signal processing devices to ensure the optimal compensation of deterministic nonlinear distortions.