Coupled Transceiver-Fiber Nonlinearity Compensation Based on Machine Learning for Probabilistic Shaping System

Thanh Tu Nguyen, Tingting Zhang, Elias Giacoumidis, Abdallah Ali, Mingming Tan, Paul Harper, Liam P. Barry, Andrew D. Ellis

Research output: Contribution to journalArticlepeer-review


In this article, we experimentally demonstrate the combined benefit of artificial neural network-based nonlinearity compensation and probabilistic shaping for the first time. We demonstrate that the scheme not only compensates for transceiver's nonlinearity, enabling the full benefits of shaping to be achieved, but also the combined effects of transceiver and fiber propagation nonlinearities. The performance of the proposed artificial neural network is demonstrated at 28 Gbaud for both 64-QAM and 256-QAM probabilistically shaped systems and compared to that of uniformly distributed constellations. Our experimental results demonstrate: the expected performance gains for shaping alone; an additional SNR performance gain up to 1 dB in the linear region; an additional mutual information gain of 0.2 bits per channel use in the constellation-entropy limited region. In the presence of coupled transceiver and fiber-induced nonlinearities, an additional mutual information enhancement of sim0.13 bits/symbol is experimentally observed for a fiber link of up to 500 km with the aid of the proposed artificial neural network.

Original languageEnglish
Article number9216471
Pages (from-to)388-399
Number of pages12
JournalJournal of Lightwave Technology
Issue number2
Early online date7 Oct 2020
Publication statusPublished - 15 Jan 2021

Bibliographical note

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see

Funding: This work was supported by the UK EPSRC - Grants EP/S003436/1
(PHOS), EP/S016171/1 (EEMC) and EP/R035342/1 (TRANSNET).


  • ANN
  • fiber nonlinearity
  • machine learning
  • nonlinear equalizer
  • probabilistic shaping
  • transceiver nonlinearity


Dive into the research topics of 'Coupled Transceiver-Fiber Nonlinearity Compensation Based on Machine Learning for Probabilistic Shaping System'. Together they form a unique fingerprint.

Cite this