Transfer Learning for Neural Networks-based Equalizers in Coherent Optical Systems

Pedro Jorge Freire de Carvalho Souza*, Daniel Abode, Jaroslaw E. Prilepsky, Nelson Costa, Bernhard Spinnler, Antonio Napoli, Sergei K. Turitsyn

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review


In this work, we address the question of the adaptability of artificial neural networks (NNs) used for impairments mitigation in optical transmission systems. We demonstrate that by using well-developed techniques based on the concept of transfer learning, we can efficaciously retrain NN-based equalizers to adapt to the changes in the transmission system, using just a fraction (down to 1%) of the initial training data or epochs. We evaluate the capability of transfer learning to adapt the NN to changes in the launch power, modulation format, symbol rate, or even fiber plants (different fiber types and lengths). The numerical examples utilize the recently introduced NN equalizer consisting of a convolutional layer coupled with bi-directional long-short term memory (biLSTM) recurrent NN element. Our analysis focuses on long-haul coherent optical transmission systems for two types of fibers: the standard single-mode fiber (SSMF) and the TrueWave Classic (TWC) fiber. We underline the specific peculiarities that occur when transferring the learning in coherent optical communication systems and draw the limits for the transfer learning efficiency. Our results demonstrate the effectiveness of transfer learning for the fast adaptation of NN architectures to different transmission regimes and scenarios, paving the way for engineering flexible and universal solutions for nonlinearity mitigation.

Original languageEnglish
Pages (from-to)6733-6745
Number of pages13
JournalJournal of Lightwave Technology
Issue number21
Early online date26 Aug 2021
Publication statusPublished - 1 Nov 2021

Bibliographical note

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see

Funding: This paper was supported by the EU Horizon 2020 program under the
Marie Sklodowska-Curie grant agreement 813144 (REAL-NET). YO acknowledges the support of the SMARTNET EMJMD programme (Project number - 586686-EPP-1-2017-1-UK-EPPKA1-JMD-MOB). JEP is supported by Leverhulme Trust, Grant No. RP-2018-063. SKT acknowledges support of the EPSRC project TRANSNET.


  • Neural network
  • coherent detection
  • flexible operation
  • nonlinear equalizer
  • transfer learning


Dive into the research topics of 'Transfer Learning for Neural Networks-based Equalizers in Coherent Optical Systems'. Together they form a unique fingerprint.

Cite this