Convolutional long short-term memory neural network equalizer for nonlinear Fourier transform-based optical transmission systems

Oleksandr Kotlyar, Morteza Kamalian-kopae, Maryna Pankratova, Anastasiia Vasylchenkova, Jaroslaw E. Prilepsky, Sergei K. Turitsyn

Research output: Contribution to journalArticlepeer-review


We evaluate improvement in the performance of the optical transmission systems operating with the continuous nonlinear Fourier spectrum by the artificial neural network equalisers installed at the receiver end. We propose here a novel equaliser designs based on bidirectional long short-term memory (BLSTM) gated recurrent neural network and compare their performance with the equaliser based on several fully connected layers. The proposed approach accounts for the correlations between different nonlinear spectral components. The application of BLSTM equaliser leads to a 16x improvement in terms of bit-error rate (BER) compared to the non-equalised case. The proposed equaliser makes it possible to reach the data rate of 170 Gbit/s for one polarisation conventional nonlinear Fourier transform (NFT) based system at 1000 km distance. We show that our new BLSTM equalisers significantly outperform the previously proposed scheme based on a feed-forward fully connected neural network. Moreover, we demonstrate that by adding a 1D convolutional layer for the data pre-processing before BLSTM recurrent layers, we can further enhance the performance of the BLSTM equaliser, reaching 23x BER improvement for the 170 Gbit/s system over 1000 km, staying below the 7% forward error correction hard decision threshold (HD-FEC).
Original languageEnglish
Pages (from-to)11254-11267
JournalOptics Express
Issue number7
Publication statusPublished - 26 Mar 2021

Bibliographical note

Published by The Optical Society under the terms of the Creative Commons Attribution 4.0 License. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.

Funding: Leverhulme Trust (ECF-2020-150, RP-2018-063); Engineering and Physical Sciences Research Council
(EP/R035342/1); H2020 Marie Skłodowska-Curie Actions (713694


Dive into the research topics of 'Convolutional long short-term memory neural network equalizer for nonlinear Fourier transform-based optical transmission systems'. Together they form a unique fingerprint.

Cite this