Implementing Neural Network-Based Equalizers in a Coherent Optical Transmission System Using Field-Programmable Gate Arrays

Pedro J. Freire, Sasipim Srivallapanondh, Michael Anderson, Bernhard Spinnler, Thomas Bex, Tobias A. Eriksson, Antonio Napoli, Wolfgang Schairer, Nelson Costa, Michaela Blott, Sergei K. Turitsyn, Jaroslaw E. Prilepsky

Research output: Contribution to journalArticlepeer-review


In this work, we demonstrate the offline FPGA realization of both recurrent and feedforward neural network (NN)-based equalizers for nonlinearity compensation in coherent optical transmission systems. First, we present a realization pipeline showing the conversion of the models from Python libraries to the FPGA chip synthesis and implementation. Then, we review the main alternatives for the hardware implementation of nonlinear activation functions. The main results are divided into three parts: a performance comparison, an analysis of how activation functions are implemented, and a report on the complexity of the hardware. The performance in Q-factor is presented for the cases of bidirectional long-short-term memory coupled with convolutional NN (biLSTM + CNN) equalizer, CNN equalizer, and standard 1-StpS digital back-propagation (DBP) for the simulation and experiment propagation of a single channel dual-polarization (SC-DP) 16QAM at 34 GBd along 17×70 km of LEAF. The biLSTM+CNN equalizer provides a similar result to DBP and a 1.7 dB Q-factor gain compared with the chromatic dispersion compensation baseline in the experimental dataset. After that, we assess the Q-factor and the impact of hardware utilization when approximating the activation functions of NN using Taylor series, piecewise linear, and look-up table (LUT) approximations. We also show how to mitigate the approximation errors with extra training and provide some insights into possible gradient problems in the LUT approximation. Finally, to evaluate the complexity of hardware implementation to achieve 200 G and 400 G throughput, fixed-point NN-based equalizers with approximated activation functions are developed and implemented in an FPGA.
Original languageEnglish
Pages (from-to)3797-3815
JournalJournal of Lightwave Technology
Issue number12
Early online date1 May 2023
Publication statusPublished - 15 Jun 2023

Bibliographical note

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see


  • Artificial intelligence
  • Artificial neural networks
  • Equalizers
  • FPGA
  • Field programmable gate arrays
  • Hardware
  • Nonlinear optics
  • Software
  • Table lookup
  • coherent detection
  • computational complexity
  • neural network hardware
  • nonlinear equalizer
  • recurrent neural networks


Dive into the research topics of 'Implementing Neural Network-Based Equalizers in a Coherent Optical Transmission System Using Field-Programmable Gate Arrays'. Together they form a unique fingerprint.

Cite this