ML-Enabled Deformable Matched Filters for Bandlimitation Compensation in Free-Space Optics

This paper proposes a neural-network-assisted deformable matched filtering framework for carrier-less amplitude and phase (CAP) modulation operating under bandwidth-limited channel conditions. Instead of replacing the analytically derived CAP matched filter, the proposed receiver learns a residual deformation of the nominal matched filter based on a compact set of physically motivated signal features extracted from the received waveform. A total of 16 time-domain, frequency-domain, and memory-related features are used to provide a low-dimensional representation of bandwidth-induced pulse distortion. These features are mapped by a fully connected neural network to complex-valued matched filter coefficients, enabling adaptive pulse-shape compensation prior to symbol-rate sampling. The network is trained end-to-end using a differentiable loss function based on error vector magnitude (EVM). Experimental results obtained using a hardware-in-the-loop CAP transmission system demonstrate that the proposed deformable matched filter significantly outperforms conventional fixed matched filtering under severe bandwidth constraints, without requiring decision feedback or increasing receiver latency.

💡 Research Summary

**
This paper addresses the problem of pulse‑shape distortion in carrier‑less amplitude and phase (CAP) modulation when the optical link is subject to severe bandwidth limitations, a situation common in free‑space optical (FSO) communications. Conventional CAP receivers employ a fixed matched filter derived under ideal, unlimited‑bandwidth assumptions. In practice, the combined effects of limited channel bandwidth, atmospheric turbulence, and hardware non‑linearities cause the received waveform to deviate from the ideal shape, leading to a mismatch between the nominal filter and the optimal receiver and consequently degrading error‑vector magnitude (EVM).

The authors propose a hybrid “deformable matched filter” framework that retains the analytically derived matched filter as a strong prior while allowing a lightweight neural network to learn a residual deformation that compensates for the bandwidth‑induced distortion. The key innovation lies in the use of a compact, physics‑informed feature vector rather than raw samples. Sixteen features are extracted from the received signal after segmentation into four temporal blocks and averaging. These features fall into three groups: (i) time‑domain statistics (RMS, variance, skewness, excess kurtosis, mean envelope), (ii) frequency‑domain descriptors (spectral spread, roll‑off frequency, spectral flatness, spectral entropy, 3 dB bandwidth, spectral centroid), and (iii) signal‑quality metrics (peak‑to‑average power ratio, peak power, autocorrelation at lag = 1 and at one symbol period, envelope crest factor). The feature set captures amplitude‑level non‑linearities, high‑frequency attenuation, and temporal correlation changes caused by bandwidth compression.

A two‑layer fully‑connected neural network maps the 16‑dimensional feature vector to complex‑valued correction coefficients Δh