Rate-Distortion Analysis of Optically Passive Vision Compression

The use of remote vision sensors for autonomous decision-making poses the challenge of transmitting high-volume visual data over resource-constrained channels in real-time. In robotics and control applications, many systems can quickly destabilize, which can exacerbate the issue by necessitating higher sampling frequencies. This work proposes a novel sensing paradigm in which an event camera observes the optically generated cosine transform of a visual scene, enabling high-speed, computation-free video compression inspired by modern video codecs. In this study, we simulate this optically passive vision compression (OPVC) scheme and compare its rate-distortion performance to that of a standalone event camera (SAEC). We find that the rate-distortion performance of the OPVC scheme surpasses that of the SAEC and that this performance gap increases as the spatial resolution of the event camera increases.

💡 Research Summary

The paper addresses the challenge of transmitting high‑volume visual data from remote vision sensors in real‑time over bandwidth‑limited channels, a problem that becomes acute in high‑speed robotic control where sampling rates must be very high. Traditional video codecs (e.g., H.264/H.265) achieve strong compression by performing computationally intensive operations such as motion compensation and discrete cosine transforms (DCT). More recent deep‑learning based auto‑encoders also require substantial processing time. Event cameras, by contrast, output asynchronous address‑event‑representation (AER) packets only when the logarithmic intensity at a pixel changes beyond a threshold, offering microsecond‑scale temporal resolution but limited compression efficiency when used alone.

The authors propose a novel sensing paradigm called Optically Passive Vision Compression (OPVC). The core idea is to perform a cosine transform of the visual scene optically, using a 4‑f optical system that implements a Fourier transform of a real, even‑symmetrized image. Because the Fourier transform of a real even function reduces to a cosine transform, the optical system can compute the transform without digital processing. A strong constant plane wave is superimposed on the Fourier plane to make the resulting field strictly positive, allowing the intensity‑only sensor (the event camera) to recover the transform magnitude. The transformed intensity field is then recorded by an event camera, which generates AER packets whenever the log‑intensity changes by a preset threshold δ.

Mathematically, the event generation follows equations (4)–(6): each pixel fires at the earliest time tₖ where |log I(t) − log I(tₖ₋₁)| ≥ δ, emitting a tuple (pixel position, polarity, timestamp). The decoder reconstructs a piecewise‑constant log‑intensity by integrating the signed events (equation 6). The OPVC pipeline adds a constant offset to the cosine‑transformed frames before feeding them to the event camera; the decoder subtracts the same offset after event integration and finally applies an inverse DCT to obtain the reconstructed image.

To evaluate performance, the authors adopt a rate‑distortion framework where “rate” is defined as the average number of events per pixel per frame (sampling rate) rather than bits per second, because each AER tuple occupies a fixed number of bits. Distortion is measured using Multiscale Structural Similarity (MS‑SSIM). The rate‑distortion function R(D; π) is formalized in equation (7). A simple, vectorized event‑camera simulator is built: it converts grayscale video frames to log‑intensity, computes the number of events between successive frames using a rounding operation (equation 8), and accumulates the total event count K (equation 9). For OPVC, a 2‑D DCT is applied to each frame (equation 10) before the event simulation, and an inverse DCT is applied after reconstruction.

Experiments use the Ultra Video Group (UVG) dataset, which contains 16 uncompressed 4K videos (3840 × 2160) with up to 600 frames. The authors downsample the videos to several lower resolutions (e.g., 960 × 540, 480 × 270) and sweep the event threshold δ over a wide range. For each configuration they compute the average sampling rate R_sim and average MS‑SSIM D_sim (equations 11–12) and plot rate‑distortion curves.

Results show that at every spatial resolution, OPVC achieves a lower sampling rate than a Standalone Event Camera (SAEC) for the same distortion level. Moreover, the performance gap widens as the sensor resolution increases. The authors attribute this to the fact that the optical cosine transform concentrates most image energy in low‑frequency components, allowing the event camera to ignore high‑frequency details that would otherwise generate many events. Qualitative visual comparisons reveal that SAEC preserves sharp edges but suffers from motion‑related artifacts, whereas OPVC eliminates such artifacts but introduces slight high‑frequency blur and noise due to the low‑pass nature of the cosine transform.

The paper acknowledges several limitations. The simulation assumes an ideal, noise‑free event camera; real devices exhibit hot pixels, leakage events, and temporal jitter. The need for an initial reference frame (to know the absolute log‑intensity) implies additional low‑rate frame transmission, which was not accounted for in the rate calculation. Future work is suggested to incorporate realistic sensor noise models, more sophisticated frame‑to‑event conversion algorithms, and a full accounting of the overhead introduced by periodic reference frames.

In conclusion, the authors demonstrate that by offloading the cosine transform to passive optics and leveraging the asynchronous nature of event cameras, OPVC can provide high‑speed, computation‑light visual compression that outperforms a conventional event‑camera‑only approach in a rate‑distortion sense, especially at higher spatial resolutions. This approach opens a promising avenue for ultra‑low‑latency vision in bandwidth‑constrained autonomous systems such as drones, fast‑maneuvering robots, and remote inspection platforms.