Image Compression Using Novel View Synthesis Priors

Real-time visual feedback is essential for tetherless control of remotely operated vehicles, particularly during inspection and manipulation tasks. Though acoustic communication is the preferred choice for medium-range communication underwater, its limited bandwidth renders it impractical to transmit images or videos in real-time. To address this, we propose a model-based image compression technique that leverages prior mission information. Our approach employs trained machine-learning based novel view synthesis models, and uses gradient descent optimization to refine latent representations to help generate compressible differences between camera images and rendered images. We evaluate the proposed compression technique using a dataset from an artificial ocean basin, demonstrating superior compression ratios and image quality over existing techniques. Moreover, our method exhibits robustness to introduction of new objects within the scene, highlighting its potential for advancing tetherless remotely operated vehicle operations.

💡 Research Summary

The paper addresses the critical bottleneck of transmitting visual feedback from tetherless underwater remotely operated vehicles (ROVs) over low‑bandwidth acoustic links. Classical codecs such as WebP or JPEG‑XL cannot meet the stringent bitrate constraints, and modern learning‑based compressors require large, diverse training sets that are rarely available in underwater inspection scenarios. The authors propose a novel prior‑based compression framework called NVSPrior, which leverages a scene‑specific Novel View Synthesis (NVS) model as a powerful prior.

Core Idea
During an initial mapping mission, the ROV collects a sparse set of images and associated camera poses from a given inspection site. These data are used to train an NVS model—specifically a 3‑D Gaussian Splatting (3DGS) network—that can render photorealistic views from arbitrary poses. The trained model is stored both on the ROV and on the surface operator’s computer, guaranteeing identical rendering capabilities.

Compression Pipeline

1. Latent Optimization (iNVS) – For each new frame, the ROV seeks a compact latent representation (camera pose plus optional transient embeddings) that makes the NVS‑rendered image as close as possible to the captured image. This is achieved by a gradient‑descent process called inverse NVS (iNVS). The loss combines pixel‑wise L2, SSIM, and feature‑matching terms, and is minimized with the Adam optimizer. Initialization is crucial; the system re‑uses the optimized latent from the previous frame when inter‑frame motion is small, dramatically reducing the number of iterations (typically 5‑7) and keeping per‑frame latency under 50 ms.

2. Residual Image Generation – After optimization, the rendered image is subtracted from the actual camera image, producing a residual (I_diff). Because the NVS prior already reconstructs most of the static scene, I_diff is highly sparse and low‑entropy. The residual is then compressed with a conventional codec (WebP or JPEG‑XL).

3. Transmission & Reconstruction – The ROV transmits the optimized latent vector together with the compressed residual. The surface side uses the identical NVS model to render the view from the received latent, decompresses the residual, and adds it back to obtain the final reconstructed image.

Technical Contributions

• Introduction of NVSPrior, the first image compression system that explicitly exploits NVS priors.
• Development of iNVS, a fast gradient‑based latent refinement method tailored for compression rather than pose estimation.
• Extensive ablation studies on loss composition, optimizer choice, and initialization strategies, providing practical guidelines for real‑time deployment.
• Demonstration of robustness to novel objects, occlusions, and challenging underwater degradations (turbidity, backscatter) on both a controlled artificial ocean basin and a real coral‑reef dataset.

Experimental Results
Compared against classical codecs (WebP, JPEG‑XL) and state‑of‑the‑art learned compressors (Mean‑Scale Hyperprior, MLIC++), NVSPrior achieves 2–3× higher compression ratios while delivering 3–5 dB PSNR and 0.02–0.04 SSIM improvements at the same bitrate. The residual image entropy drops by roughly 15 % when using the composite loss versus a simple L2 loss. Even when new objects are introduced, the increase in residual size remains modest, confirming the method’s adaptability.

Limitations & Future Work
The approach assumes the NVS model is already available; updating the prior when the environment changes significantly requires offline re‑training, which can be time‑consuming. Channel impairments (packet loss, bit errors) are not addressed within the compression scheme and are left to existing acoustic modem protocols. Future directions include online prior updating, multi‑ROV collaborative priors, and dedicated entropy models for the residual stream.

Impact
By dramatically reducing the amount of data that must traverse acoustic links while preserving high‑fidelity visual information, NVSPrior makes real‑time tele‑operation, structural inspection, and environmental monitoring feasible for tetherless underwater platforms. The framework opens a new research avenue where scene‑specific generative models serve as compressive priors, potentially extending to other bandwidth‑constrained domains such as aerial drones or space robotics.