Unsupervised Hyperspectral Image Super-Resolution via Self-Supervised Modality Decoupling

Fusion-based hyperspectral image super-resolution aims to fuse low-resolution hyperspectral images (LR-HSIs) and high-resolution multispectral images (HR-MSIs) to reconstruct high spatial and high spectral resolution images. Current methods typically apply direct fusion from the two modalities without effective supervision, leading to an incomplete perception of deep modality-complementary information and a limited understanding of inter-modality correlations. To address these issues, we propose a simple yet effective solution for unsupervised HMIF, revealing that modality decoupling is key to improving fusion performance. Specifically, we propose an end-to-end self-supervised Modality-Decoupled Spatial-Spectral Fusion (MossFuse) framework that decouples shared and complementary information across modalities and aggregates a concise representation of both LR-HSIs and HR-MSIs to reduce modality redundancy. Also, we introduce the subspace clustering loss as a clear guide to decouple modality-shared features from modality-complementary ones. Systematic experiments over multiple datasets demonstrate that our simple and effective approach consistently outperforms the existing HMIF methods while requiring considerably fewer parameters with reduced inference time. The source source code is in \href{https://github.com/dusongcheng/MossFuse}{MossFuse}.

💡 Research Summary

The paper tackles the problem of hyperspectral image super‑resolution (HMIF), where a low‑resolution hyperspectral image (LR‑HSI) and a high‑resolution multispectral image (HR‑MSI) are fused to reconstruct a high‑resolution hyperspectral image (HR‑HSI). Existing approaches fall into three categories: linear matrix/tensor factorization, coupled multi‑branch deep networks, and repetitive integration schemes. All of them directly merge the two modalities, which leads to two major drawbacks: (1) redundant features because shared information is not explicitly identified, and (2) loss of modality‑specific complementary details (high‑frequency spatial cues from HR‑MSI and fine‑grained spectral signatures from LR‑HSI). Consequently, fusion quality suffers and computational cost remains high.

The authors propose a novel unsupervised framework called MossFuse (Modality‑Decoupled Spatial‑Spectral Fusion). The key insight is that, due to the commutative property of spatial degradation (D_s) and spectral degradation (D_λ), applying D_s to HR‑MSI and D_λ to LR‑HSI yields the same low‑resolution multispectral image y. This common y defines a modality‑shared latent subspace (F_S). Each modality also contains a modality‑complementary component: F_C^Y (spatial high‑frequency details lost in LR‑HSI) and F_C^x (spectral fine‑structure lost in HR‑MSI). The overall decomposition can be written as
Y → F_S ⊕ F_C^Y, x → F_S ⊕ F_C^x.

MossFuse explicitly learns this decomposition through a set of dedicated modules:

1. Modality‑specific encoders that map Y and x into a shared latent representation F_S and their respective complementary features.
2. Subspace clustering loss (L_sub) that forces the shared components from both modalities to cluster around the same centers while pushing the complementary components into distinct clusters. This loss provides a clear supervisory signal for disentanglement without any ground‑truth HR‑HSI.
3. Self‑supervised reconstruction branch that recombines F_S with each complementary part to reconstruct the original Y and x. Reconstruction losses (L_rec) include pixel‑wise L2 and spectral angle (SAM) terms, encouraging the network to respect the physical degradation models.
4. Degradation‑parameter estimator that learns the blur kernel and spectral response functions jointly with the main network, making the method robust to unknown sensor characteristics.
5. Modality aggregation module that fuses the refined subspace representations into the final HR‑HSI, using learned weighting to balance spatial sharpness and spectral fidelity.

The architecture is end‑to‑end trainable, requires no paired HR‑HSI ground truth, and uses far fewer parameters than typical dual‑branch deep nets.

Extensive experiments on five public datasets (CAVE, Harvard, Chikusei, Pavia, Indian Pines) demonstrate that MossFuse consistently outperforms state‑of‑the‑art methods across PSNR, SSIM, SAM, and ERGAS. Gains range from 1.2–2.5 dB in PSNR and 0.02–0.04 in SSIM, while SAM improvements are 0.5–1.2°. Parameter count is reduced by 30–50 % and inference time is cut roughly in half (≈0.45 s for 256×256 inputs on a modern GPU), highlighting its suitability for real‑time or embedded deployment.

The paper also discusses limitations. The subspace clustering loss requires a pre‑specified number of clusters K, which may need tuning for scenes with highly heterogeneous spectral content. Moreover, extremely complex terrain or abrupt spectral changes could challenge the clean separation of complementary features. The authors suggest future work on adaptive K selection, non‑parametric clustering, or graph‑based relational learning to mitigate these issues.

In summary, MossFuse introduces a clear “shared subspace → complementary disentanglement → reconstruction” pipeline that addresses the core shortcomings of existing HMIF methods. By decoupling modalities, it eliminates redundancy, preserves modality‑specific details, and achieves superior reconstruction quality with a lightweight, fast model, opening the door for practical applications in remote sensing, environmental monitoring, and precision agriculture.