FreshMem: Brain-Inspired Frequency-Space Hybrid Memory for Streaming Video Understanding

Transitioning Multimodal Large Language Models (MLLMs) from offline to online streaming video understanding is essential for continuous perception. However, existing methods lack flexible adaptivity, leading to irreversible detail loss and context fragmentation. To resolve this, we propose FreshMem, a Frequency-Space Hybrid Memory network inspired by the brain’s logarithmic perception and memory consolidation. FreshMem reconciles short-term fidelity with long-term coherence through two synergistic modules: Multi-scale Frequency Memory (MFM), which projects overflowing frames into representative frequency coefficients, complemented by residual details to reconstruct a global historical “gist”; and Space Thumbnail Memory (STM), which discretizes the continuous stream into episodic clusters by employing an adaptive compression strategy to distill them into high-density space thumbnails. Extensive experiments show that FreshMem significantly boosts the Qwen2-VL baseline, yielding gains of 5.20%, 4.52%, and 2.34% on StreamingBench, OV-Bench, and OVO-Bench, respectively. As a training-free solution, FreshMem outperforms several fully fine-tuned methods, offering a highly efficient paradigm for long-horizon streaming video understanding.

💡 Research Summary

FreshMem tackles the emerging challenge of enabling multimodal large language models (MLLMs) to understand video streams in real time, moving beyond the traditional offline paradigm where the entire video is available beforehand. Existing approaches—either input‑level modulation (frame merging, sparsification) or memory‑augmented architectures (KV‑cache eviction, token condensation, external memory banks)—reduce computational load but inevitably discard fine‑grained details or fail to preserve long‑range coherence, especially under strict causal constraints.

Inspired by two neuroscientific principles—logarithmic time perception (Weber‑Fechner law) and memory consolidation via Sharp‑Wave Ripples (SWRs)—the authors propose a brain‑mirrored hybrid memory system called FreshMem. The system comprises two synergistic modules:

1. Multi‑scale Frequency Memory (MFM) projects overflow frames into the frequency domain using a discrete Fourier transform (DFT). To make DFT feasible for streaming, an incremental update rule is introduced:
   (C_t