Compression Tells Intelligence: Visual Coding, Visual Token Technology, and the Unification

“Compression Tells Intelligence”, is supported by research in artificial intelligence, particularly concerning (multimodal) large language models (LLMs/MLLMs), where compression efficiency often correlates with improved model performance and capabilities. For compression, classical visual coding based on traditional information theory has developed over decades, achieving great success with numerous international industrial standards widely applied in multimedia (e.g., image/video) systems. Except that, the recent emergingvisual token technology of generative multi-modal large models also shares a similar fundamental objective like visual coding: maximizing semantic information fidelity during the representation learning while minimizing computational cost. Therefore, this paper provides a comprehensive overview of two dominant technique families first – Visual Coding and Vision Token Technology – then we further unify them from the aspect of optimization, discussing the essence of compression efficiency and model performance trade-off behind. Next, based on the proposed unified formulation bridging visual coding andvisual token technology, we synthesize bidirectional insights of themselves and forecast the next-gen visual codec and token techniques. Last but not least, we experimentally show a large potential of the task-oriented token developments in the more practical tasks like multimodal LLMs (MLLMs), AI-generated content (AIGC), and embodied AI, as well as shedding light on the future possibility of standardizing a general token technology like the traditional codecs (e.g., H.264/265) with high efficiency for a wide range of intelligent tasks in a unified and effective manner.

💡 Research Summary

The paper “Compression Tells Intelligence: Visual Coding, Visual Token Technology, and the Unification” builds on the emerging principle that the ability to compress data efficiently is a hallmark of intelligence. It argues that this principle, well‑established in large language models (LLMs), extends naturally to the visual domain, where two historically separate research tracks have pursued the same ultimate goal: maximizing information fidelity while minimizing computational cost.

The first track is classical visual coding, a discipline rooted in Shannon’s information theory. The authors review the canonical three‑stage pipeline—transform (e.g., DCT, DWT), quantization, and entropy coding—and trace its evolution from JPEG and JPEG‑2000 to modern video standards such as HEVC/H.265 and VVC/H.266. They then discuss neural codecs, which replace hand‑crafted modules with end‑to‑end trained autoencoders, learned hyper‑priors, and context models. Neural codecs achieve rate‑distortion performance that rivals or surpasses the best hand‑crafted standards, especially at low bitrates.

The second track is visual token technology, which has emerged alongside multimodal large models (MLLMs). Visual tokenization converts images or video into sequences that can be processed by transformer‑based language models. The paper distinguishes continuous tokenizers (patch embeddings as in CLIP, DINOv2) from discrete tokenizers (latent encoders followed by vector‑quantization codebooks such as VQ‑VAE or VQ‑GAN). After tokenization, a token‑compression stage reduces the number of tokens using attention‑based selection, similarity clustering, reinforcement‑learning‑driven pruning, or pooling mechanisms. The objective here is not pixel‑wise reconstruction but preservation of semantic content that is useful for downstream tasks such as visual question answering, captioning, or robotic perception.

The core contribution is a unified theoretical framework that brings the two paradigms under a common optimization lens. By introducing both Shannon entropy (bit‑level uncertainty) and semantic entropy (uncertainty of high‑level concepts) into a single Lagrangian, the authors derive a joint rate‑distortion‑accuracy trade‑off. They model three sources of information bottlenecks: (1) dimensionality reduction in the transform stage, (2) bit allocation in quantization, and (3) token selection in the compression stage. This formulation reveals that improving compression efficiency in one domain (e.g., better quantization) can directly benefit the other (e.g., higher semantic fidelity of tokens) when the loss functions are aligned.

Experimental validation focuses on task‑oriented visual tokens. Using benchmark datasets for VQA, image classification, and embodied‑AI perception, the authors show that, for a fixed bitrate, token‑based pipelines achieve 10‑15 % higher downstream accuracy than pipelines that first compress with a traditional codec and then extract features. Moreover, multimodal LLMs that ingest these compact token streams require fewer transformer layers and less FLOPs while maintaining or improving overall multimodal reasoning performance.

Finally, the paper looks ahead to standardization. It cites ongoing efforts such as MPEG’s Video Coding for Machines (VCM) and JPEG AI, arguing that a unified codec‑token standard could deliver a single bitstream that simultaneously satisfies human visual quality and machine‑centric semantic needs. Such a standard would dramatically reduce storage and transmission costs across a wide range of applications—including multimodal LLMs, AI‑generated content pipelines, and embodied AI systems—while providing a principled metric (compression efficiency) for measuring and comparing intelligence across modalities.