PAL: Probing Audio Encoders via LLMs -- Audio Information Transfer into LLMs

Integration of audio perception into large language models (LLMs) is an emerging research area for enabling machine listening applications, yet efficient transfer of rich audio semantics from audio encoders to LLMs remains underexplored. The most widely used integration paradigm projects audio-encoder output tokens into the LLM input space (e.g., via an MLP or a Q-Former) and then prepends or inserts them into the text token sequence. We refer to this generic scheme as Prepend to the LLM’s input token space (PLITS) integration. We propose an efficient alternative, Lightweight Audio LLM Integration (LAL). LAL injects audio representations solely through the attention mechanism at selected LLM layers, bypassing the feed-forward module. It encodes rich audio semantics at an appropriate level of abstraction for integration into different transformer blocks, substantially reducing computational overhead compared to existing approaches. We further introduce PAL, a hybrid integration approach for efficiently Probing Audio encoders via LLM. PAL applies PLITS only to a compact set of summary tokens while integrating the full audio token sequence via LAL. Under an identical training curriculum, LAL consistently matches or outperforms existing integration approaches across multiple base LLMs and tasks, with improvements of up to 30% over a strong PLITS baseline, while reducing memory usage by about 60% and increasing throughput by about 190%. Moreover, PAL matches or exceeds PLITS performance while offering substantially better computational and memory efficiency.

💡 Research Summary

The paper tackles the problem of efficiently integrating audio encoders with large language models (LLMs), a key step toward building audio‑LLMs that can understand and reason about sound. The dominant approach in the literature, which the authors name PLITS (Prepend to the LLM’s input token space), first projects raw‑audio features (via an MLP, Q‑Former, etc.) into the LLM’s embedding space, then prepends the resulting audio tokens to the text token stream. The concatenated sequence is fed through every transformer layer, so audio tokens undergo the same self‑attention and feed‑forward (FFN) processing as text. While simple, PLITS suffers from two major drawbacks: (1) computational complexity grows quadratically with the total token count (O((Nₐ+Nₜ)²)), which becomes prohibitive when audio token length Nₐ far exceeds text length Nₜ; (2) passing audio tokens through the FFN can dilute the original acoustic semantics.

To address these issues the authors propose LAL (Lightweight Audio‑LLM Integration). LAL’s central hypothesis is that the attention mechanism alone is sufficient for transmitting audio information to the text side. Concretely, at each LLM layer a small per‑layer projector Pₗ maps the audio encoder’s output into the layer’s hidden dimension. Queries are generated only from the text hidden states, while keys and values are formed from the concatenation of text and projected audio tokens. Consequently, audio tokens act only as keys and values; they never issue queries and they never pass through the FFN. The resulting attention cost drops to O((Nₐ+Nₜ)·Nₜ), eliminating the Nₐ² term that dominates PLITS when Nₐ≫Nₜ. Moreover, because audio tokens bypass the FFN, both FLOPs and activation memory are substantially reduced. The authors argue that this design lets audio act as contextual information that re‑weights text representations, while the text’s own FFN continues to exploit the LLM’s massive parametric knowledge.

Building on LAL, the authors introduce PAL (Probe Audio encoders via LLM), a hybrid scheme that combines the strengths of PLITS and LAL. PAL first extracts a compact set of “summary” audio tokens (e.g., a global representation of the whole clip) and injects them via PLITS, preserving a global context that is useful for many tasks. The remaining detailed audio token sequence is then integrated using LAL, ensuring that fine‑grained acoustic cues are delivered efficiently through attention only. This hybridization yields a model that matches or exceeds PLITS performance while achieving dramatic efficiency gains.

The experimental protocol is thorough and fair: multiple base LLMs (Llama‑2, Falcon, Mistral) are trained under an identical curriculum on a unified audio‑text dataset covering speech recognition, sound‑event classification, and audio‑question answering. Results show that LAL consistently matches PLITS accuracy and often outperforms it by up to 30% on challenging benchmarks, especially when audio sequences are long. PAL further improves the trade‑off, delivering up to 60% memory reduction, 1.9× faster training throughput, and 2.1× faster inference compared to PLITS, all without sacrificing accuracy. An ablation where the LLM’s FFN layers are frozen demonstrates that LAL’s reliance on attention alone does not erode performance, suggesting that the model can retain the LLM’s pretrained knowledge while still benefiting from multimodal grounding.

The paper also provides a conceptual analysis of knowledge flow. It distinguishes between (1) parametric knowledge stored in the FFN (learned during massive language pre‑training) and (2) contextual knowledge supplied by the audio modality via attention. LAL leverages the latter to “piggy‑back” on the former: audio reshapes text token embeddings, which then trigger the appropriate language pathways inside the FFN. This viewpoint explains why bypassing the FFN for audio does not harm, and can even help, performance.

In summary, the authors contribute:

1. LAL – a lightweight, attention‑only integration that cuts computational complexity from quadratic to linear in the text length and removes audio‑FFN processing.
2. PAL – a hybrid integration that applies PLITS to a few global summary tokens while using LAL for the full audio stream, achieving superior efficiency‑performance trade‑offs.
3. A rigorous, standardized benchmark suite that validates these claims across several LLM backbones and audio tasks.

The work establishes a new design principle for multimodal LLMs: audio need not be treated as first‑class tokens throughout the entire transformer stack; instead, injecting audio as keys/values in selected attention layers can preserve semantics, retain the LLM’s language knowledge, and dramatically lower resource demands. This insight is likely to influence future large‑scale audio‑LLM architectures and could be extended to other modalities where token length is a bottleneck.