Fine-Grained Activation Steering: Steering Less, Achieving More

Activation steering has emerged as a cost-effective paradigm for modifying large language model (LLM) behaviors. Existing methods typically intervene at the block level, steering the bundled activations of selected attention heads, feedforward networks, or residual streams. However, we reveal that block-level activations are inherently heterogeneous, entangling beneficial, irrelevant, and harmful features, thereby rendering block-level steering coarse, inefficient, and intrusive. To investigate the root cause, we decompose block activations into fine-grained atomic unit (AU)-level activations, where each AU-level activation corresponds to a single dimension of the block activation, and each AU denotes a slice of the block weight matrix. Steering an AU-level activation is thus equivalent to steering its associated AU. Our theoretical and empirical analysis show that heterogeneity arises because different AUs or dimensions control distinct token distributions in LLM outputs. Hence, block-level steering inevitably moves helpful and harmful token directions together, which reduces efficiency. Restricting intervention to beneficial AUs yields more precise and effective steering. Building on this insight, we propose AUSteer, a simple and efficient method that operates at a finer granularity of the AU level. AUSteer first identifies discriminative AUs globally by computing activation momenta on contrastive samples. It then assigns adaptive steering strengths tailored to diverse inputs and selected AU activations. Comprehensive experiments on multiple LLMs and tasks show that AUSteer consistently surpasses advanced baselines while steering considerably fewer activations, demonstrating that steering less achieves more.

💡 Research Summary

The paper tackles the problem of activation steering—directly modifying the intermediate activations of large language models (LLMs) during inference—to shape model behavior without costly fine‑tuning or reinforcement learning. Existing steering approaches intervene at the “block” level, i.e., they add bias or rescaling vectors to whole multi‑head attention (MHA) blocks, feed‑forward networks (FFNs), or residual streams. While convenient, a block’s activation is a high‑dimensional vector (hundreds to thousands of dimensions) that bundles together many heterogeneous features: some are beneficial for a downstream task, others are irrelevant, and a few may even be harmful. Consequently, block‑level steering is coarse, inefficient, and overly intrusive because it amplifies both useful and detrimental signals simultaneously.

Key Insight – Heterogeneity of Block Activations
The authors empirically demonstrate this heterogeneity on LLaMA2‑7B‑Chat using the BoolQ dataset. When they steer individual dimensions of a selected attention head or FFN, performance varies dramatically: a single dimension (e.g., the 84th) can raise accuracy to 74.53 %—outperforming full‑block methods such as ITI (71.56 %) and SADI (73.70 %). Conversely, other dimensions (e.g., the 44th) degrade performance. This shows that mixing beneficial and harmful dimensions in a block dilutes the effect of steering.

Atomic Unit (AU) Formalization
To address the problem, the paper decomposes each linear projection y = W x into scalar‑wise contributions: each input dimension x_i multiplies a column W_:,i of the weight matrix. The column W_:,i is defined as an Atomic Unit (AU), and x_i becomes the AU‑level activation (a scalar). Steering a single scalar x_i is mathematically equivalent to steering its associated AU. This fine‑grained view enables selective manipulation of only those AUs that positively influence the task.

Theoretical Explanation – AU Controls Token Distributions
Building on prior work that interprets LLM behavior in embedding space, the authors argue that different AUs bias the model toward distinct output‑token distributions. They validate this by scaling individual AU coefficients from 10 up to 100 000 and measuring KL divergence between the model’s output at each strength and the output at the maximal strength. Two AUs (44 and 84) converge to markedly different token distributions, and the pairwise KL divergence between them grows with strength, confirming that each AU drives the model toward a different set of tokens. Qualitative examples show AU 84 promoting the correct “yes” token while suppressing “no,” whereas AU 44 amplifies irrelevant or incorrect tokens.

AUSteer: A Practical AU‑Level Steering Framework
AUSteer consists of two main components:

1. Activation Momentum – A new discriminative metric that compares the change of each AU’s activation between positive and negative contrastive samples. It is a counting‑based, scale‑invariant statistic that can be aggregated globally, allowing the identification of the most “discriminative” AUs across the whole model.

2. Adaptive Steering Scalars – Instead of injecting a fixed vector, AUSteer computes a per‑sample scalar that scales with the current activation value of the AU, preserving directionality. Moreover, each AU receives a strength weight proportional to its momentum score: highly discriminative AUs get stronger interventions, while less important ones receive weaker or no steering.

Experimental Evaluation
The authors evaluate AUSteer on several LLM families (LLaMA2‑7B, LLaMA3‑8B, Gemma‑2B) across diverse tasks: commonsense reasoning (BoolQ, ARC), mathematical problem solving (MATH), open‑ended generation (StoryCloze), and safety‑oriented benchmarks (toxicity reduction). Baselines include state‑of‑the‑art block‑level methods such as ITI, SADI, and ST‑A, which typically modify thousands of dimensions.

Results show that AUSteer, steering at most 100 AUs per model, consistently outperforms baselines by 2–4 percentage points in accuracy while reducing the intervention footprint by an order of magnitude. In safety tests, AUSteer cuts the generation of harmful tokens by over 30 % compared to the unsteered model, demonstrating that precise AU selection also improves alignment. Ablation studies confirm that both activation momentum and adaptive scaling are essential: removing momentum leads to random AU selection and performance collapse; using a fixed scalar instead of adaptive scaling reduces gains.

Implications and Future Directions
The work establishes that block‑level activation steering is fundamentally limited by internal heterogeneity and that AU‑level steering offers a more precise, efficient, and less intrusive alternative. AUSteer’s reliance on simple statistics rather than pretrained sparse autoencoders makes it broadly applicable across model families. Future research could explore context‑dependent AU selection, multi‑objective steering (balancing accuracy, safety, and factuality), and deeper theoretical links between AU weights and the geometry of the model’s token distribution manifold.

Conclusion
By dissecting block activations into atomic units, demonstrating that each AU governs a distinct output token distribution, and introducing a lightweight yet effective AU‑level steering method, the paper convincingly shows that “steering less achieves more.” AUSteer sets a new benchmark for cost‑effective, fine‑grained control of LLM behavior and opens a promising avenue for safer, more controllable language models.