Neural FOXP2 -- Language Specific Neuron Steering for Targeted Language Improvement in LLMs

LLMs are multilingual by training, yet their lingua franca is often English, reflecting English language dominance in pretraining. Other languages remain in parametric memory but are systematically suppressed. We argue that language defaultness is governed by a sparse, low-rank control circuit, language neurons, that can be mechanistically isolated and safely steered. We introduce Neural FOXP2, that makes a chosen language (Hindi or Spanish) primary in a model by steering language-specific neurons. Neural FOXP2 proceeds in three stages: (i) Localize: We train per-layer SAEs so each activation decomposes into a small set of active feature components. For every feature, we quantify English vs. Hindi/Spanish selectivity overall logit-mass lift toward the target-language token set. Tracing the top-ranked features back to their strongest contributing units yields a compact language-neuron set. (ii) Steering directions: We localize controllable language-shift geometry via a spectral low-rank analysis. For each layer, we build English to target activation-difference matrices and perform layerwise SVD to extract the dominant singular directions governing language change. The eigengap and effective-rank spectra identify a compact steering subspace and an empirically chosen intervention window (where these directions are strongest and most stable). (iii) Steer: We apply a signed, sparse activation shift targeted to the language neurons. Concretely, within low to mid layers we add a positive steering along the target-language dominant directions and a compensating negative shift toward the null space for the English neurons, yielding controllable target-language defaultness.

💡 Research Summary

Neural FOXP2 addresses the persistent problem that multilingual large language models (LLMs) default to English despite being trained on many languages. The authors hypothesize that a sparse, low‑rank control circuit—referred to as “language neurons”—governs this defaultness. By identifying and manipulating these neurons, the model can be made to treat a target language (Hindi or Spanish) as its primary output language without full retraining.

The method consists of three stages.

1. Localize – For each transformer layer, a Sparse AutoEncoder (SAE) is trained to reconstruct the residual‑stream activations as a sparse linear combination of latent features. Each feature is evaluated for (a) selectivity: the difference in activation between matched English and target‑language prompts, and (b) causal lift: the change in early‑step logit mass toward target‑language tokens when the feature is nudged by a small amount. Features that are both highly selective and have a positive lift slope are designated as language‑neuron candidates.

2. Steering Directions – The authors construct English‑to‑target activation‑difference matrices for each layer and perform a layer‑wise Singular Value Decomposition (SVD). The leading singular vectors capture the dominant low‑rank directions that drive language change. By inspecting eigengaps and effective‑rank spectra, they identify a compact steering subspace and an “intervention window” (typically low‑to‑mid layers) where these directions are strongest and most stable.

3. Steer – A signed, sparse activation shift is applied to the selected language neurons. Positive shifts are added along the target‑language dominant directions, while compensating negative shifts are applied toward the null space of English‑direction components. This manipulation is performed directly in the SAE latent space, then decoded back to the original activation dimension, ensuring that only a tiny fraction of the model’s internal representation is altered.

Experiments are conducted on LLaMA‑3 8B, targeting Hindi and Spanish. Evaluation spans machine translation, question answering, natural language inference, and summarization. The primary metric for language defaultness, ΔM (target‑English logit‑mass difference) measured over the first three decoding steps, shows a substantial increase (≈0.35–0.48) after intervention, indicating that the target language becomes the default under weak prompting. Standard task metrics (BLEU, EM, ROUGE, F1) are largely preserved, with occasional modest gains, demonstrating that the surgical edit does not degrade overall capability. Ablation studies confirm that removing the identified neurons reduces ΔM, while forced activation of them boosts ΔM, establishing a causal link.

Strengths of the work include a clear causal definition of language defaultness, the use of SAE‑derived features to avoid superposition pitfalls, and a mathematically grounded low‑rank analysis that yields a stable intervention window. Limitations are the computational overhead of training per‑layer SAEs, the focus on only two target languages, and the reliance on manually chosen shift magnitude (ε) without an automated optimization scheme. The paper also leaves open how the method scales to larger models (e.g., 70B) or to scenarios with mixed‑language prompts.

Overall, Neural FOXP2 provides the first end‑to‑end pipeline that (i) discovers language‑specific control features, (ii) isolates a low‑dimensional steering subspace, and (iii) applies a minimal, sign‑controlled edit to make a non‑English language the default. This reframes multilingual adaptation from data‑heavy retraining to internal control‑mass redistribution, opening avenues for efficient, safe, and interpretable language‑specific tuning of future LLMs.