MoLoRA: Composable Specialization via Per-Token Adapter Routing

Multi-adapter serving systems route entire sequences to a single adapter, forcing a choice when requests span multiple domains. This assumption fails in two important settings: (1) multimodal generation, where text and image tokens require different adapters within the same sequence, and (2) mixed-capability requests like “write code to solve this equation,” which need expertise from multiple specialized adapters. We introduce per-token routing, which routes individual tokens to adapters based on either vocabulary structure (for multimodal models) or learned gating (for semantic specialization). Per-token routing is provably optimal, achieving work N for N tokens versus K \cdot N for per-sequence routing with K adapter types. Our key contribution is MoLoRA (Mixture of LoRA), which enables composable specialization: load multiple domain-specific adapters and let a learned router select the appropriate adapter per-token. We demonstrate that specialization dramatically beats scale: MoLoRA enables Qwen3-1.7B to exceed Qwen3-8B across four reasoning benchmarks while being 4.7x smaller. This enables modular expertise at inference time: train focused LoRAs independently, combine them without retraining, and add new capabilities by simply loading new adapters.

💡 Research Summary

The paper addresses a fundamental limitation of existing multi‑adapter serving systems for large language models (LLMs). Current approaches such as S‑LoRA and Punica route an entire request (i.e., a whole sequence) to a single LoRA adapter, which works poorly when a request contains multiple modalities or requires several specialized capabilities. The authors identify two concrete scenarios: (1) multimodal generation where text and image tokens interleave, and (2) mixed‑capability queries like “write Python code to solve this differential equation,” which need both programming and mathematical expertise.

To solve this, they propose per‑token routing, a framework that assigns each token to an adapter independently. The routing function r(i) can be deterministic—based on vocabulary ranges that encode modality information—or learned, using a small router network that produces logits over K adapters for each token. The deterministic case requires only M‑1 integer comparisons (M = number of modalities) and runs in O(1) per token, while the learned case incurs a softmax over K·d operations (d = hidden dimension). The authors prove two theorems: (1) computational optimality—per‑token routing achieves the minimal work of N·c_pass compared with K·N·c_pass for per‑sequence routing; (2) equivalence to block‑diagonal sparse attention, showing that existing MoE and sparse‑attention optimizations can be directly reused.

Building on this framework, they introduce MoLoRA (Mixture of LoRA). MoLoRA loads multiple domain‑specific LoRA adapters simultaneously and uses a router to select the appropriate adapter for each token. The implementation groups tokens by their selected adapter using a histogram and atomic increments, then performs batched GEMM operations per group. This dispatch kernel mirrors the one used in Mixture‑of‑Experts (MoE) systems, allowing the reuse of adaptive tiling, block‑sparse kernels, and other performance tricks. System‑level engineering includes CUDA graph capture and a “hot‑set” memory pool that reduces P99 latency by 67×.

Empirically, the authors evaluate MoLoRA on the Qwen3‑1.7B model with four specialized adapters (code, math, general reasoning, creative writing). On four reasoning benchmarks—GSM8K, Math, BBH, and GPQA—MoLoRA‑augmented Qwen3‑1.7B outperforms the much larger Qwen3‑8B model, achieving gains of +14%, +8%, +2.5%, and +2.1% respectively, despite being 4.7× smaller in parameter count. For multimodal workloads involving K modalities, per‑token routing reduces the number of forward passes from K to 1, yielding a 4.1× raw speedup that compounds to 5.5× after system optimizations.

The paper also discusses limitations. Deterministic vocabulary routing assumes contiguous token ranges for each modality, which does not hold for encoder‑based multimodal models (e.g., Flamingo, LLaVA) where image and text tokens share the same vocab space. Semantic specialization (code vs. math) cannot be distinguished by vocab alone, requiring learned routing. Finer‑grained sub‑modality routing and multi‑attribute routing (modality × domain × task) also demand learned, possibly hierarchical routers.

In summary, MoLoRA provides a theoretically optimal, practically efficient, and modular solution for composable specialization in LLM serving. By enabling token‑level adapter selection, it allows a single deployment to combine many expert LoRAs without retraining, offering a scalable path to enrich large models with new capabilities at inference time. This work bridges the gap between adapter‑based fine‑tuning and Mixture‑of‑Experts routing, opening new avenues for cost‑effective, high‑quality multi‑domain LLM services.