When Speculation Spills Secrets: Side Channels via Speculative Decoding In LLMs

Deployed large language models (LLMs) often rely on speculative decoding, a technique that generates and verifies multiple candidate tokens in parallel, to improve throughput and latency. In this work, we reveal a new side-channel whereby input-dependent patterns of correct and incorrect speculations can be inferred by monitoring per-iteration token counts or packet sizes. In evaluations using research prototypes and production-grade vLLM serving frameworks, we show that an adversary monitoring these patterns can fingerprint user queries (from a set of 50 prompts) with over 75% accuracy across four speculative-decoding schemes at temperature 0.3: REST (100%), LADE (91.6%), BiLD (95.2%), and EAGLE (77.6%). Even at temperature 1.0, accuracy remains far above the 2% random baseline - REST (99.6%), LADE (61.2%), BiLD (63.6%), and EAGLE (24%). We also show the capability of the attacker to leak confidential datastore contents used for prediction at rates exceeding 25 tokens/sec. To defend against these, we propose and evaluate a suite of mitigations, including packet padding and iteration-wise token aggregation.

💡 Research Summary

The paper “When Speculation Spills Secrets: Side Channels via Speculative Decoding In LLMs” uncovers a previously unexamined privacy risk inherent to modern large‑language‑model (LLM) serving pipelines that employ speculative decoding. Speculative decoding speeds up inference by generating multiple candidate tokens in parallel using a smaller draft model, a retrieval‑based heuristic, or self‑drafting, and then having the large target model verify those candidates in a single iteration. While this yields 2×–5× latency reductions, the authors observe that the number of tokens emitted per iteration depends on whether the speculation was correct. Correct speculation yields several tokens in one iteration; a mis‑speculation forces the system back to standard autoregressive generation, emitting only one token. Crucially, this token‑count pattern is strongly correlated with the user’s input prompt.

The threat model focuses on two adversaries. A network attacker (e.g., a malicious ISP or compromised router) can observe encrypted packet sizes and inter‑packet timing of streaming LLM responses. Because packet size correlates with the number of tokens sent, the attacker can infer the per‑iteration token count without seeing the actual content. A malicious user, on the other hand, can craft inputs and directly inspect the token‑count trace to exfiltrate confidential data stored in the model’s retrieval or datastore components.

The attack proceeds in two phases. In an offline profiling stage, the attacker collects token‑count traces for a known set of prompts (the authors use 50 medical‑question prompts from MedAlpaca) across four speculative decoding schemes: REST, LADE, BiLD, and EAGLE. A random‑forest classifier is trained to map a trace to its originating prompt. In the online stage, the attacker measures packet sizes for an unknown query, reconstructs the trace, and feeds it to the classifier. Experiments show striking success rates: at temperature 0.3, identification accuracies reach 100 % for REST, 91.6 % for LADE, 95.2 % for BiLD, and 77.6 % for EAGLE. Even at temperature 1.0, where sampling is more random, accuracies remain far above the 2 % random baseline (REST 99.6 %, LADE 61.2 %, BiLD 63.6 %, EAGLE 24 %). The authors also demonstrate that a malicious user can leak confidential datastore contents at a rate exceeding 25 tokens per second by forcing correct speculations and observing the emitted tokens.

To mitigate these side‑channels, the paper proposes two complementary defenses. First, token aggregation across multiple iterations obscures the per‑iteration token count; this reduces attack accuracy by up to 50 % with negligible bandwidth overhead. Second, packet padding adds dummy bytes to each transmitted packet. Random padding (variable length) cuts accuracy by roughly 70 % but inflates payload size by a factor of 8.7. Fixed‑size padding eliminates the attack almost entirely (98 % accuracy reduction) at the cost of a 230× increase in packet size. The authors argue that a hybrid approach—moderate fixed padding combined with occasional aggregation—offers a practical trade‑off between security and performance.

The contribution of the work is threefold: (1) identification of a novel side‑channel unique to speculative decoding, distinct from previously known token‑length or timing channels; (2) demonstration that this channel enables high‑accuracy query fingerprinting and data exfiltration even when standard token‑size padding is applied; (3) systematic evaluation of mitigation strategies, quantifying their security gains against bandwidth and latency costs.

Limitations include the relatively small prompt set (50 queries) and the focus on specific speculative decoding implementations; broader real‑world workloads and noisy network conditions may affect attack robustness. Future research directions suggested include dynamic padding schemes that adapt to traffic patterns, redesigning speculative decoders to produce input‑independent token counts, and integrating multi‑modal side‑channel defenses (combining packet size, timing, and CPU usage signals).

In summary, while speculative decoding is a powerful tool for accelerating LLM inference, it inadvertently creates a measurable side‑channel that leaks private user inputs and internal datastore contents. Service providers must therefore adopt padding, aggregation, or redesign strategies to safeguard privacy without sacrificing the performance benefits that speculative decoding promises.