VoiceAgentRAG: Solving the RAG Latency Bottleneck in Real-Time Voice Agents Using Dual-Agent Architectures

We present VoiceAgentRAG, an open-source dual-agent memory router that decouples retrieval from response generation. A background Slow Thinker agent continuously monitors the conversation stream, predicts likely follow-up topics using an LLM, and pre-fetches relevant document chunks into a FAISS-backed semantic cache. A foreground Fast Talker agent reads only from this sub-millisecond cache, bypassing the vector database entirely on cache hits.

💡 Research Summary

VoiceAgentRAG tackles the latency bottleneck that Retrieval‑Augmented Generation (RAG) introduces in real‑time voice assistants. The authors propose a dual‑agent architecture consisting of a background “Slow Thinker” and a foreground “Fast Talker”. The Slow Thinker continuously monitors the conversation stream, uses a large language model (LLM) to predict 3–5 likely follow‑up topics, retrieves relevant document chunks from a production vector database (Qdrant Cloud), and pre‑populates an in‑memory FAISS semantic cache. These operations run asynchronously while the user is listening to the current response, effectively overlapping retrieval with user think‑time.

The Fast Talker handles the latency‑critical path. For each user utterance it first computes an embedding (via OpenAI’s text‑embedding‑3‑small), then queries the FAISS cache. If the similarity between the query embedding and cached document embeddings exceeds a calibrated threshold τ = 0.40, the top‑k chunks are returned in sub‑millisecond time (≈0.35 ms) and fed to the LLM for generation. On a cache miss the system falls back to the traditional RAG pipeline (embedding → Qdrant search → LLM) and caches the retrieved chunks for future use, also notifying the Slow Thinker via a “PriorityRetrieval” event.

Key design choices include indexing the cache by document embeddings rather than query embeddings, which prevents mismatches where a cached answer is semantically close to the predicted query but not to the actual user question. The cache implements put, get, and eviction (TTL = 300 s, LRU) operations, and detects near‑duplicate chunks (cosine similarity > 0.95) to keep the cache size manageable.

The authors evaluate the system on a synthetic “NovaCRM” knowledge base (76 chunks) across ten conversation scenarios (200 total turns). Compared with a baseline RAG implementation that always queries Qdrant, VoiceAgentRAG achieves a 75 % overall cache hit rate (79 % on warm turns) and a 316× speedup on retrieval latency (110 ms → 0.35 ms). The saved retrieval time totals 16.5 seconds across the 150 cache‑hit queries. Scenarios with sustained topics (pricing deep‑dive, API integration, security) reach hit rates of 80–95 %, while highly dynamic, mixed‑topic conversations achieve lower rates (40–60 %). Cache warm‑up is rapid: hit rate climbs from ~58 % in turns 1‑4 to ~86 % by turns 5‑9, stabilizing around 80 % thereafter.

A sensitivity analysis of the similarity threshold shows that a value of 0.55 (appropriate for query‑to‑query matching) is too strict for query‑to‑document matching, resulting in only 15 % hits. Lowering τ to 0.40 balances precision and recall, yielding the reported performance.

Limitations identified include (1) the embedding API latency (~200 ms) which dominates the overall response time, (2) dependence on accurate topic prediction—mis‑predictions reduce cache effectiveness, and (3) the current single‑node in‑memory cache, which may not scale to large‑scale deployments. Future work proposes integrating local embedding models to cut API latency, enriching topic prediction with multimodal context, and extending the cache to a distributed FAISS cluster for scalability and fault tolerance.

In summary, VoiceAgentRAG demonstrates that decoupling retrieval from generation via a predictive, dual‑agent system can virtually eliminate the RAG retrieval latency in voice pipelines, making sub‑200 ms end‑to‑end response times feasible for natural conversational flow. The open‑source implementation and thorough empirical evaluation provide a practical blueprint for building low‑latency, knowledge‑grounded voice assistants.