Adaptive Confidence Gating in Multi-Agent Collaboration for Efficient and Optimized Code Generation

While Large Language Models (LLMs) have catalyzed breakthroughs in automated code generation, Small Language Models (SLMs) often encounter reasoning bottlenecks and failure loops when addressing complex logical requirements. To overcome these challenges, we propose DebateCoder, a multi-agent collaborative framework designed to improve the reasoning ability of SLMs (e.g., Pangu-1B) in resource-constrained environments. DebateCoder uses a structured role-playing protocol with three agents: User Agent (A_UA), Technical Agent (A_TA), and Quality Assurance Agent (A_QA). It also includes an Adaptive Confidence Gating mechanism with a 95% threshold to balance accuracy and inference efficiency. In addition, we introduce a multi-turn deliberation module and a reviewer-guided analytical debugging loop for orthogonal pre-generation debate and post-generation refinement. Experiments on HumanEval and MBPP show that DebateCoder achieves 70.12% Pass@1 on HumanEval, outperforming MapCoder while reducing API overhead by about 35%. These results indicate that collaborative protocols can mitigate limitations of small-parameter models and provide a scalable, efficient approach to high-quality automated software engineering.

💡 Research Summary

DebateCoder introduces a lightweight yet powerful multi‑agent framework that enables small language models (SLMs), specifically Pangu‑1B, to achieve high‑quality code generation comparable to much larger models. The system defines three specialized agents—User Agent (A_UA), Technical Agent (A_TA), and Quality Assurance Agent (A_QA)—each instantiated with the same base model but guided by distinct persona‑specific prompts. In the first “parallel initialization” step, every agent independently produces a detailed plan and a confidence score (0‑100) reflecting its perceived solvability of the task. The average confidence Γ is compared against a pre‑set threshold τ of 95 %. If Γ ≥ τ, the problem is deemed low‑complexity and the framework skips all subsequent debate rounds, directly proceeding to plan synthesis, thereby saving tokens and latency.

When Γ < τ, DebateCoder launches an iterative debate phase lasting up to three rounds. In each round, an agent receives the full set of plans generated by its peers in the previous round, critiques them, and revises its own plan accordingly. After each revision, confidence scores are recomputed, allowing early exit if consensus is reached before the maximum round count. This cross‑agent deliberation encourages divergent reasoning—A_TA may swap data structures after A_QA highlights edge‑case concerns, while A_UA ensures functional intent remains intact.

The final round’s plans are aggregated by a dedicated Synthesis Agent, which resolves any remaining contradictions and produces a master plan (P*). P* serves as the definitive specification for the Coding Agent, which emits executable Python code. If the generated code fails the benchmark test suite, a Reviewer Agent analyses the failure log, problem description, and code snippet to produce a cause analysis and a concrete fix plan. A Debugging Agent then applies this plan, avoiding the “failure loops” that often plague SLMs when they receive only binary pass/fail signals.

Experiments were conducted on four benchmarks: HumanEval, HumanEval‑ET, MBPP, and MBPP‑ET, using the same Pangu‑1B backbone for all agents. Compared with a direct zero‑shot baseline and the state‑of‑the‑art MapCoder (which relies on larger models), DebateCoder achieves 70.12 % Pass@1 on HumanEval, 60.98 % on HumanEval‑ET, and 63.22 % on MBPP. The average performance across all datasets is 58.91 %, surpassing MapCoder’s 55.95 % and the direct baseline’s 39.51 %. Moreover, the adaptive confidence gating reduces API calls and token consumption by roughly 35 %, especially for tasks where the initial confidence is high.

Key contributions include: (1) a role‑based multi‑agent architecture that compensates for the limited reasoning depth of SLMs; (2) an adaptive confidence gating mechanism that dynamically balances accuracy and efficiency; (3) a structured multi‑turn debate coupled with a reviewer‑guided debugging loop that mitigates self‑correction “failure loops”; and (4) extensive empirical validation showing that small models can approach the performance of much larger systems when equipped with well‑designed collaborative protocols.

Limitations are acknowledged: the confidence threshold and maximum debate rounds are fixed, which may not be optimal for every problem domain; extremely complex algorithmic tasks still expose the capacity ceiling of Pangu‑1B; and the current design does not incorporate dynamic threshold adaptation or meta‑learning to further refine the debate process. Future work could explore adaptive threshold tuning, agent‑specific fine‑tuning or knowledge distillation, and meta‑learning strategies to prevent “debate collapse” where agents reinforce each other’s errors.

In summary, DebateCoder demonstrates that sophisticated multi‑agent collaboration, when carefully engineered for small models, can deliver both high accuracy and computational efficiency, offering a practical pathway for cost‑constrained deployment of automated code generation systems.