HybridQuestion: Human-AI Collaboration for Identifying High-Impact Research Questions

The “AI Scientist” paradigm is transforming scientific research by automating key stages of the research process, from idea generation to scholarly writing. This shift is expected to accelerate discovery and expand the scope of scientific inquiry. However, a key question remains unclear: can AI scientists identify meaningful research questions? While Large Language Models (LLMs) have been applied successfully to task-specific ideation, their potential to conduct strategic, long-term assessments of past breakthroughs and future questions remains largely unexplored. To address this gap, we explore a human-AI hybrid solution that integrates the scalable data processing capabilities of AI with the value judgment of human experts. Our methodology is structured in three phases. The first phase, AI-Accelerated Information Gathering, leverages AI’s advantage in processing vast amounts of literature to generate a hybrid information base. The second phase, Candidate Question Proposing, utilizes this synthesized data to prompt an ensemble of six diverse LLMs to propose an initial candidate pool, filtered via a cross-model voting mechanism. The third phase, Hybrid Question Selection, refines this pool through a multi-stage filtering process that progressively increases human oversight. To validate this system, we conducted an experiment aiming to identify the Top 10 Scientific Breakthroughs of 2025 and the Top 10 Scientific Questions for 2026 across five major disciplines. Our analysis reveals that while AI agents demonstrate high alignment with human experts in recognizing established breakthroughs, they exhibit greater divergence in forecasting prospective questions, suggesting that human judgment remains crucial for evaluating subjective, forward-looking challenges.

💡 Research Summary

**
The paper “HybridQuestion: Human‑AI Collaboration for Identifying High‑Impact Research Questions” proposes a three‑phase hybrid framework that combines large language models (LLMs) with human expert judgment to discover the top scientific breakthroughs of 2025 and the most consequential research questions for 2026 across five major disciplines (AI, physics, chemistry, biology, economics).

Phase 1 – AI‑Accelerated Information Gathering
The authors first harvest metadata from the OpenAlex repository for the years 2015‑2025. For each year they construct a keyword co‑occurrence graph, apply node2vec to obtain year‑specific embeddings, and compute a “hotness” score for every keyword. The hotness metric blends raw frequency with semantic proximity, using a Gaussian kernel whose bandwidth is set dynamically based on a low‑percentile of the cosine‑distance distribution. Keywords are then clustered greedily in descending hotness order; a dynamic distance threshold (derived from a separate percentile) determines cluster membership. Two keyword sets emerge: (i) “breakthrough_keywords” that are both historically prominent and show a rise in 2025, and (ii) “question_keywords” that either rank high in absolute hotness or exhibit rapid hotness acceleration, intended to seed prospective 2026 questions. To capture non‑academic signals, the authors run a “Deep Research” step: two LLMs (one US‑based, one China‑based) each retrieve five pieces of supplementary context (media reports, industry news, policy briefs) about 2024‑2025 developments. A third LLM merges these into a unified narrative.

Phase 2 – Candidate Question Proposing
Six diverse LLMs (three US, three China) are prompted with the synthesized information. For each breakthrough keyword, the system fetches high‑citation 2025 papers and supplies the deep‑research context; the model then generates a concise description of a potential breakthrough. For each question keyword, both recent (2025) and foundational (post‑2015) high‑citation papers are retrieved, combined with the contextual narrative, and the model proposes a forward‑looking research question. Each model independently produces 100 breakthrough candidates and 100 question candidates, yielding a total pool of 600 items per list. A cross‑model voting stage follows: each LLM casts 100 approval votes on the combined pool, and the 100 items receiving the most votes are retained for the next stage. This ensemble voting mitigates model‑specific biases and balances regional perspectives.

Phase 3 – Hybrid Question Selection
The final selection proceeds in two human‑AI voting stages.
Stage 1 (100 → 30): Thirty graduate‑level human voters and seventy simulated AI agents (instantiated via a “Virtual Lab”‑inspired method) vote without weight restriction; each participant can vote for any number of candidates they deem viable. Human and AI votes are weighted equally (1:1).
Stage 2 (30 → 10): Ten domain experts (one per discipline) join the process, and the voting weight shifts to a human‑dominant ratio of 7:1 (human:AI). The top ten items after this stage constitute the final “Top 10 Breakthroughs of 2025” and “Top 10 Grand Questions for 2026”.

Experimental Findings
Applying the pipeline to the five target fields, the authors report that AI‑generated breakthrough lists align closely with human expert consensus (≈85 % overlap), successfully surfacing well‑known milestones such as advanced reinforcement‑learning systems and breakthroughs in quantum‑computing hardware. In contrast, the prospective question list shows a lower alignment (≈60 % overlap). Human experts preferentially elevated meta‑level challenges—standardization of evaluation metrics, data‑infrastructure, causal reasoning in foundation models—whereas AI tended to prioritize emerging technical capabilities (e.g., new model architectures, novel material synthesis pathways). This divergence underscores the current limitation of LLMs: strong pattern‑recognition on existing data but weaker capacity for long‑term value judgment and policy‑oriented foresight.

Contributions

1. Extends LLM‑driven ideation from tactical problem solving to strategic horizon scanning, demonstrating a systematic method for reviewing past breakthroughs and forecasting future research frontiers.
2. Introduces a novel end‑to‑end hybrid workflow that integrates AI in both the data‑synthesis (phase 1) and candidate‑filtering (phase 2) stages, while preserving human oversight through a staged voting mechanism.
3. Produces two curated, high‑impact lists that can serve as strategic guides for researchers, funding agencies, and policy makers.

Limitations & Future Work
The hotness calculation relies on several hyper‑parameters (percentile thresholds, kernel bandwidth) whose sensitivity is not fully explored. The simulated AI agents in Stage 1, while useful for scaling, may not faithfully replicate expert reasoning; future work should incorporate real‑world AI assistants with calibrated confidence scores. The expert panel size (10 specialists) limits generalizability; expanding the panel and including interdisciplinary perspectives would strengthen robustness. Potential extensions include automated hyper‑parameter optimization, meta‑reinforcement learning to let AI improve its question‑generation policy over successive cycles, and integration of societal impact metrics to better capture the “value” dimension of future research questions.

In summary, the study demonstrates that a carefully designed human‑AI hybrid system can efficiently mine massive scholarly corpora, generate plausible breakthrough candidates, and, when combined with expert judgment, produce a credible set of forward‑looking research questions. While AI excels at breadth and speed, human expertise remains indispensable for evaluating the subjective, long‑term significance of prospective scientific challenges. This work offers a practical blueprint for future AI‑augmented research agenda setting.