LLM-based In-situ Thought Exchanges for Critical Paper Reading

Critical reading is a primary way through which researchers develop their critical thinking skills. While exchanging thoughts and opinions with peers can strengthen critical reading, junior researchers often lack access to peers who can offer diverse perspectives. To address this gap, we designed an in-situ thought exchange interface informed by peer feedback from a formative study (N=8) to support junior researchers’ critical paper reading. We evaluated the effects of thought exchanges under three conditions (no-agent, single-agent, and multi-agent) with 46 junior researchers over two weeks. Our results showed that incorporating agent-mediated thought exchanges during paper reading significantly improved participants’ critical thinking scores compared to the no-agent condition. In the single-agent condition, participants more frequently made reflective annotations on the paper content. In the multi-agent condition, participants engaged more actively with agents’ responses. Our qualitative analysis further revealed that participants compared and analyzed multiple perspectives in the multi-agent condition. This work contributes to understanding in-situ AI-based support for critical paper reading through thought exchanges and offers design implications for future research.

💡 Research Summary

This paper presents a comprehensive investigation into the use of Large Language Model (LLM)-based AI agents to support junior researchers in critical paper reading. The core problem addressed is the lack of access to diverse peer perspectives, which is crucial for developing critical thinking skills through academic reading.

The research was conducted in two main phases. First, a formative study with eight junior researchers was carried out to understand current reading practices and gather design requirements. This study involved an initial ideation session using Google Docs with a human peer (the experimenter) and a subsequent iteration session with a low-fidelity AI agent prototype. The insights led to four concrete Design Goals (DGs): DG1) Facilitate cross-disciplinary thought exchange beyond the paper’s immediate content, DG2) Provide section-based critical thinking guidance (e.g., pre-reading questions), DG3) Offer cooperative support without disrupting the user’s reading flow, and DG4) Include follow-up support like key sentence highlighting.

Guided by these DGs, the researchers developed an “in-situ thought exchange interface.” This interface allows users to read a PDF paper in a main viewer while simultaneously interacting with AI agents in a dedicated chat panel, enabling seamless context-aware discussions without window switching. The agents can respond to user highlights or questions, or proactively pose critical questions related to specific paper sections.

The main evaluation was a two-week controlled user study with 46 junior researchers (Master’s and PhD students). Participants were randomly assigned to one of three between-subjects conditions: 1) a No-agent condition where they read papers alone using a basic PDF viewer with annotation tools, 2) a Single-agent condition where they read while engaging in a dialogue with one AI agent, and 3) a Multi-agent condition where they interacted with multiple AI agents, each simulating a different academic persona (e.g., a cognitive psychologist, a computer scientist). The study measured outcomes using a standardized critical thinking skills test (pre- and post-study), along with quantitative logging of annotations and interaction behaviors, and qualitative data from surveys and interviews.

The key findings are as follows. In answer to RQ1, both the Single-agent and Multi-agent conditions led to a statistically significant improvement in participants’ critical thinking scores compared to the No-agent condition. This demonstrates the fundamental value of integrating agent-mediated thought exchanges into the reading process.

Addressing RQ2, while the Single and Multi-agent conditions did not differ significantly in final critical thinking scores, they shaped the reading process and user strategies in distinctly different ways. Quantitative analysis revealed that participants in the Single-agent condition made significantly more reflective annotations on the paper content itself after receiving agent responses. This suggests a one-on-one dialogue fosters deeper, more introspective engagement with the primary text. Conversely, participants in the Multi-agent condition engaged more actively with the agents’ responses themselves. They spent more time comparing, questioning, and commenting on the different viewpoints presented by the multiple agents, indicating a practice of comparative analysis and synthesis across perspectives.

Qualitative findings reinforced this distinction. Participants in the Multi-agent condition reported benefits from “comparing and contrasting different angles,” which enhanced their analytical skills. However, some also reported feeling “overwhelmed by too much external information,” highlighting a key design challenge for multi-agent systems: managing cognitive load and information abundance.

In conclusion, this work makes significant contributions by: 1) providing empirical evidence that LLM-based agents can effectively scaffold critical thinking during academic reading, 2) delineating the differential impacts of single versus multi-agent interactions on reading behaviors (reflective depth vs. perspective-broadening analysis), and 3) offering a set of validated design goals and implications for future human-AI collaborative systems aimed at augmenting, rather than replacing, higher-order human intellectual work. The research underscores the potential of AI as a discussion partner in scholarly contexts, paving the way for more sophisticated tools to support research training and critical analysis.