ISD-Agent-Bench: A Comprehensive Benchmark for Evaluating LLM-based Instructional Design Agents

Large Language Model (LLM) agents have shown promising potential in automating Instructional Systems Design (ISD), a systematic approach to developing educational programs. However, evaluating these agents remains challenging due to the lack of standardized benchmarks and the risk of LLM-as-judge bias. We present ISD-Agent-Bench, a comprehensive benchmark comprising 25,795 scenarios generated via a Context Matrix framework that combines 51 contextual variables across 5 categories with 33 ISD sub-steps derived from the ADDIE model. To ensure evaluation reliability, we employ a multi-judge protocol using diverse LLMs from different providers, achieving high inter-judge reliability. We compare existing ISD agents with novel agents grounded in classical ISD theories such as ADDIE, Dick & Carey, and Rapid Prototyping ISD. Experiments on 1,017 test scenarios demonstrate that integrating classical ISD frameworks with modern ReAct-style reasoning achieves the highest performance, outperforming both pure theory-based agents and technique-only approaches. Further analysis reveals that theoretical quality strongly correlates with benchmark performance, with theory-based agents showing significant advantages in problem-centered design and objective-assessment alignment. Our work provides a foundation for systematic LLM-based ISD research.

💡 Research Summary

The paper addresses the pressing need for a standardized benchmark to evaluate large‑language‑model (LLM) agents that aim to automate the full lifecycle of Instructional Systems Design (ISD). While recent work has shown that LLMs can assist with lesson planning or content generation, no existing benchmark systematically assesses agents that must perform multi‑step planning, contextual adaptation, tool use, and iterative refinement across the entire ADDIE process. To fill this gap, the authors introduce ISD‑Agent‑Bench, a large‑scale benchmark consisting of 25,795 generated scenarios and a curated test set of 1,202 cases.

The core of the benchmark is the “Context Matrix” framework, a two‑dimensional construct that combines (1) a Context Axis of 51 variables organized into five categories—Learner Characteristics, Institutional Context, Educational Domain, Delivery Mode, and Constraints—and (2) an ISD Axis that enumerates 33 sub‑steps derived from the ADDIE model, grouped into 13 evaluation items. By crossing these axes, the authors systematically cover a wide spectrum of realistic instructional design situations, from elementary language lessons to corporate leadership courses delivered via synchronous online platforms.

Scenario generation proceeds through a multi‑stage pipeline. First, 10,577 SCOPUS abstracts containing “instructional design” are harvested as seed contexts. After filtering and de‑duplication, GPT‑4o expands each seed into a full scenario, specifying SMART learning objectives, resource constraints, learner prior knowledge, and required deliverables. To mitigate combinatorial explosion, stratified sampling and targeted augmentation are used, yielding an additional 16,953 varied scenarios that balance under‑represented learner ages, class sizes, and self‑directed learning contexts. Quality control combines rule‑based field checks with LLM‑based logical validation, and difficulty levels (Easy/Medium/Hard) are computed from weighted composites of goal count, domain expertise, resource breadth, course duration, and budget.

Evaluation methodology blends outcome quality and process behavior. An ADDIE‑based rubric scores the completeness, alignment, and pedagogical soundness of the agent’s artifacts (learning objectives, design documents, assessment items, etc.). Simultaneously, trajectory analysis records tool calls, phase transitions, and re‑planning loops to assess efficiency and adherence to the prescribed workflow. To avoid the “LLM‑as‑judge” bias, a multi‑judge protocol employs three distinct LLMs—Gemini‑3‑Flash, GPT‑5‑mini, and Solar‑Pro3—as independent assessors. Inter‑judge reliability reaches 0.905, confirming the robustness of the scoring process.

Four prototype agents are built to benchmark the dataset:

1. React‑ADDIE – integrates the classical ADDIE framework with ReAct‑style reasoning (Observe‑Think‑Act loops), allowing the agent to dynamically select sub‑tasks, invoke external tools, and revise its plan based on feedback.
2. ADDIE‑Agent – a baseline that follows a strict phase‑by‑phase decomposition without explicit reasoning loops.
3. Dick‑Carey‑Agent – implements the nine‑step Dick & Carey model, emphasizing tight alignment between objectives, instruction, and assessment.
4. RPISD‑Agent – follows Rapid Prototyping ISD, focusing on iterative prototype development and quick pilot testing.

Experiments on the 1,017 test scenarios reveal that React‑ADDIE achieves the highest overall score (86.49), outperforming the pure theory‑based ADDIE‑Agent (82.96) and a technique‑only baseline (84.07). The advantage is most pronounced in problem‑centered design and objective‑assessment alignment, suggesting that embedding established ISD theory acts as a quality‑assurance scaffold that guides the LLM’s reasoning. Moreover, correlation analysis shows that the theoretical “quality” of an agent becomes increasingly predictive of performance as scenario difficulty rises, indicating that theory‑driven structure mitigates the combinatorial complexity of harder design tasks.

The authors acknowledge several limitations. Scenario generation relies heavily on LLMs, so human expert validation is limited; the corpus is English‑centric, raising questions about cross‑lingual applicability; and ethical considerations such as data privacy, bias, and accountability in real‑world deployments are not addressed. Future work is proposed in three directions: (1) hybrid human‑LLM collaborative design pipelines, (2) integration of multimodal tools (e.g., simulation, video) and connection to actual Learning Management Systems, and (3) longitudinal studies measuring learner outcomes after agents produce instructional materials.

In summary, ISD‑Agent‑Bench provides the first comprehensive, systematic benchmark for evaluating LLM‑driven instructional design agents. By coupling a richly parameterized Context Matrix with a rigorous multi‑judge evaluation protocol, the benchmark enables reproducible comparison of agents across a realistic spectrum of educational contexts. The empirical findings demonstrate that the synergy of classical ISD theory and modern ReAct‑style reasoning yields superior performance, offering a clear research pathway for building more reliable, scalable, and pedagogically sound AI agents in the education domain.