Evaluating Generative AI in the Lab: Methodological Challenges and Guidelines

Generative AI (GenAI) systems are inherently non-deterministic, producing varied outputs even for identical inputs. While this variability is central to their appeal, it challenges established HCI evaluation practices that typically assume consistent and predictable system behavior. Designing controlled lab studies under such conditions therefore remains a key methodological challenge. We present a reflective multi-case analysis of four lab-based user studies with GenAI-integrated prototypes, spanning conversational in-car assistant systems and image generation tools for design workflows. Through cross-case reflection and thematic analysis across all study phases, we identify five methodological challenges and propose eighteen practice-oriented recommendations, organized into five guidelines. These challenges represent methodological constructs that are either amplified, redefined, or newly introduced by GenAI’s stochastic nature: (C1) reliance on familiar interaction patterns, (C2) fidelity-control trade-offs, (C3) feedback and trust, (C4) gaps in usability evaluation, and (C5) interpretive ambiguity between interface and system issues. Our guidelines address these challenges through strategies such as reframing onboarding to help participants manage unpredictability, extending evaluation with constructs such as trust and intent alignment, and logging system events, including hallucinations and latency, to support transparent analysis. This work contributes (1) a methodological reflection on how GenAI’s stochastic nature unsettles lab-based HCI evaluation and (2) eighteen recommendations that help researchers design more transparent, robust, and comparable studies of GenAI systems in controlled settings.

💡 Research Summary

The paper investigates how the inherent non‑determinism of generative AI (GenAI) disrupts traditional laboratory‑based user‑study methods in human‑computer interaction (HCI). By reflecting on four controlled lab experiments—two conversational in‑car assistants and two image‑generation tools for professional design—the authors identify systematic methodological problems that arise when the same input can produce different outputs. Using a multi‑case study design, researcher notes, study artifacts, and inductive thematic analysis, they extract five overarching challenges: (C1) amplified reliance on familiar interaction patterns, (C2) fidelity‑control trade‑offs, (C3) redefined feedback loops and trust, (C4) gaps in usability evaluation, and (C5) interpretive ambiguity between interface and system behavior.

Each challenge is illustrated with concrete observations. For example, participants often default to known UI conventions (C1) and become frustrated when the GenAI response deviates unpredictably. High‑fidelity models increase output variability, making experimental control difficult (C2), while pre‑generated low‑fidelity samples sacrifice realism. Trust is eroded by hallucinations and factual errors, requiring new measurement instruments beyond standard SUS or NASA‑TLX (C3). Traditional metrics such as task completion time or error rate fail to capture creative or exploratory outcomes, prompting the need for mixed‑methods indicators like creativity scores and idea diversity (C4). Finally, distinguishing whether a problem stems from the interface design or the underlying model becomes ambiguous, necessitating dual logging of system events and user actions (C5).

To address these challenges, the authors propose five methodological guidelines (G1‑G5), each paired with specific practice‑oriented recommendations, totaling eighteen actionable items. G1 advises redesigning participant onboarding to set expectations about variability and to teach prompt‑crafting techniques. G2 offers a fidelity‑selection matrix that aligns model choice and sampling strategy with research goals. G3 expands trust and feedback measurement by adding intent‑alignment questions, a trust‑tracking questionnaire, and error‑awareness checks. G4 enriches usability evaluation with creativity, exploration depth, and qualitative interview components, moving beyond pure performance metrics. G5 calls for transparent data collection through dual logging of system metrics (latency, hallucination flags, token counts) and user interaction logs (clicks, speech, gaze).

Each recommendation is presented as a checklist item—e.g., “record prompt variability”, “flag hallucinations in real time”, “correlate trust scores with output accuracy”—to facilitate immediate adoption by researchers. The paper emphasizes that these guidelines are modular resources that can be mixed and matched depending on the study’s context, thereby preserving experimental rigor while embracing the stochastic nature of GenAI.

Overall, the contribution is twofold: (1) a reflective, evidence‑based articulation of how GenAI’s stochastic behavior unsettles core assumptions of controlled HCI evaluation, and (2) a concrete, practitioner‑focused set of guidelines and recommendations that enable more transparent, robust, and comparable lab studies of generative AI systems. This work thus provides a methodological roadmap for the HCI community as GenAI becomes increasingly embedded in everyday and professional interactive technologies.