Automation Bias in the AI Act: On the Legal Implications of Attempting to De-Bias Human Oversight of AI

This paper examines the legal implications of the explicit mentioning of automation bias (AB) in the Artificial Intelligence Act (AIA). The AIA mandates human oversight for high-risk AI systems and requires providers to enable awareness of AB, i.e., the human tendency to over-rely on AI outputs. The paper analyses the embedding of this extra-juridical concept in the AIA, the asymmetric division of responsibility between AI providers and deployers for mitigating AB, and the challenges of legally enforcing this novel awareness requirement. The analysis shows that the AIA’s focus on providers does not adequately address design and context as causes of AB, and questions whether the AIA should directly regulate the risk of AB rather than just mandating awareness. As the AIA’s approach requires a balance between legal mandates and behavioural science, the paper proposes that harmonised standards should reference the state of research on AB and human-AI interaction, holding both providers and deployers accountable. Ultimately, further empirical research on human-AI interaction will be essential for effective safeguards.

💡 Research Summary

The article examines the novel inclusion of automation bias (AB) in the European Union’s Artificial Intelligence Act (AIA) and evaluates its legal ramifications for high‑risk AI systems. The authors begin by noting that while the AIA adopts a risk‑based framework and obliges “human oversight” for high‑risk AI, it uniquely references AB—a well‑studied psychological tendency to over‑rely on automated outputs—in Article 14(4)(b). This explicit mention is unusual in EU legislation, and the legislative history provides little justification for singling out AB over other cognitive biases.

The paper then analyses how AB fits into the AIA’s structure. High‑risk classification is based on safety components or fundamental‑rights impact, yet the design characteristics that make a system prone to AB are not part of the risk criteria. The authors trace the AB clause to the Commission’s April 2021 draft and to a European Parliament opinion that warned about “over‑confidence in AI” and cited experimental evidence, but the final text offers only a brief definition and no scientific elaboration.

A central argument concerns the asymmetric allocation of responsibility. The AIA obliges AI providers to “enable awareness” of AB for the natural persons tasked with oversight, while operators (deployers) are only required to perform generic human‑in‑the‑loop duties. Empirical research shows that AB emerges from a combination of technical design (e.g., automation‑friendly interfaces), contextual factors (time pressure, information overload), and user characteristics (training, expertise). Because these determinants largely reside in the deployer’s environment, placing the awareness duty solely on providers creates a “responsibility asymmetry” that may leave the real source of bias unaddressed.

Enforcement challenges are highlighted next. Proving that AB actually occurred in a specific decision‑making episode is difficult, especially where no objective “ground truth” exists (e.g., criminal justice, public procurement). The AIA provides no concrete metrics, monitoring tools, or post‑hoc verification procedures for AB, rendering the “awareness” requirement abstract and potentially unenforceable.

To bridge these gaps, the authors propose several policy measures. First, treat AB itself as a regulated risk, imposing design‑, training‑, and evaluation‑related obligations on both providers and deployers. Second, embed the latest behavioural‑science findings into the forthcoming harmonised standards, creating a “Human‑Centred Design for Automation Bias” guideline. Third, mandate continuous education and simulation‑based awareness programmes, coupled with meta‑feedback mechanisms (e.g., decision‑log analytics) that can flag over‑reliance in real time. Fourth, adopt a graduated sanction regime that distinguishes between failure to raise awareness and actual bias‑induced harm, thereby balancing liability between providers and operators.

In conclusion, the paper argues that while the AIA’s recognition of human fallibility is a step forward, the current approach—relying on a vague awareness duty without robust scientific grounding or enforceable mechanisms—falls short of effectively mitigating AB. Ongoing interdisciplinary research on measuring and counteracting automation bias in human‑AI interaction is essential for refining EU AI governance and ensuring that high‑risk AI systems do not undermine health, safety, or fundamental rights through unchecked human over‑reliance.