Strong evidence
Keeping people in charge of consequential AI decisions
Why do laws and standards insist on human oversight of AI, and what makes oversight real?
Bottom line
EU law requires high-risk AI systems to be built so people can effectively oversee them1, including the ability to override or stop them and to stay alert to automation bias2. Research shows that bias is real: people tend to over-rely on automated advice3.
Article 14 of the EU Artificial Intelligence Act requires high-risk AI systems to be designed and developed so that natural persons can effectively oversee them while they are in use1.
The law spells out what that means in practice. The people overseeing a system must be able to stay aware of automation bias, decide not to use the system, disregard, override or reverse its output, and interrupt it through a stop button or similar procedure2.
The concern is grounded in research. A systematic review of 74 studies on automation bias, the tendency to over-rely on automation, found it was shaped by trust and confidence in the system, and made worse by workload, task complexity and time pressure3.
The same review found ways to reduce it: training, emphasising that the human user is accountable, and design choices such as showing updated confidence levels and giving information rather than a recommendation4.
Intergovernmental standards point the same way. The OECD AI Principles, adopted in 2019 and updated in 2024, call for AI that respects human rights and democratic values5.
What this does not show
- The automation bias review focuses mainly on healthcare decision support systems.
- The EU AI Act's oversight duties apply to systems classified as high-risk under the Act, not to every AI system.
Claims and sources
Article 14 of Regulation (EU) 2024/1689 requires high-risk AI systems to be designed and developed so that they can be effectively overseen by natural persons during the period in which they are in use.
- European Parliament and Council of the European Union (2024). Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). Official Journal of the European Union. https://eur-lex.europa.eu/eli/reg/2024/1689/oj
Article 14 requires that people assigned to oversight be enabled to remain aware of the tendency to over-rely on output (automation bias), to decide not to use the system or to disregard, override or reverse its output, and to intervene or interrupt it through a stop button or similar procedure.
- European Parliament and Council of the European Union (2024). Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). Official Journal of the European Union. https://eur-lex.europa.eu/eli/reg/2024/1689/oj
A systematic review of 74 studies found automation bias was mediated by user factors such as trust and confidence and by environmental factors including workload, task complexity and time constraints.
- Goddard K, Roudsari A, Wyatt JC (2012). Automation bias: a systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Association. doi:10.1136/amiajnl-2011-000089
Mitigators of automation bias included training, emphasising user accountability, and design factors such as updated confidence levels on system output and providing information rather than a recommendation.
- Goddard K, Roudsari A, Wyatt JC (2012). Automation bias: a systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Association. doi:10.1136/amiajnl-2011-000089
The OECD AI Principles, adopted in 2019 and updated in 2024, call for trustworthy AI that respects human rights and democratic values.
- Organisation for Economic Co-operation and Development (2019). OECD AI Principles. OECD.AI Policy Observatory. https://oecd.ai/en/ai-principles
Version 1. First published , updated . Approved by The Human First editors on . Sources last verified 2026-10-24. Machine-readable version.