Human-in-the-Loop, But Where?

Image Credits: Generated with Gemini

The conversation about AI oversight has shifted from "Should humans supervise AI?" to "At which exact points in the workflow must a human still be responsible for saying yes or no?" The first question is philosophical, while the second is operational. And in 2026, with AI agents executing multi-step workflows across healthcare, finance, and public administration, the operational question is the one that determines whether systems work safely or fail catastrophically.

Human-in-the-loop, the principle that humans should review and validate AI decisions before they take effect, has become a regulatory favorite. The EU AI Act requires human oversight for high-risk AI systems, mandating that natural persons must be able to interpret outputs, detect automation bias, and intervene to prevent harm. Australia's Privacy Act reform will enforce similar rules for automated decision-making starting late this year, requiring organizations to notify individuals when decisions are made without meaningful human involvement and to offer a right to request human review. The logic is simply that if an AI system makes a mistake in a high-stakes domain, a human checkpoint should catch it before damage occurs.

The problem is that human-in-the-loop does not work the way regulators sometimes assume it does. Research consistently shows that humans do not catch AI errors at the rate required to justify relying on them as safety mechanisms. A 2025 analysis of algorithmic decision-making governance found that humans in "in the loop" governance functions provided correct oversight only about half the time, with lapses primarily caused by motivation to ensure compliance with organizational goals rather than responsible AI principles. Microsoft synthesized approximately 60 papers on AI overreliance and reached a conclusion that should concern anyone designing oversight systems. Overreliance on AI makes it difficult for users to meaningfully leverage the strengths of AI systems and to oversee their weaknesses, and detailed explanations often increase user reliance on all AI recommendations regardless of accuracy (Getsignify, 2026).

This is automation bias, the tendency to favor suggestions from automated systems even when contradictory evidence is present. It manifests in two types of errors. Errors of commission occur when humans follow incorrect AI advice, and errors of omission occur when humans fail to act because the AI did not prompt them to do so. In healthcare, automation bias has already been documented in clinical decision support systems, where clinicians override their own correct decisions in favor of erroneous advice from diagnostic software in 6% of cases (PMC, 2011). A prospective study in mammography demonstrated that radiologists, regardless of experience, were significantly influenced by AI-suggested BI-RADS categories, and when the AI provided incorrect suggestions, the accuracy of radiologists' assessments dropped significantly (ICENet, 2025).

The regulatory assumption is that human oversight prevents harm. The empirical reality is that human oversight often rubber-stamps automated decisions, especially under time pressure, high workload, or after long periods of error-free operation, which generate what researchers call "learned carelessness" (Getsignify, 2026). As one report warned, companies implementing AI agents will initially require human approval for every action, but users will quickly be bombarded with thousands of permission requests daily, leading them to mindlessly click through approvals or enable auto-approve features to avoid constant interruptions (Witness AI, 2026). 

Organizations will discover too late that their carefully designed human-in-the-loop controls were defeated not by sophisticated attacks, but by the simple human tendency to streamline annoying workflows.

This does not mean human oversight is useless. It means that where you place humans in the workflow matters far more than whether you place them there at all. Article 14 of the EU AI Act requires that high-risk AI systems be designed to allow individuals conducting oversight to decide, in any particular situation, not to use the system or to otherwise disregard, override, or reverse the output. This is the right instinct, but it requires three conditions that are often missing in practice. First, the human must have interpretable information from the AI system, including clear explanations of its recommendations and an indication of confidence level. Second, the human must have unambiguous authority and practical ability to reverse, ignore, or halt the AI's decision or operation, which requires designing user interfaces with clear, timely alerts and straightforward override functions. Third, the human must retain the cognitive capacity to evaluate the AI's output critically, which degrades over time as automation takes over decision-making processes.

The third condition is the hardest to maintain. A study warns of the "out-of-the-loop" performance loss, where automation causes operators to lose situational awareness and manual skill over time, and automation bias is aggravated by task complexity, time pressure, high workload, and long periods of error-free automation operation. Another study found that consistent engagement with an AI assistant leads to greater skill decrements than engagement with traditional automation systems, because AI takes over cognitive processes, leaving fewer opportunities to keep skills honed (Getsignify, 2026). This creates a paradox. The longer an AI system operates without error, the less capable the human overseer becomes at catching errors when they finally occur.

So where must humans stay in the chain? The answer depends on the stakes and the reversibility of the action. In domains where errors have irreversible consequences, human review before execution is non-negotiable. The EU AI Act mandates that high-risk biometric systems require at least two qualified people to verify identifications before taking action. In healthcare, final treatment decisions, especially for irreversible procedures, require a physician's judgment that cannot be delegated to an algorithm. 

In domains where errors are reversible but costly, human review after execution may be sufficient. An article described a model where AI agents function independently within a workflow while leaving critical decisions to humans, tackling simpler, time-consuming tasks such as document preparation, case triage, and data extraction, with their actions remaining visible and subject to review (Forbes, 2026). This is human-on-the-loop rather than human-in-the-loop. The AI operates autonomously within guardrails, and the human role shifts from operator to overseer, providing final executive approval only when the AI flags uncertainty or crosses predefined thresholds.

The shift from human-in-the-loop to human-on-the-loop reflects a recognition that the bottleneck is not AI capability but human capacity. If software can operate itself, humans no longer sit inside the workflow, they sit above it, providing judgment, expertise, nuance, creativity, and empathy, while AI executes and carries out the operational burden. This is a better model for most enterprise applications, but it requires careful design. The AI must be transparent about what it is doing, why it is doing it, and when it is uncertain. The human must have the authority and the cognitive bandwidth to intervene when necessary. And the organization must accept that some decisions, no matter how efficiently an AI can make them, are not appropriate to automate.