Why a clinician has to stay in the loop

"A human is in the loop" is the standard reassurance offered about AI in medicine, including by us. It is worth asking what the evidence says about whether having a human in the loop actually works, because the answer is more interesting than the slogan.

The failure mode has a name

Researchers call it automation bias. A 2017 study in BMC Medical Informatics and Decision Making defines it as "the tendency to use automated cues (such as CDS alerts) as a heuristic replacement for vigilant information seeking and processing".

In plain terms: once software is checking, people stop checking as hard. Not through laziness, but because that is what tools are for, and this instinct is usually correct. It is what makes automation useful.

It fails in two directions. Omission errors, where users "fail to notice problems because they were not alerted to the problem by CDS". And commission errors, where users "comply with incorrect recommendations".

The numbers

The study tested prescribers with decision support that was sometimes correct and sometimes wrong. When the support was incorrect, participants made 28.7 percent more omission errors, missing genuine prescribing errors the system had not flagged, and 56.9 percent more commission errors, declining to prescribe safe medicines because of false alerts.

The control condition is the most sobering part. With no decision support at all, participants missed nearly half of all prescribing errors.

Read those together and you get the honest picture. Unaided humans are not reliable at this. Humans with imperfect software are better in some ways and systematically worse in others. Neither is a solved problem, and "a human reviews it" is not by itself a safety guarantee.

What actually determines whether oversight is real

The literature on mitigating automation bias points at conditions rather than intentions. Time pressure makes it worse, because strained attention falls back on heuristics, and a rushed reviewer approving an AI output is not meaningfully checking it.

Research on suppression strategies, where the system deliberately withholds its suggestion until the clinician has formed their own view, suggests that sequencing matters: a person who has already reasoned independently engages with a recommendation differently from one who is handed a conclusion first.

So the useful questions about any "human in the loop" claim are practical. Does the person have time to disagree? Do they see the case before they see the answer? Are they accountable for the outcome, or only for having clicked?

What this means for how Dr.life is built

This research is the reason for a design choice that might otherwise look like hedging.

The AI in Dr.life does not produce a diagnosis for a clinician to approve. It answers questions from the record, asks the follow-ups a clinician would ask, and hands over the conversation. The clinician who arrives is making the assessment, not ratifying one.

That distinction is the whole point. A doctor rubber-stamping a machine's conclusion is exactly the arrangement this literature says degrades. A doctor who has the history gathered and then makes the call is using the software for what it is good at.

The uncomfortable admission

We should be straight that this is a claim about design intent, and that automation bias is a pull that no design fully escapes. A clinician who reads an AI summary before forming a view has already been anchored, and that is true of our product as much as anyone's.

What can be said is that the risk is known, documented, and designed against, rather than waved away with the phrase "human in the loop". Anyone who offers you that phrase without being able to describe how they handle time pressure, sequencing and accountability is offering reassurance rather than a safeguard.