An Open Letter to Healthcare Leaders

Dear healthcare leaders,

In a world obsessed with hot takes, here's mine:

AI agents, unless they have been properly trained, are probably doing a lousy job. And adding more of them to the mix does not make things easier. It actually makes it harder.

That is not a criticism of AI. I have spent years building a company around what AI can do in healthcare, and I believe in it. But I have watched too many organizations treat getting an AI agent live as the finish line, and move on without asking the question that matters most: if our AI agent gave a patient the wrong answer today, how would we know?

Most of the time, the answer is: you probably would not.

Here is the little lie no one wants to say out loud: AI agents are built so customers can have 24/7 access to help. More often than not, that is not what customers in healthcare experience. They do not feel helped. They feel stuck. And when someone is stuck on a question about their own health, they do not always ask it a different way or wait for a human to pick up. They go somewhere else for an answer, or worse, they do nothing at all. They do not advance the ball on their own care, because the AI agent that was supposed to be a front door has become a barrier instead.

Healthcare conversations don't stay inside defined scenarios. Patients are emotional, confused, or scared. Members ask questions no workflow anticipated. AI agents perform well when the stakes are low or the answers are not filled with nuance. Healthcare is rarely one of those cases. So they miss important signals, give answers that are close (but wrong), and fail quietly in ways that leave no obvious trace. The interaction concludes. The metric looks clean. And downstream, a patient makes a care decision based on incomplete information, or a compliance obligation goes unmet, or a safety signal disappears into a log no one is reading.

This is not a technology problem. It is the reality of deploying autonomous and probabilistic systems into environments as complex and high-stakes as healthcare. Every AI agent will fail. That is not a prediction. It is a given. The difference is in how often. And the frequency of failure comes down to how well you train your agent, and whether you keep training it, on purpose and with intention, as an ongoing discipline rather than a one-time setup. That takes an entire QA system built around continuous monitoring and improvement, not a launch checklist. The question is not whether failure will happen — it is whether your organization will see it, and whether you will catch it before it causes harm.

Most organizations are not set up to catch it. Not because they don’t care, but because they do not have the right infrastructure. A safety signal embedded in natural language does not wait for a weekly review cycle. A wrong answer a member accepted does not trigger an escalation. A compliance issue that surfaces in a conversation and disappears is invisible to tools that were not built to recognize it. These failures are quiet, and they compound — and by the time most organizations find out, the damage is already done.

Supervision is the answer, and I want to be specific about what that means.

Supervision is not a dashboard or a monthly QA report. Supervision is a continuous, always-on system for seeing what your AI agents are actually saying and doing in live conversations, so you can catch problems early and correct them before they scale. It starts before deployment, using real conversation data to determine where AI actually belongs and where human judgment is non-negotiable. It continues once agents are live, listening across both AI-led and human-led conversations in one system, with the healthcare-specific intelligence to recognize what matters. It also means hearing how your customers actually respond: what they bring into the conversation that the AI agent does not know how to handle, where they get frustrated, where they give up. And it never stops, because AI performance is not static. Language drifts. Edge cases accumulate. The agent that performed well at launch will not necessarily perform well six months from now.

There is also a reason audit and operations sit in separate functions. It is not because operations cannot be trusted. It is because no function should be left to grade its own work. The same logic applies here. The system watching your AI agents should not be the same system running them.

I am not asking you to slow down. I am asking you to stay in control as you go fast. Because autonomy without supervision is a liability in healthcare. Today. In the conversations your AI agents are having right now.

If the question — how would we know? — gives you pause, that is the gap worth closing. We put together a guide on what supervision actually requires in practice. It is not a product brochure. It is the framework.

[Supervising AI Agents in Healthcare: How to Scale Autonomy Without Creating Risk →]

Amy Brown
CEO, Authenticx