Everyone has AI fatigue. It's in every vendor pitch and every conference keynote. Which is why why we need to stop and ask whether any of it is being used competently.
That's the conversation I wanted to have with Tyler Harmon, CEO of Iaso Automated Medical Systems, on a recent episode of the Global Medical Device Podcast. Tyler has lived both sides of this story. He started as an inventor, co-founding a hemorrhage-control device, then spent years building and fixing quality systems at some of the largest device companies in the business. Today he's putting machine learning to work in the ICU, on models that flag patients at risk of Acute Respiratory Distress Syndrome (ARDS) earlier than a clinician might catch it alone.
Here's the takeaway from that conversation, in my words: the difference between AI that makes a patient safer and AI that looks incredible in a demo comes down to competence. And for a medtech team, competence is mostly your existing quality discipline, applied to AI without giving it a special exemption.
AI is already in your build, whether or not it's governed
Ask a product team today if they use AI and you'll get a careful answer. Ask if anyone on the team has pasted a requirement into a chatbot, drafted a test protocol with one, or leaned on a copilot in their IDE, and the answer changes. It's already in the workflow. The only variable is whether it's governed.
Tyler's first objective in our conversation was to demystify the technology, and I think that's smart. A lot of teams talk about AI as if it were a new category of magic. But look behind the curtain and what you"ll find is software. As Tyler described it, these systems are nonlinear math applied to problems we used to solve with deterministic code. Useful, sometimes remarkable, and still software. If you can understand it, you can govern it. If you treat it as magic, you'll govern it poorly.
Why a consumer model is built to be unpredictable
This was something I hadn't considered before.
A consumer large language model (LLM) has a bit of engineered randomness in it. Tyler's point was that the randomness is deliberate, a design choice that keeps the experience fresh and keeps you engaged. The product's job is to hold your attention.
That is important to understand, because the implication for randomness in a medical device is... disconcerting, at best. The property that makes a chatbot feel delightful is the same property you'd never accept in a regulated system. A device is supposed to behave the same way on Tuesday that it behaved on Monday. Tyler talked about the wiggle room inside a neural network and the work involved in engineering enough of it out to get behavior you can trust in a clinical setting.
The same logic explains why these tools can be so confident and so wrong at the same time. Good citations were never a user need for a consumer engagement product, so hallucinated sources aren't a bug in the eyes of the people who built it. In a medical context that's the whole ballgame. It's why Tyler's team leans on approaches like grounding outputs in vetted sources, keeping a human in the loop on anything that touches a decision, and refusing to train on synthetic data that would let a model overfit to its own inventions.
BONUS RESOURCE: Click here to download your free 3-in-1 SaMD Gap Assessment Tool!
Start with the only question that matters: is this a SaMD?
When a team asks me how to govern their new AI feature, I've started answering with a question back, and it's the same one Tyler mentioned: Is the thing you're building Software as a Medical Device (SaMD) or not?
That single question does most of the work. If it meets the definition, you don't need a novel AI playbook. You run it through the pipeline you already own: classify it, model the risk, and answer the oldest question in this industry, which is how could this hurt someone. The AI part might change the inputs, but not the discipline.
For the regulatory side of this, the clearest starting point is our breakdown of FDA guidance on AI-enabled devices, including what the agency actually expects from AI-enabled device software functions.
The one genuinely new wrinkle is that a machine learning model has the ability to keep learning after you ship it, and a cleared device is supposed to stay predictable. That difference is what the Predetermined Change Control Plan (PCCP) exists to manage. We walked through the mechanics of that in a separate piece on what a PCCP requires for an AI-enabled SaMD, so I won't relitigate it here. The point for this conversation is simple: the PCCP is beneficial only if the quality foundation under it is solid.
If you're still nailing down whether your product even qualifies, start with our guide to Software as a Medical Device and the classification questions that follow from it.
Treat the model like a supplier
Here's a way to frame the conversation that engineering teams usually appreciate.
When you bring an AI capability into your device, you're effectively hiring a company. You often can't say who built the underlying model, how it was trained, or what changed in the last release. That's supplier management, and it's software management, and there's no point in pretending otherwise.
So what does that mean? It means you need to treat model outputs as Software of Unknown Provenance (SOUP) until you've done the work to know better. Interrogate the vendor the way you'd interrogate any critical supplier. Build the evidence trail. AI validation for a medical device is the same verification and validation muscle you already use, but applied to a component that happens to be probabilistic. If a supplier claims they built their own model, that claim deserves scrutiny, not blind acceptance.
Competence is your quality system, applied to AI
Tyler's framing for the governance side was to treat AI as its own vertical inside the quality system you already have. It lives inside your existing processes as a documented part of them. In practice that looks like AI-specific procedures written as child documents under your existing quality management system, aimed squarely at the two failure modes that are unique to these tools: randomness and hallucination.
This is also where the industry is heading formally. AI governance as a documented, auditable vertical is now the substance of ISO/IEC 42001:2023, the management-system standard for AI. At Greenlight Guru we treat it that way in our own products, because a reviewer isn't going to accept the AI did it as an answer, and neither should you. If you want to see what that governance discipline looks like in practice, we walked through it when we detailed our own ISO/IEC 42001 certification for AI governance.
BONUS RESOURCE: Click here to download your free 3-in-1 SaMD Gap Assessment Tool!
Governing AI this way is what frees your team to use it effectively. Tyler described these tools as a force multiplier for clinicians, the way a mechanical engineer uses 3D-CAD modeling software. The engineer still owns the design. The tool just lets them do more of what they're good at. That's the mindset a competent medtech team brings to AI internally too. Use it aggressively, and stay accountable for every output.
Launch with confidence, so you can scale with confidence
If there's one thing I'd take from Tyler into your next planning meeting, it's this. The teams that use AI well are the ones who refused to exempt it from the discipline they already trust, whatever model they happen to be running. They classified it, they modeled the risk, they vetted the supplier, and they wrote it into the quality system as a real vertical.
That's what lets you launch an AI-enabled device with confidence, and it's what lets you keep scaling after clearance without the whole thing wobbling. The competence is ordinary. It's the work you already know how to do, applied to a tool that's designed to look smarter than it really is.
Keep reading
If you are building AI governance into your quality system, these related guides go deeper on the specific components:
- FDA guidance on AI-enabled devices
- Software as a Medical Device: the ultimate guide
- FDA PCCP requirements for AI SaMD
- Greenlight Guru achieves ISO 42001 certification for AI
- AI in clinical investigations today
- Greenlight Guru AI: what we've built and where we're headed
If you're building AI into a device right now and you want to see how a purpose-built quality system handles the SaMD, risk, and supplier side of this in one place, book a walkthrough with our team. Bring the hard questions. Those are the ones we want to answer before a reviewer brings them up.
BONUS RESOURCE: 3-in-1 SaMD Gap Assessment Tool
