You trained a model that works. It clears the accuracy target on your test set, the demo goes well, and engineering is ready to call it finished (demos always go well, that's the whole point of a demo, right?). Then someone in quality asks the $27,000 question... How do we know it's safe, and can you prove it?
I hear a version of that exchange all the time, on the podcast and in the hallway at events. I sat in on our recent session on validating AI in development software, and the part that stuck out had nothing to do with a specific rule. It was how far apart two groups of smart people can drift while building the same product.
Engineers know their model. They know how it was built, what it trained on, where it gets shaky. Across the building, quality and regulatory are working a different problem. They know exactly what a submission and an audit will demand, down to the record. Between those two groups, the evidence that should connect them goes missing. I call it the parallel worlds problem, and nowhere does it bite harder than with AI.
See, a model can pass every internal benchmark your data scientists care about and still be, in regulatory terms, unvalidated. Performance is not the same as proof. And AI has a habit of being sold on the strength of a good demo, which is exactly how a team ends up defending a design decision nobody wrote down.
So let me walk through the way the session framed it: how to actually validate the AI and machine learning (ML) inside a device before it goes out the door, not the regulatory backstory of why you have to.
BONUS RESOURCE: Click here to download a free PDF of your Risk Management Plan Template.
Start by drawing a boundary around what the model is allowed to do
You can't validate a thing you haven't defined. That sounds obvious until you watch a team try to write acceptance criteria for a model whose intended use is still a paragraph of marketing language.
Before any test matters, you need a tight statement of the model's job. What clinical question does it answer? What are its inputs, and what should happen when it gets one outside that range? One of our Guru Services folks, has a useful way of putting it: The dangerous version of AI is the "unbounded" kind, where nothing constrains how the model can change or what it does with a case it has never seen. Bounding the problem first is what makes everything downstream testable.
Write down the edges. The populations the model was built for, the ones it wasn't, the confidence level below which it should hand the decision back to a human. Those edges become the spec you validate against.
Your data is the first thing that gets pulled apart
If I had to guess where an auditor starts, it's your data. The architecture gets the attention in the demo. The data gets the scrutiny in the review.
An ML component is only as trustworthy as the set it learned from, so the provenance of that data has to be as traceable as any other design input (yes, including the data your vendor swears is already clean). Where did it come from? Does it represent the patients who will actually be scanned, measured, or monitored once the device is in the field? A model trained on one hospital's population and pointed at a different demographic is a validation problem disguised as performance.
The thing that trips teams up most is separation. Your training data, your tuning data, and your final test data have to stay firewalled from each other. The moment information leaks from the test set back into training, your headline accuracy number turns into fiction, and you won't find out until the model underperforms on real patients. Treat the held-out test set like evidence in a locked room. And don't let anyone touch it until the final validation run.
Verification and validation answer different questions
These two words get used as if they mean the same thing, so we need to get this straight before we can go any further we need to get this straight before we can go any further.
Verification asks whether you built the model right. Does the code match the design, do the components hit their numbers, does the pipeline behave. Validation asks the harder question of whether the finished thing does the right job for the real user in the real workflow. A model can be perfectly verified and still fail validation, because it solves a slightly different problem than the clinician in front of it actually has.
For classical software we've drawn this line for years, so it's important to evaluate how software validation is meant to work before you layer ML on top. What AI changes is that the "does it do the right job" question now rides on statistical behavior across a population, not a deterministic output you can trace line by line. So your protocol has to define performance in clinical terms, on data the model has never seen, at the operating point you'll actually deploy, not the one that flattered it in a slide.
The session was firm on one more point: match the depth of your validation to risk. Not every component earns the same battery of tests, and the computer software assurance approach versus older computer system validation is FDA's own push in that direction. Put your rigor where a failure would reach a patient. Don't bury a low-risk feature under the paperwork you'd wrap around a diagnostic call.
Set your acceptance criteria against patient harm
Ninety-five percent accuracy means nothing on its own. Ninety-five percent at what task, missing which cases, with what happens to the patient when it's wrong?
The methodology that holds up starts from risk, the way ISO 14971 asks you to. Work out what harm looks like when the model is wrong, then set thresholds that reflect that harm. A false negative on a serious condition and a false positive that sends someone for a second look are different events, and your acceptance criteria should treat them that way. Sensitivity, specificity, and the errors you're willing to live with all trace back to the risk file, not to whatever metric made the model look best.
Build it this way and the validation record answers the only question that matters in a review. You didn't just show the model is accurate. You showed it's safe enough for its intended use, and you showed your work.
BONUS RESOURCE: Click here to download a free PDF of your Risk Management Plan Template.
Plan for the model to keep changing after launch
Here's what makes AI genuinely different from a catheter. It can change after it ships, either because you retrain it or because the world around it moves.
A model slowly degrades once the incoming data stops looking like the data it learned from. A new scanner or a shift in patient mix, and performance slips without a single line of code changing. So validation can't be a one-time gate at the end. You need a monitoring plan that watches real-world performance and warns you when the model has wandered off its validated behavior, plus a clear rule for when a human stays in the loop.
If you know up front how the model is allowed to evolve, a predetermined change control plan (PCCP) lets you define those boundaries in advance and update inside them without a fresh submission every time. It only works when the underlying processes are solid to begin with, which is the whole reason validation discipline comes first. And any third-party or off-the-shelf model you pull in still counts as software of unknown provenance under IEC 62304, so it needs its own evidence, not a vendor's reassurance.
Bonus resource: For the regulatory backdrop I'm deliberately not rehashing here, FDA's guidance on AI-enabled devices lays out what the agency actually expects.
Bonus resource: And for a look at what happens when validation gets skipped, I broke down the Purolea warning letter and what it really means for AI in medtech.
Where this leaves your quality system
Every step above produces evidence, and that evidence has to live somewhere an auditor can follow it. The methodology isn't the hard part. Keeping the trail intact, from intended use to data to test results to monitoring, while engineers move fast and the model keeps evolving, is the actual work.
An ISO 13485 lead auditor reads a submission looking for exactly that thread. When the records match what engineers actually built, reviews go fast. When they don't, you get a finding. And you get it at the worst possible time.
Here's where a quality system built for the way devices get made now earns its keep. More than 1,000 device companies run on Greenlight Guru because it connects design, risk, and traceability in one place, so the evidence for your AI/ML work gets captured as you build instead of reconstructed the week before an audit. The engineers were never in the wrong world. They just needed the trail to follow them into the room where it counts. If you're putting a model inside a device, see how a purpose-built platform keeps that evidence audit-ready.
Want the full methodology, including the parts I couldn't fit here? Watch the session on validating AI in development software on demand.
Keep reading
If you are building out your AI/ML validation process, these related guides go deeper on the specific components:
BONUS RESOURCE: Risk Management Plan Template
