Why LLM Unpredictability is a Liability in MedTech

July 13, 2026 ░░░░░░

#464 Why LLM Unpredictability is a Liability in MedTech

Artificial intelligence has officially entered the mainstream cultural zeitgeist, creating a wave of excitement—and a fair share of fatigue—across the medical device industry. In this episode, host Etienne Nichols sits down with Tyler Harmon, biomedical engineer and CEO of Iaso Automated Medical Systems, to cut through the marketing buzzwords. Together, they explore the technical realities behind the technology stack, shifting the conversation away from generic AI toward specific, actionable engineering frameworks.

The discussion highlights a critical distinction between traditional machine learning models and consumer-oriented Large Language Models (LLMs). Harmon explains that while technologies like convolutional neural networks (CNNs) have successfully processed medical imaging for years, modern LLMs introduce an intentional element of randomness to mimic human conversation. This lack of predictability presents unique challenges for medical device developers who operate in a deterministic, safety-critical environment where reproducibility is paramount.

Looking toward practical deployment, the episode addresses how companies can responsibly govern these tools both within their software architectures and their internal Quality Management Systems (QMS). From classifying external AI models as Software of Unknown Provenance (SOUP) under IEC 62304 to leveraging machine learning for early detection of Acute Respiratory Distress Syndrome (ARDS) in the ICU, this conversation serves as an essential guide for innovators looking to build the next generation of safe, compliant, and effective medical technologies.

Watch the Video:

Listen now:

Love this episode? Leave a review on iTunes!

Have suggestions or topics you’d like to hear about? Email us at podcast@greenlight.guru.

Key Timestamps

00:05 – Introduction to the dual nature of AI in MedTech: embedded clinical algorithms versus internal process optimization.
02:14 – Demystifying the math: Breaking down artificial intelligence into linear and non-linear algorithmic transformations.
04:30 – The Turing Test, Markov chains, and why consumer LLMs are mathematically designed to be unpredictable.
07:15 – Real-world success stories: How convolutional neural networks (CNNs) revolutionized emergency stroke triage.
09:42 – Inside Iaso Automated Medical Systems: Using non-LLM machine learning to identify Acute Respiratory Distress Syndrome (ARDS) in critical care.
12:10 – AI Governance in the QMS: Designing specialized Standard Operating Procedures (SOPs) and Machine Learning Management Systems (AIMS).
15:35 – Evaluating recent FDA 510(k) clearances for LLM-adjacent technologies and managing third-party stacks as SOUP.

Top takeaways from this episode

Classify External AI as SOUP: Treat third-party language models and external tech stacks as Software of Unknown Provenance (SOUP) under IEC 62304 frameworks, implementing rigorous risk management boundaries to isolate the core medical device logic.
Engineer Out Randomness: Recognize that consumer LLMs purposefully integrate randomness layers to maximize user engagement. For clinical safety, developers must utilize architectural harnesses or alternative machine learning methods (like CNNs or random forests) to force more deterministic outcomes.
Establish an AI Management System: Expand your organizational compliance beyond standard Quality Management Systems (QMS) and Information Security Management Systems (ISMS). Implement specific AI standard operating procedures and work instructions to govern internal token usage and data handling.
Prioritize Clinical Enablement Over Automation: Focus clinical software engineering on clearing workflow bottlenecks and flagging early-stage critical conditions (such as ARDS) to allow bedside clinicians to deploy their hands-on expertise faster.

References:

Berlin Criteria: The formal, quantitative medical classification standard used by clinicians to diagnose and grade the severity of Acute Respiratory Distress Syndrome.
IEC 62304: The international standard governing medical device software lifecycle processes, specifically detailing the management of Software of Unknown Provenance (SOUP).
Connect with Etienne Nichols on LinkedIn to stay updated on the latest episodes and industry insights.

MedTech 101 Section

Understanding Non-Linear Math and LLMs

Think of a traditional medical device software algorithm like a standard thermometer tracking a fever. It follows a straight, predictable line: if the temperature input increases by one degree, the reading on the screen changes by exactly one degree. This is a linear system.

Modern AI, like Large Language Models (LLMs), works more like a seasoned doctor trying to diagnose a complex case by listening to a patient's story. The human brain doesn't just look at variables in a straight line; it connects random pieces of past experiences, reads between the lines, and notes subtle shifts in tone. To replicate this mathematically, software engineers introduce non-linearity.

Instead of a straight line, the math behaves like a web of thousands of intersecting pathways. To make the system feel even more human, creators add a controlled "randomness layer" (similar to a digital coin flipper) so the software doesn't always choose the most obvious, predictable word next. While this makes chatting with a computer feel incredibly natural, it presents an engineering challenge for medical device developers who require identical, reproducible results every single time.

Memorable quotes from this episode

"If we as innovators can't explain things to a more general audience, we generally don't understand them ourselves. And if you can't do that, it's probably not the best idea to be implementing it into your products." - Tyler Harmon

"I am probably going to be the biggest advocate you'll ever talk to about 'doctors need enablement, not replacement.' We need to give them the tools, the force multipliers to tackle the challenges they're going to face this century." - Tyler Harmon

Feedback Call-to-Action

We love hearing from our community of MedTech professionals. Do you have thoughts on how AI governance should evolve, or is there a specific industry topic you want us to tackle next? We read every message and pride ourselves on providing personalized responses to our listeners. Share your feedback, reviews, and topic suggestions directly with our production team at podcast@greenlight.guru.

Transcript

Etienne Nichols: Hey, everyone. Welcome back to the Global Medical Device Podcast. My name is Etienne Nichols. Today we're going to be talking a little bit about, a little bit more in depth about AI than we have in the past.

And there's really two kinds of AI stories in MedTech. Now, before I get too far, I know a lot of people have AI fatigue. Everybody and their dog is using AI right now.

But there are some things that people don't talk about and there's really, you know, the things behind all of the AI.

We could talk about AI in medical devices themselves, but what we want to talk about today is the, the structure behind it or the how your processes can be affected by AI and how you can really be more effective with AI. So, my guest today to talk about this is live both sides of that line.

He started as an inventor at Georgia Tech, co-founding a hemorrhage control device for the US Special forces.

Since then, he's built and fixed quality systems at some of the biggest names in the bigot in the business.

Apple, Draeger, Fisher and Paykel and Johnson & Johnson acquisition. Today he's the CEO of Iaso Automated Medical Systems, bringing machine learning into the ICU for critical pulmonary care. And by the end of the.

And what am I getting wrong here? I'm seeing your face here.

Tyler Harmon: No, no, you're getting everything right. It's just the IASO is the name of the company. It's a Greek word.

Etienne Nichols: You know, I'm looking at that, I was like, ah, did I lowercase that or what have I done here? No, Iaso. Thank you so much. I.

Man, Tyler, it's great to have you with us. Tyler Harmon. I don't even think I've said your name yet, but great to have you with us. We were supposed to do this live in Singapore, but I just didn't make it across the world in time.

So anyway, glad to be with you today.

Tyler Harmon: Yeah, so glad we can make it up at the end. Thanks so much for having me on. Really glad to talk about this topic because I know it's going to become more and more pertinent now that AI has kind of entered the zeitgeist as more of a mainstream topic and now, we're kind of seeing it settle more into what it's probably going to be as a steady state.

I really feel like it's important for people to understand how it can exist in products moving forward, how it probably shouldn't exist in products moving forward from a safety perspective.

And then, you know, how we, as medical device developers and inventors and people who work in this industry can effectively and safely use it in the stuff that we work on.

Etienne Nichols: Okay, so there's several things I want to dive into there, and particularly what you said about whether or not they should or shouldn't use it in their product. But before we do, I really, I want to go back in time just a little bit, because you had mentioned when we were doing kind of a prep call about how we really shouldn't be using the word AI.

It's more LLM. And I want to dive back into that real quick, if I might, and I'll circle back.

Tyler Harmon: Yeah, so I'm a big proponent of kind of calling things what they are. And I feel like we have a lot of consumer technologies that are very good and very effective at what they do in forms of things like ChatGPT, Anthropics, Cloud products, and then all the various big tech company systems that they've kind of spun out on their own. But they all are sort of based on the sort on the same core technology.

And I feel like there's a bit of obfuscation that can kind of happen when people don't understand the technical details behind it. And really you have to have a background in this stuff to understand it.

It requires a lot of relatively advanced mathematics. So, my backgrounds in biomedical engineering, so I had to take a, you know, inordinate amount of calculus. And my years at Georgia Tech, which I'm grateful because it's pro.

It's prepared me to understand these topics.

Not everybody's going to have that level of understanding. And I truly believe that we as innovators, if we can't explain things to a more general audience, we generally don't understand them ourselves.

And if you can't do that, it's probably not the best idea to be implementing it into your products.

So, I believe a lot in demystification of this stuff.

And so, we really don't have what I feel like is a true generalizable definition of AI.

And in the modern parlance and the conversations I've been having over the past two years, AI is really just meant large language models. That's just sort of been the bucket term we've been putting it in. And that's because of the interface that people interact with.

These models is mostly through chat Formats and I'd love to do kind of drill into the different types of AI that people are using and how. It's kind of been in the medical device industry for a while, but we're just now seeing certain products come onto the market that are using LLMs and you know that those have their own considerations in play. And there's this whole new technology stack we could be implementing, but we need to be careful and considerate as we do it.

Etienne Nichols: Okay, so a couple of questions because I love that you said that about being able to distill things into more of a general way of thinking. I do remember doing linear algebra and some of those matrix maths.

So, which is helpful for me to think of it from a visual standpoint. But how would you break that down? Because I just think of.

Yeah, let me just stop with the question. How would you break that down into a general audience?

Tyler Harmon: Yeah, so I really like to think about it, really. You know, machine learning, deep learning AI, we've kind of mixed it all and just sort of made them marketing terms from a, from a certain perspective, which is great because it's getting the concepts out there and more in front of more people.

But at the end of the day you're. When I, I'm asked to distill this to like students, you know, who don't have this background a lot of. And I really, you know, shout out to Christian Espinoza at the Blue Goat, you know, a cybersecurity firm, he really challenged me on this when we were in Singapore to really drill into being able to define this.

It's really about using math that's non-linear on linear problems and being able to move things in between the linear and nonlinear space in order to replicate how a person's brain works.

That's just really short and sweet of it. And whether that's using kind of more old school methodologies like random forests or convoluted neural networks, or using newer technologies like large language models, it's kind of the same approach you're introducing this element of non-linearity.

So, things that can't scale or be more generally applicable and separated out because there's that standard linearity test that we all learn in like algebra, right. If you can't turn it into a line that scales, if it can't be applied by breaking into two pieces and adding them together or doing the operation separately,

it's nonlinear, it's really that basic. And the math gets a lot more complicated from there. But I feel like if we understand that you're moving things between simple linear deterministic algorithms and non-linear, non-deterministic algorithms and moving between those two spaces.

That's really all AI is doing at a fundamental level.

Etienne Nichols: Okay, so I want to go one layer, maybe ask one more question about that, because I've talked to some people. They say, well, artificial intelligence hasn't really happened yet because it's not truly the same way a person would.

And is that even a potential from a mathematical standpoint?

Tyler Harmon: Yeah, I think when people talk about that, a lot of the times they're drilling into the Turing Test.

Dr. Turing was a fundamental theorist in the areas of computational science way back during the 1940s. So, the fundamentals of this question we've been asking ourselves as engineers for decades.

And there still are some kind of wiggle rooms around whether the Turing Test is a good test on whether something is a generalizable or artificial intelligence.

But the reason LLMs have sort of become that go to definition of AI in my experience is because of the interfacing and because of the way we interact with them.

Because there I always drill into the user needs. Right. Kind of coming from a medical device engineering background, we're taught to start with the user. We're taught to start with the user needs.

What are LLMs designed around as a user need? Well, they want to engage with people. They want to keep people talking to it for as long as possible because they're consumer products.

That's what their motivation is. They're trying to get people to spend tokens, they're trying to get people to interact with it. Because the very definition of machine learning, the more data you have in general, the better your model is going to get.

Up to a certain point, when you look at those as the reasons these systems work the way they do that, it makes more sense.

I feel like the reason LLMs have become the general parlance is because mathematically they're made to sound like people.

And really there's kind of two pieces of that mechanism I'd like to dive into more. But do you have any questions about that?

Etienne Nichols: I really think that's great. Let's go ahead and dive into them.

Tyler Harmon: So really, there's two components of what LLMs do that kind of help it replicate how a person talks.

There's these underlying mechanisms, and if you dig really deep, all the way back in the 1970s, we had these things called Markov chains, which are effectively next word predictor systems. Exactly the way Google does.

Google's big innovation is that they got really, really good at using Markov chain technology at doing search of large, large information and generalizable the Internet. Right. That was what made Google such a powerhouse. And they've grown out from there.

What chatbot companies have generally done is they've taken some of that same underlying technology,

added a machine learning component so it's constantly learning, reiterating and retraining on itself over time, and then also added this kind of randomness component. So, there's this layer in most LLMs now it's not all of them. Some of them are specifically have designed this piece out for specific purposes.

But in general, they have this component where people don't like talking to something where the answer is always predictable.

So, there's always this layer that's effectively just a random number generator that's saying, hey, how accurate do I want the next word predictor to be? Do I want to throw a curveball?

Because they discovered through their experiments and through their consumer testing that having that next word predictor be something a little bit unpredictable helps keep people engaged because our brains find patterns very, very easily.

And so, if we as a consumer can start predicting that next word ourselves, we, we become less engaged with the product.

And so, from a fundamental user need perspective, they designed in this randomness element. And I think this randomness piece is something that we really need to dive into as an industry because I think that's the component that's going to get us into some trouble moving forward because it's something that isn't generalizable for the medical device industry.

I feel like that's the piece we need to really be talking about and analyzing.

Etienne Nichols: So, what you're saying is that's actually you could design that out? Because I'm trying to think, okay, how do you keep the word predictable? I'm just trying to wrap my head around that.

Tyler Harmon: Yeah. So, at the end of the day, there is some randomness built into the math, right. Convoluted neural nets, even some earlier versions of machine learning and AI, they have the randomness because it's almost impossible to predict directly between linear and nonlinear space.

That's an incredibly difficult mathematical problem that people way smarter than me have been working on for decades. And something I can't claim to fully understand myself, but I know it's there.

I know there's that wiggle because of the linear and non-linearity.

That's why even when we look at image processing systems like Shout out to the folks at Viz AI, they were doing this very early on and very effectively.

They developed a stroke imaging system that got FDA cleared and it helped expedite the workflow of diagnosing strokes in the emergency department and hospitals by doing really effective CNN analysis, convoluted neural net analysis of those images of potential stroke patients.

And that has saved thousands of lives and helped prevent lots of disability cases.

But at the end of the day, the CNN, it can't be 100% accurate all the time. It can't be 100% sensitive and 100% specific because that's just not how the linear to nonlinear math works.

But there's this second component where purposely from a consumer product perspective, there is a layer to the network like the transformer that's actually doing the math that purposefully adds in randomness.

And I do think it's possible to pull that randomness layer out and do things that are more purely transformer based and get those numbers to be less random and to get things closer to the way a more traditional transformer and a Markov chain system could behave.

Now that's something that I have not personally experimented with from a technical perspective. That's something that we kind of have on our roadmap at east, but we're focused more on slightly older versions of AI that really only have the randomness that's built into that linear to nonlinear transformation.

So that's kind of my perspective is that it's really more effective to focus on how can you control that linear and non-linear lack of predictability in a way that makes things safe and effective.

Etienne Nichols: We didn't really talk about IASO that much and I know, I just love to hear just a little bit about what you're doing there. I want to get into the actual how this can all be applied to your structures and your architecture and documentation in a medical device company.

But I'd love to hear what you're doing at Iaso.

Tyler Harmon: Yeah. So, we are developing a first of its kind. AI is a medical device system that is using non LLM models to identify patients that are at risk of acute respiratory distress syndrome.

Now, acute respiratory distress syndrome is a condition a lot of people haven't heard of outside of pulmonology and critical care. But it's effectively at its core the heart and the lungs failing at the same time.

When you look at the heart and lungs as a system.

Because my team and myself, we have a background in biomedical engineering, we just drew a system layer box around the heart and the lungs together and said, okay, how could this be failing?

And it really, that led us to, oh, there's patients who are really suffering from acute respiratory distress syndrome. And it kind of had a baseline level before the pandemic. It really spiked during COVID and now it's come back to a bit of a base layer.

But there's a lot of epidemiological data that says that it is increasing over time now, and for reasons that we don't have quite good epidemiological correlative data to suggest.

But we know that those levels are increasing year over year.

And so, the problem we're trying to solve is that a lot of times it's difficult to diagnose this condition because you really need a lot of expertise.

While there is a mathematical formula, the Berlin criteria, that does give an underlying definition of the quantitative elements.

The qualitative elements of ARDS are very difficult to spot. You normally have to have a lot of experience, and the people who have been doing medicine the longest are usually the best at it.

However, there's not going to be that many people with that much experience for long because we have a generation of folks that are retiring, and we have a whole new generation of, you know, medical professionals coming in that just aren't going to have those 20 years of experience.

And so, what we're seeking to do is to give them some of that experiential ability with a, you know, with an AI tool and help identify the patients are at risk.

Now, we still ask them, you know, to go ahead and validate everything that the AI is potentially telling them. And we're very much so in the R&D phase that, you know, standard regulatory definition of this is not an advertisement for a medical device.

This is just suggestions of a theoretical product.

But essentially what we're trying to do is identify patients that are at risk and otherwise may not have the time to get the treatment they need and pull them into that workflow and that treatment pathway sooner.

Etienne Nichols: Yeah.

Well, that's awesome. I love that you're doing that.

It's interesting when I hear about the doctors who have the experience being able to determine this versus that. I think of that illustration of the thermometer, the hand on the forehead, like, oh, this person has a fever.

And we'll train them to be sensitive enough. But once you have a device, nobody does that anymore. There's no point in training someone to determine whether someone has a fever when you could just immediately tell.

So, I think that's really cool to be that much more accurate. There are.

I love doctors, and I would never discount them in any way, but I think there's. Their expertise can be suited in a lot of different, better ways than maybe we use utilize it currently, so.

Tyler Harmon: Oh yeah. And you know, just for context, Etienne, my, you know, I am probably going to be the biggest advocate you'll ever talk to about doctors need enablement, not replacement. Right.

My, my, my father's a clinician. You know, my mother's was a NICU nurse for 30 years.

I am the last person who wants to pull doctors and nurses away from the bedside.

But I think we need to give them the tools, the force multipliers to tackle the challenges they're going to face this century.

Etienne Nichols: Yeah. And just like you and I as engineers, you know, I was a mechanical engineer giving me solid works. I didn't take that as taking away my job. It's like now I don't have to deal with pins, and I can think in 3D, and you know, we can, you know, blow the doors off this thing now, so.

Tyler Harmon: Exactly.

Etienne Nichols: Exactly.

Yeah. So, tell me. So, I, I love what you're doing. I, I also know that you're doing a lot of things. Maybe can't tell us everything that you're doing, but you've done a lot with AI when it comes to your documentation, your QMS structure.

I'd love to hear more about that, whatever you're willing to share.

Tyler Harmon: Yeah. So, I'm perfectly happy to talk about it. And it's definitely early days. And it really kind of came from the fact that, you know, when I've worked at these companies that are sort of doing the best in field practice.

Right. You know, Apple's one of the great examples I like to bring up because they, you know, even though they're a first in class consumer development company. Right. Most of their products are consumer facing.

They leaned fully into accepting that, hey, if we're going to offer these irregular rhythm testing products and these ECG processing single lead products and these SpO2 products, we need to treat them like medical devices and treat them with that level of care and rigor.

And it turns out that because they were already so high quality, they could just take a lot of those processes and sort of copy paste them over.

So, I believe a lot in learning from other industries. And so, one thing that I've learned from working with developers that are coming from outside the medical device industry, so there's a lot of things that are best practice in software engineering that we can really learn from.

And you know, enabling that, that speed of development, enabling that speed, you know, to testing, enabling automated testing as a part of our QMS structures is really important.

But I also think that, you know, we could also be doing a lot more the AI governance level. I think that, you know, due to how differently these different LLM companies, you know, market their products and develop their products even though they're using the same core technology, we need to be very thoughtful about what that company's core market is and whether it's designed with us in mind as medical device developers.

And so really, we kind of treat it almost as its own vertical within the QMS.

Before I started this company, I was a consultant and most of my work was in helping people who had a software as a medical device or a software as a medical device component get their products onto the market and through the development process.

And because the FDA is really, they try to be very forward thinking, but it's like turning a massive ship.

And so, the guidances around software as a medical device really have only kind of caught up to being truly what software development best practice was for the past 10 or 15 years.

It's really caught up in the past five years or so, which huge credit to the folks there. I know it's a huge effort to try and adjust how such a large agency works, but I think that we're currently a little ill equipped to be dealing with how AI as a medical device is going to behave. And there's great people at the FDA who I've had conversations with, who are really doing awesome work in that space.

But it's going to be something where I think industry has to take the lead on this and we have to set the best example and then work with our regulatory partners to try and set frameworks that help make things safe and effective.

And while that's still going on, I think it's incumbent upon us as developers to try and build that safety and effectiveness framework into everything we do with AI. It's effectively just another software as a service product that we're integrating into our workflows and into our products.

And it starts at the governance and the policy level.

We started with an AI mission statement. This is how AI is going to work in our company.

This is how we're going to think about things. And you know, this may not be right for every company or every organization, but we're a big believer in the human and the loop process.

It can make things less efficient at times. You know, sometimes it's a lot easier just to give an AI a prompt and just let it run, especially for things like, you know, natural language search.

That's something where we don't normally have a human in the loop necessarily, if we're just doing a broad, you know, search of a big amount of data, trying to answer a specific question.

But we do have things we're trying to be responsible by minimizing the amount of token usage we do and things like that, that kind of keep our ethos built into the way we work.

And we're currently in the process of developing things like AI specific SOPs that may live as children documents and you know, SOPs being standard operating procedures for the, for the folks listening that they may operate as kind of like child documents or child procedures and the more generalizable software elements. But we think because of these linear and nonlinear components and also the potential randomness introduced by consumer LLMs that we integrate into our workflows, it's really worth having those dedicated pieces of the QMS and even going into the work instruction level, you know, for the folks who are doing the day to day operational work of what we do, distilling these kind of complicated concepts into a level that helps them with their everyday work, where they may not necessarily have a very technical role in the company, helping them understand how to distill our ethos and the way we want to work into their day to day through work instruction. So, we've almost kind of looked at it as, you know, when you're developing a management system for a company, I think everybody's worked at a company where they have both like a environmental health and safety management system that's its own set of procedures governed by its own set of staff that we as maybe technical folks in the medical device industry, we interact with that system, we understand and we train on it, but we're not necessarily in charge of how that operates. Right. We listen to the professionals that do that within the large organizations.

We have the quality management system that we operate under that sort of our vertical within these organizations. Then there's maybe also a cybersecurity vertical with the information security management system generally governed by standards like SOC 2 or ISO 27001.

And there's great standards being developed around AI systems, but we sort of looked at it as, you know, we need a QMS, a ISMS, and now we need a AIMS or machine learning management system is kind of what we like to call it internally.

So that, that's sort of the framework under which we're working.

Etienne Nichols: Okay, okay. So, I want to ask a question, kind of circling back to the very beginning when you, when you mentioned how a lot of these medical device products probably should not have AI within them.

I don't know if you meant AI or LLM specifically. I'd love you to expound on that and then we can come back to this. Or is this related? I don't, I don't want to go completely irrelevant, but yeah, no, it's related.

Tyler Harmon: It's very much so related. So, I think that when we're thinking about how we build AI technologies, so I think that the real truth we need to face is that we've been building machine learning components into medical devices since the 90s.

It's really well trodden ground if you know how to look. And if you look at the peer reviewed research that's been done, there's lots of great work in developing our product.

We really leaned into systematic literature reviews and things like that. We're developing our user needs.

So, I really encourage people, if you don't feel like you have a good answer, if you don't think you can explain this to a boardroom and to your technical people and to whoever else you got to get in front of, go to the literature.

It does a really great job at explaining this, especially look at things that come from outside purely medical research.

It's a really great guidance.

And I think the thing that we need to scrutinize though is that just two weeks ago there was a product that went through the 510(k) process and got cleared that it was really our first LLM based medical device and it was advertised as the first AI MD device.

When you dug a layer deeper, it integrated what AWS's LLMs do, but it was integrating the Alexa systems, not necessarily the newer quote unquote, frontier models. And once again, another term we don't have a real definition for.

The way I've started talking about frontier models is if it's something you developed yourself and you have kind of your own closed loop, your own island definition of what that model is, you're really working on a frontier model. So, you know, I, I have started telling other developers like, hey, if you're building your own model, if you're not leaning into being a harness, which is really just a system that sits on top of that LLM, then you're developing a frontier model.

And I think there's a lot more of us that are doing frontier model work out there than we really give ourselves credit for.

And I encourage other medical device developers lean into that because you're doing something cutting edge. And the folks that developed this device that was sitting on top of the Alexa system, it, it does something really, really important. It Helps people interact with the information of how they should change their insulin dosing purely by having to talk to their Alexa. Maybe it's all it, you know, reduces a telehealth call to a NP or a clinician, or it just gets them that information faster when they're, you know, maybe not feeling so well. That's a very important function. I'm very glad this product's on the market. I'm not going to say as a blanket statement that, you know, LLMs should not be in medical devices at all, but I think there is a layer there where there is kind of the backstop of, okay, well, if this person's feeling really unwell, they probably know to call their, you know, their doctor. Right? If it's someone who is a type 1 or type 2 diabetic, this is a condition they've lived with for a very long time.

And so, you know, there's a certain level of user knowledge that acts as a backstop, as a validation for some of that randomness that's built into the system.

And Alexa, I wouldn't call it so much a true frontier LLM because it has been in existence for a while and I don't know its underlying architecture, obviously, because I'm not.

Never worked at AWS, you know, never worked directly with that system.

But I know that it's more.

It's more based on transformer technology, sort of closer to that CNN technology, from what I understand, than the more modern frontier models like ChatGPT or Claude.

And I think when we're building out these layers to do specific functions on top of ChatGPT's product suite or Claude or Gemini, anything where you're pulling from someone else's tech stack.

And I think this is what this company's done that just got their 510(k) cleared. You really have to treat it as software of unknown providence, as suit, which once again, we've got great guidances around, we've got great methodologies. It's been in IEC 62304 for nearly a decade now, I guess more than a decade since the 2015 kind of amendment that was put out.

And so, we have a framework, we know how to do this. I think we just kind of have to get a little better at calling a spade a spade in this space.

And so, you need to really lean into the risk management processes that we understand.

And look at, hey, can I get access to a level of code or a level of this model where I can engineer out certain elements of this randomness to make it be Safe and effective for my product and for my use case.

And so, you know, I think where we get into trouble is when we bolt on harnesses on top of the LLMs and then treat the LLM as the product and we act more as distributors than developers.

I think that's kind of a gray area that's developing an AI as a medical device or kind of the doctor as AI space, which is, you know, something that I think we should be very cautious about as an industry.

Etienne Nichols: That, that randomness is difficult for me to. Because I, I, I agree with you. Everything you said, I was trying to think. I'm like, do I really agree with you? I, I think I do. As far as everything being just like soup software of unknown providence, it's just software.

It's just maybe riskier software.

But the, the level of risk I think that is released once we have an AI in the field, how does that work versus a traditional device once it's released? If, if that algorithm is not locked or, you know, just a closed algorithm, how does that work?

Tyler Harmon: Yeah, and I think it's very interesting because the, the way these, our relationships to these LLM companies develop is going to be, you know, it's going to be fascinating over the next couple years enough an area where we have to be very conscious because at the end of the day, if you're just saying, hey, you know, we are a harness that sits on top of like ChatGPT, you know, whatever version of it you're putting out there.

At the end of the day, unless you have a specific box soft version that that company has said, hey, we're giving you a specific license for a specific version and we won't change it and we won't or we will revalidate it for you to certain specifications.

But those supplier relationships and treating it as a supplier quality engagement is really, really difficult from the perspective of they're a massive company and a lot of folks that are doing these AIs, medical device products are, you know, work are more in the startup space.

And I know that there are some folks’ kind of at the strategic companies that are doing some interesting work in that, in this space and they have the relationships where they can go to an OpenAI or an Anthropic and say, hey, we want a specific model.

We will pay you this price to ensure it's managed a specific way.

You'll act as a vendor under our system, and they have the resources to manage that.

But we as startup companies, we may not have that level of pull or level of Engagement because we can't offer that pricing, right, we can't offer that amount of capital in exchange.

And so, I definitely think that it's something that can be done, but you kind of have to treat it as a black box.

We've all been taught from an engineering perspective, black box development, white box development, where you can see what black box is, you're passing in information, you're getting out a certain outcome, it's relatively predictable.

Right.

But we just don't understand how the inner components of it work. And that's how when you're integrating SaaS elements, even like AWS, at the end of the day, these microservices that we're integrating into cloud based products and there's lots of really good, really well made cloud based medical device products out there, but you're having to treat them as a black box because AWS is a vendor that you may have management agreements with them, you may have, you know, certain supplier quality agreements with them, but at the end of the day they have lots of customers and they may not be able to dedicate, be able to dedicate a certain resource to you.

So, I'm actually kind of reeling in anticipating your next question here at the end and you can correct me if I'm veering off.

I'm actually a big proponent of smaller language models and bringing those inside the white box components of what we do as companies.

Because when it's no longer a vendor relationship, when you can take open-source components that have open licenses and can be used, however we need to pull them into our systems, treat them as white box development where we have more control over how those layers behave.

Because like I said, there's no engineering the randomness out of it.

Etienne Nichols: Totally.

Tyler Harmon: I don't think that's a problem that can be solved reasonably within the next few years.

That's just not how inference works.

But when we pull them in and we white box the systems and we manage them as small language models, where instead of trillions of inference nodes, you have maybe a few billion, which I know sounds like a large number, but it's a much smaller scale LLM, when those are more manageable, we can also manage the pre-training and post training of the system.

So that's another component in addition to the linearity and non-linearity and the randomness layer, there's, you know, there's the inherent component of good data in, good data out, where whatever you're training the system on is really important as well. And when you pull it in as a small language model, you're more able to manage how it's trained.

Etienne Nichols: Yeah. What about the. I'm trying to remember what the wording is for this. When you train on a data set and it gets so used to that, then it applies it.

Yeah. Could potentially start not working very well on real world data.

Tyler Harmon: Right. So that's when you, that's when you. I would call overfitting.

That's right. It's a very large risk. Whenever we are looking at, whenever we're looking at, you know, especially transformer-based technologies, you know, things like convoluted neural networks, that's a big risk. And really at the end of the day, anytime you're using synthetic data, you know that that's a huge risk under inherently because what happens is if you're telling it to generate a bunch of information, there's probably actually a machine learning component helping, whatever your synthetic data tool to generate all that information. And so, it's going to insert some randomness or if there's no machine learning component and it's completely determinative, there's no data set out there that's completely clean and perfect.

And so, if you're pulling these, even these open-source data components, there's things like MIS entered variables, there's things that just come from human error of doing things over and over again.

And when you're getting different kinds of errors from your synthetic data sets, it can lead to all kinds of problems.

I think you have to be really self-scrutinizing and looking at okay, am I overfitting? Am I getting a very high sensitivity and a very low specificity for the outcome that I want it.

That was something that we've been very careful about in our development process at Iaso is are we using too many variables? Are we using too few variables? Are we overfitting the system?

And we frankly have had to lean into folks, our team members who don't have a pure medical device background or are coming from intensive computer science and software engineering backgrounds to help us understand because it's just not something that I've had to learn in my professional experience.

So, you know that overfitting, it's a problem that can occur just in machine learning and AI products in general, but it's especially prevalent with synthetic data. And so, we, as a part of our policy at Iaso, we do not train on synthetic data ever.

It is never entering our training pipeline for as much as we can possibly help it.

I hope there never comes a day where I have to write a CAPA around that, but that's our intent and what we're saying at the policy level.

Etienne Nichols: So, there's a couple things I want to talk about, and I don't know which you tell me where you want to go. I guess one of the things is you mentioned adding those AI specific documents for your QMS, and I'm curious how you can do that, add those documents without it being just documentation for documentation’s sake, actually being effective.

The other thing I'm really curious about is accountability.

And if there is that randomness that we're inherently, from a mathematical standpoint, never going to completely eliminate from the system, or at least not in the, in the near future, how does that work with accountability and the potential for errors? And I think I may be anticipating the answer, but I don't know which one you want to tackle first.

I just want to make sure I don't forget one.

Tyler Harmon: Well, I think, I think we can talk about both. I think there's a two birds with one stone here situation at the end because I think that that underlying problem is sort of the root cause that we're trying to address.

Right. You know, sometimes when we're doing a CAPA in the general medical device space, we do our five why’s and we try and get to the root cause of something, and we address that root cause.

And sometimes that root cause is just retraining, or the corrective action, I should say, is just retraining on certain things. Sometimes it's writing whole new procedures. And so, we kind of took a step back and we looked at sort of an industry CAPA from the perspective of, hey, we don't feel like we have the right context to manage this with the existing tools.

And so, I definitely, I've been in the analysis paralysis situation for. Believe you me, when you get organizations that operate all over the world, it's incredibly easy for every office to want to do it exactly their way, exactly their way of thinking.

And you end up kind of with just these paperwork churn exercises.

But, you know, in order to avoid that, I think you have to look at, okay, what is the actual problem we're trying to solve?

And so, when we're looking at our AI SOPs and our AI policy documents, we're saying, how are we solving for this randomness component that may not be as prevalent in our other soup components or in our other SaaS tools that we're using within the organization and how that manifests?

Right, because that randomness component can manifest just as hallucinations. Because part of what AI LLM systems do is they're made to just say, hey, I agree with you. Hey, what you're saying is good because people in a consumer setting, they don't want that friction.

We don't like and arguing with each other necessarily.

You know, Etienne, you and I have very many friendly disagreements, but that's because we have a similar background and have this dynamic. You can't have that with everybody. Right. A chatbot doesn't want you to have that with it.

And so, with that under consideration, we tried to design out, well, if it is hallucinating, how do we check for that?

Generally, that includes adding a human in loop component and understanding what's the risk of the probability of that happening. Like for instance, if you're using a tool that is specifically designed, shout out to the folks at site AI, they have a great product, but it's basically a series of system prompts and components and harness elements built on top of Chat GBT's products. Right? And now a lot of companies are building out these MCP components to their products.

They're no longer depending on just one model and they're trying to pull from whichever model is performing best.

I think that's great. I really encourage vendors to do that. But we will certainly still lean into as an organization having the human and loop component as a lot of what our SOPs and our work instructions tell us to do.

That that's kind of going to, that's kind of probably going to be a large chunk of what our SOPs are. Hey, do everything you would do for a soup product being, you know, used in a SAMD context.

So, referencing the parent procedure policy, but consider the hallucination risk.

Consider adding human the loop to the process.

Consider what the organizational and the product risk are in this process.

The computer systems assurance guidance from the FDA was really a state change in how we look at managing software within these organizations. And I think that leaning into that framework, but adding the context of, hey, what is the actual risk case of this product? Having that really well defined in our QMS systems and then also understanding that everything that comes out of LLM, no matter how good of a relationship you have with the vendor, you have to treat it as a black box soup system and manage that however you, you generally would.

Etienne Nichols: Yeah, okay, so let's, let's go back to some of the documentation around that though, that I, I get what you're saying as far as all of these things are important to have that as an ethos for your company.

Maybe one of the things that I'm curious about is how do you make sure that the words on the paper are implemented, especially when things are moving so fast. How did the, how does everybody really make sure that is ingrained in the actual actions of your company as well?

Maybe this is an age-old problem with documentation, but yeah, no, it's definitely an age-old problem. But I think it's something that deserves re-discussion at this very point in time. And part of how we've managed it as a company is we have a diverse set of backgrounds in our company.

You know, we've pulled from people who have done, you know, more academia in the software engineering space. We pull from people who worked in the consumer space so that we have those different perspectives and so that they can kind of help push back and really listen, really listen to your people when they say, hey, I don't know if this needs an SOP or work instruction because I'm not adding process or product risk to what I'm doing.

And you know, people who've worked with me in consulting capacity know I'm always about doing more with less. I'm always about the, the risk of analysis paralysis and just doing kind of what, what I call recursive iterative QMSing where it's really tempting when you've gotten a warning letter or you've gotten a recall to just build in layers to your SOP to make it more rigid.

I think that over time it's really easy for that to happen. I think that's something that we don't see as much in the startup space, that maybe is more prevalent in the strategic space.

But yeah, I think listening to folks with those outside perspectives is a really important element in how we are going to be implementing it at Iaso and then be willing to say, hey, maybe this is not a QMS process and maybe it doesn't even go through our CSA workflow like natural language search.

What I do as an executive every day to try and help me culminate bits of information that's not really a QMS process unless I'm taking that information and trying to claim, oh, this is a part of a systematic literature review that I'm doing, or oh, this is something that's going to go into a user need or a product understanding those are QMS processes when you're.

Because those can trickle through and impact the safety and effectiveness of the device. But things that are more of admin functions, that are operational functions or they're just things that help things move faster.

Right. Like my software engineers, I don't Tell them, hey, don't use cloud code. I tell them, be mindful of our ethos and be mindful of what you're doing. And then when it comes back from the R&D space and into the product development space, that's when it's going to interact more with the.

With the QMS.

And, you know, shout out to my chief quality and regulatory officer, Ladavia Flournoy Fowler, who you. You met at LSI Dana Point.

She is incredibly good at understanding that line and guiding us around that line. And so I, I lean into her experience a lot.

Etienne Nichols: Yeah, we had her on a virtual summit. It was fantastic. She's really great at what she does. So.

So, I. Okay, so we've covered a lot of ground. I know we've talked about maybe doing a few different episodes on this. And those of you listening, if you want to hear more from Tyler, I feel like he's a wealth of information in a world where we.

It feels like maybe AI is making us dumber somehow. You. You are. Have, man, held on to your IQ points, which is awesome. And I also wanted to know, maybe a personal curiosity question.

How do you personally interact with LLMs or do you. Or what's your. Yeah, what's your.

Tyler Harmon: What's going on there?

I'm very open to the fact. To the fact that I was a bit of a late adopter in that space, you know, kind of borderline Luddite, because I, you know, introducing that component of randomness for me was something that really didn't sit with me very well as an engineer, because I like to engineer in precision. I like being precise.

I don't like, you know, kind of leaving things up to chance from a certain perspective. And also, when I interacted with it very early, like the vert, you know, the models that were coming out back in 2023, 2024, there was a lot of randomness because they were really focused about, like, hey, how do we keep people engaged so we can get more data so we can make these models more effective? Which. That's fine as a business plan. I have no arguments there.

I think from the executive perspective, I totally get why they went in that direction because they're consumer products.

But when you're working in the engineering space and the medical device space and any other space that's highly regulated, you kind of have to be very careful with that.

So, I'm really just. Just now, in the past few months, you know, have started listening to my team and say, hey, if you're generating content, if you're if you just need to generate information or if you need to collate information very quickly.

You know, I was kind of old school. I was going through PubMed advanced search and like saying, you know, ARDS plus COVID 19 plus, you know, pulmonary plus Mayo Clinic, like just giving it these really, really deterministic, you know, keywords.

Because I was trying to, you know, use the old methodologies of search.

Now I use it for a lot of natural language search problems. I do try and minimize it. And I don't use systems that are actively charging as tokens because once again, those are sort of opaque on the back end and token costs can fluctuate.

And I'm a man on a budget at the end, I'm running a startup, trying to run it lean.

So anytime I don't have a predictable cost, I try and lean away from that. But in my experience, I've really leaned into using Gemini as a natural language search tool, partially because we are a Google suite company and I've always really liked Google's products. And in my conversation’s kind of casually with folks at Google, I feel like they have a really good ethos and a mindset around how they're developing these tools.

So not being a first mover is kind of how I approach it and then using it for very specific use cases that I can justify to myself and to my board and to my exec team.

Because once again, we have a variety and diversity of opinions.

We have people on our team that have said, hey, this is great for what I do, I want to use it all the time. Please give me as much access to these tools as possible. I need access to several tools.

There are some people on our team that say, hey, this doesn't really fit what I do day to day. And unless you guys tell me that there's a definitive use case, I'm going to lead away from it.

I think all those perspectives are valid based on the kind of work we're doing.

From a personal standpoint, I'm just now getting into the natural language search. I do a little bit of content generation when it comes into building slide decks, but I'm always conscious of the fact that this thing could be giving me randomness, it could just be giving me a search result that I kind of agree with. So, then I dive a second layer in and say, okay, if it's telling me all of these things, what is the primary sourcing on this information? And I'll tell it specifically, hey, when you're giving me an answer on an academic question, pull from PubMed, pull from Medline, pull from Google Scholar, pull from various known online databases that I've been using for all of my time in engineering ever since undergraduate and have a good understanding of how those work and kind of go, you know, use it as a, as a collater.

But then treat it almost as you would a systematic literature review that's coming back to you. Dive into the individual sources on the questions you have and use that as a secondary layer.

Etienne Nichols: Yeah, totally agree with that. And I was, I think I was a little bit shocked when I found out not everybody reads all the references or the sources and I just assume if I say anything that someone is going to look up the source and if I didn't look it up already, then you know that that just makes you feel terrible.

Tyler Harmon: So, oh, definitively. And I, you know, I've gotten caught with, you know, hey, oh, this source looks reputable but then you click through it. You know, I've even gone so far as I've told Gemini, hey, give me the, the DOI link for, for, for a actual search. Don't link me to the dad. Blame Google search.

Yeah, to an actual thing and it will say, it'll give me a URL that's a DOI link and I click through and I have to say bad. I know you gave me the Google search again and now I had to go try and find where I have access to this article, where do I have an academic relationship that allow me access to this article.

And it's a whole layer where it's just like Dadgummit. If you had just had the non-randomness component to this AI product specifically for what I was doing, which is why I called out Site AI.

That's a tool I've been using, and they have specifically tried to engineer so that it's giving you those direct links. And that's something that is really useful because it keeps me from having to prompt Engineer every time I go into a chat to say, hey, remember, don't ever do this.

And it becomes a piece of work I don't have to remember to do and can make it a relatively mindless thing that I don't have to think about.

Etienne Nichols: So, you said site AI. Is it sciteai.com or S S I

Tyler Harmon: S C I T E A I. Yeah, okay.

Etienne Nichols: Okay, I'll check that out.

I wanted to say one thing about Google. I know we're basically going to getting to the point where we need to wrap this up, but a lot of people, at least in the marketing side of Things, which is where I am now, you know, move from mechanical engineering.

Everybody moves some somewhere I suppose but there was some question about Google search. Are they cannibalizing themselves? You know, their Google search is going away. How are ad spends going to be?

You know, it's a dumb company. What are you thinking now?

But turns out the way they've done things, my understanding now is their ads are worth more than ever at the way they, for them anyway. They've, they're making more money off that ad spend than ever before.

And so, it's interesting to see how the company is applying, you know, these tools and we, I just, just kind of another plug to learn from other industries.

Tyler Harmon: Yeah, for, for sure. And you know it. I, I think they're very thoughtful in what they do. I've never had the pleasure for working for them directly, but you know, having gone to Georgia Tech I have a lot of friends, a of lot, a lot of ex colleagues that are working there and these are very thoughtful, smart people that I can kind of personally vouch for and I feel like what they're doing is really, really smart. And I think that because of the way that I've seen Google kind of move with the Pixel watches, sort of slowly integrating some of the similar components and kind of taking Apple's cues and being second movers in that space in terms of how to integrate CMD into their hardware, hardware products, I think they're, they're Sharps in this space. I think, you know, at the end of the day when you're dealing with randomness, you're doing a little bit of gambling. Right. And not, you know, not to, not to reveal the fact that I like to play cards, but there's, there's Sharps and there's Marks and there's people who know what they're doing.

There's people who kind of don't know how to manage the randomness. And I think that you know, personally, in my experience the folks, Apple and the folks in Google are really Sharps in this space.

Etienne Nichols: Yeah, very cool. Well, any last pieces of advice? Whether it's how people can find you or what you recommend. Any pieces of advice before the, before we shut this down?

Tyler Harmon: Oh, you know, I, I'm relatively findable. You know, I'm really mostly on, just on LinkedIn, you know, a couple of really done, really well done photos that make me look better than I should from the LSI conference on there.

But then also, you know, iasoams.com if you want to learn a little bit more about the work that we're doing.

And I think, you know, I'd really encourage people that as much as these products seem hyper complex, as they feel insurmountable, as we're kind of deluged with information about them daily, and it kind of feels like they're being forced on us by being integrated into the consumer products we use every day,

they aren't beyond your understanding.

You know, anybody who's taken algebra, and even if you haven't taken algebra, you can go learn this stuff pretty, pretty quickly. Right. I'm a big believer in there's no wrong students, there's only maybe wrong methods and wrong teachers.

And so, you know, figure out your way of learning, try and dig into this stuff, you know, lean and if, and if you don't know, find someone in your, in your life to ask, you know? You know, to ask.

I, I have, as an executive, you know, as a first time executive, you know, I, as a first time CEO, I've really had to learn that skill of like, hey, I went from dealing with the, you know, technical problems every day to dealing mostly with investors and board members and all these sorts of things.

And it's a different skill set.

So, you know, I've had to kind of have the humility. And it's something that my, my executive team is very, very diligent about is helping me. That's why I pulled people that I trust onto my executive team, is to help keep me humble.

And I think that's that it's a learned skill that I wasn't always very good at and I like to think I'm getting better at. So, you know, be curious, be humble, don't feel overwhelmed because you can understand this.

Tyler Harmon: Yeah.

Etienne Nichols: You know, I used to have a phrase that the opposite of fear was not bravery or courage because you have to have fear in order to be courage, but the opposite of fear is curiosity.

And I think when you're curious, you forget a lot of that fear. So, yeah, I will echo that definitely be curious.

Tyler, this is great. Really appreciate it. Thank you so much for coming on the show. Thank you. I hopefully we'll have be able to have some more conversations in the future.

You feel like a wealth of knowledge every time I talk to you. I feel like, man, that wasn't long enough. But I look forward to the next time we get to have a conversation.

Thank you so much. Those of you listening, thank you so much for listening to the Global Medical Device podcast. We will see you all next time.

Thanks for tuning in to the Global Medical Device Podcast if you found value in today's conversation, please take a moment to rate, review and subscribe on your favorite podcast platform. If you've got thoughts or questions, we'd love to hear from you, email us at podcast@greenlight.guru.

Stay connected for more insights into the future of MedTech innovation.

And if you're ready to take your product development to the next level. Visit us at www.greenlight guru. Until next time, keep innovating and improving the quality of life.

About the Global Medical Device Podcast:

Untitled (8.5 × 3 in)

The Global Medical Device Podcast powered by Greenlight Guru is where today's brightest minds in the medical device industry go to get their most useful and actionable insider knowledge, direct from some of the world's leading medical device experts and companies.

Like this episode? Subscribe today on iTunes or Spotify.

Etienne Nichols

Etienne Nichols is the Head of Industry Insights & Education at Greenlight Guru. As a Mechanical Engineer and Medical Device Guru, he specializes in simplifying complex ideas, teaching system integration, and connecting industry leaders. While hosting the Global Medical Device Podcast, Etienne has led over 200...