SaMD Issues, Defects & Detection with Shawnna Monterrey

Updated August 11, 2026 ░░░░░░

#468 SaMD Issues, Defects & Detection Shawnna Monterrey

Most discussions around medical device quality stop at commercial launch. Once a product ships, teams tend to celebrate and move on to the next development cycle. However, the real engineering work often begins the moment a device leaves the manufacturing floor. In this episode of the Global Medical Device Podcast, host Etienne Nichols sits down with Shawnna Monterrey, founder of Beanstalk Ventures and an FDA-accredited third-party reviewer with 25 years of medical device software experience, to explore what happens after product deployment.

Monterrey shares rare insights gained from evaluating FDA submissions and troubleshooting high-impact field issues across platforms ranging from glaucoma imaging at ZEISS to CTDNA cancer assays at Illumina. The conversation covers the often-overlooked requirements of manufacturing transfer, deployability, and software upgrade mechanisms. Monterrey explains how inadequate upstream characterization—such as neglecting physical shipping stresses or omitting subsystem-level DFMEAs—directly manifests as costly "dead on arrival" (DOA) failures and field complaints.

The discussion also dives deep into the mechanics of defect detection, comparing hardware tolerance stack-ups with complex software root cause analysis. Monterrey illustrates how robust unit testing, clear design documentation, and structural post-market surveillance prevent catastrophic field recalls. Finally, the episode highlights the critical need for open communication channels between R&D, manufacturing, and post-market complaint handling teams to feed field intelligence back into future product iterations.

Watch the Video:

Listen now:

Love this episode? Leave a review on iTunes!

Have suggestions or topics you’d like to hear about? Email us at podcast@greenlight.guru.

Key Timestamps

00:00 - Introduction to Etienne Nichols and guest Shawnna Monterrey, CEO of Beanstalk Ventures.
01:15 - Crucial pre-shipping checks that first-time medical device founders routinely miss.
02:05 - Software transfer to manufacturing, deployability, eStar submissions, and cybersecurity requirements.
03:10 - Root causes of "Dead on Arrival" (DOA) product deliveries and shipping reliability testing.
04:20 - The concept of injection detection: Why detecting bugs earlier in R&D saves exponential costs.
05:45 - Unanticipated failure modes, software-hardware interaction, and the necessity of bottom-up DFMEAs.
07:30 - Software defect isolation, unit testing vs. system-level troubleshooting, and simulating user environments.
08:15Etienne Nichols: - Case study: Class 1 ventilator recall, software algorithm flaws, and root cause analysis across 80,000 units.
10:40 - Field upgradeability, patchability in legacy firmware devices, and managing regulatory trade-offs.
12:15 - Transforming customer complaints from isolated fires into upstream process and product design improvements.
14:00 - Usability issues, off-label user behavior, and manufacturer liability regarding indications for use.
16:30 - Closing feedback loops: Structuring open communication between R&D, post-market teams, and field service.

Top takeaways from this episode

Prioritize Software Deployability Upstream: Under current FDA eStar submission standards and cybersecurity guidance, software deployment, upgrade mechanisms, and maintenance processes must be documented and tested well before shipping.
Execute Bottom-Up DFMEAs: While FDA risk management emphasizes top-down system hazard analysis (ISO 14971), robust subsystem-level DFMEAs are essential to capture unexpected interaction defects between electromechanical hardware and software.
Unit Testing Accelerates Root Cause Analysis: Simulating inputs via automated software unit tests allows engineering teams to reproduce obscure field defects instantly without needing to replicate complex human-patient variables.
Design for Field Upgradeability: Building patchable, field-upgradeable firmware and software architectures protects device manufacturers from catastrophic physical recalls across large installed bases.
Bridge R&D and Complaint Management: Companies must establish formal feedback channels between post-market complaint handling teams and R&D engineers to ensure real-world failure trends drive future design controls.

References:

Etienne Nichols LinkedIn Profile: https://www.linkedin.com/in/etiennenichols/
FDA eStar Program: The FDA's electronic submission template used to streamline medical device 510(k) and De Novo review processes.
ISO 14971: The international standard for the application of risk management to medical devices.
Cardiac Arrest: Five Years as a CEO on the Fed's Hit List by Howard Root: Recommended book detailing off-label use, regulatory enforcement, and legal liability in MedTech.

MedTech 101 Section

Injection Detection Think of building a medical device like baking a cake from a recipe. If you accidentally add salt instead of sugar at the start (injecting a defect), it is easy and cheap to toss out the flour and start over. But if you don't taste the cake until after it is baked, frosted, packaged, and delivered to a customer's party, fixing that mistake requires shipping a whole new cake, apologizing to the buyer, and paying for delivery. In MedTech software and hardware, "injection detection" means testing early and often so you catch design "bugs" while they are still in the mixing bowl rather than after thousands of devices are in patients' hands.

Design Failure Mode and Effects Analysis (DFMEA) Imagine examining every individual part of a car engine—from the biggest piston down to the smallest rubber seal—and asking: "How could this specific part break, and what happens to the driver if it does?" A DFMEA is a systematic, bottom-up engineering blueprint where teams evaluate each component or software line to predict failures before the device is ever built.

Memorable quotes from this episode

"The sooner a defect is injected into the product and the later you find it, the more expensive it is going to be to correct. You want to tighten that gap up as close as possible." - Shawnna Monterrey

"A lot of defects manifest themselves in software, but they are actually electromechanical issues that the software didn't intend to catch." - Shawnna Monterrey

Feedback Call-to-Action

What post-market challenges has your medical device team encountered after product launch? We want to hear your thoughts, topic requests, and guest suggestions. Send your feedback directly to podcast@greenlight.guru. Every message is reviewed personally by our team to help shape future episodes.

Transcript

Etienne Nichols: Hey everyone, welcome back to the Global Medical Device Podcast. My name is Etienne Nichols. I'm the host for today's episode, and today, I want to talk about what happens after launch? Most conversations about product quality stop at launch. You ship, you celebrate, you move on, but the real work starts the day your product leaves the building.

And so, my guest today has lived every side of that. Shawnna Monterrey has built medical device software for 25 years, from glaucoma imaging at ZEISS, if I'm saying that right, to CTDNA cancer assays at Illumina. And today, she runs Beanstalk Ventures, and through Beanstalk Consulting, she's an FDA-accredited third-party reviewer. If you don't know what that is, we have another episode where we talked about that.

But that means she now reads other teams' submissions and sees the same failure patterns over and over and over again, similar to what the FDA…I mean, she is seeing what the FDA sees. So, I wanted to walk through this whole loop of shipping product, detecting defects, and so on. And how to evaluate those customer complaints, and then the hard part, which is reading those signals backwards to what you needed upstream, to what you should have done to the beginning, and forward that to what people actually expect downstream. So, Shawnna, welcome to the show, glad you're here with us.

How are you doing today?

Shawnnah Monterrey: I'm doing great! Thanks for having me.

Etienne Nichols: Well, I don't know how you want to start exactly. Do you want to talk about shipping product? I mean, you've shipped everything from…imaging instruments to cancer screening essays, now we're into software and full digital health platforms. When you say something's ready to ship, what are you checking that first-time founder wouldn't think to check? What are some of the things they don't ever think about?

Shawnnah Monterrey: Well, a lot of things have to do with, transfer to manufacturing, right? So, when you're looking at an instrument, there's a process to get the device, as you built it in R&D, transferred into manufacturing, right? So, a lot of things that are missing upstream will be, can I scale this device, and can I manufacture it? But when you look at it from a purely software perspective, a lot of times that part gets skipped, because technically there is no manufacturing.

I would say that's somewhat true, somewhat not true. I think when you look at, you know, software, you need to think about the deployment mechanisms and whatnot. So, a lot of things that get overlooked at that stage, this is, like, pre-shipping to the client, is basically how do you assemble and build the actual device itself, and ultimately get it deployed? And so, a lot of things from an instrument perspective that are overlooked when submitted the FDA's, manufacturability requirements, service and maintenance requirements, calibration requirements, like, things that really impact the product that should be integrated into the design. And so, FDA's not looking at that, because those are solely considered servicing and manufacturing tools.

Right, and so they get overlooked, and then they get done later, and then you find out that you have problems in manufacturing. So, problem number one is you can't even get it shipped to clients, right? Because you have upstream manufacturing issues that should have been addressed much sooner.

Same with software. So, software, a lot of times the, and this is no longer true with the current eStar submission. FDA is looking at deployability of the software. They care about the software upgrade process. They want to, see the upgrade and maintenance process associated with software, so that's new.

Yeah. It was covered before, but I think touched on lightly, but now with cybersecurity, it's a big focus point.

And so really understanding how you're going to actually document the build and deployment process, and how you're ultimately going to then deploy it to the end users is sometimes overlooked as well. Thinking, oh, we'll figure this out later, but it actually impacts the product and how it's used, and ultimately how it's tested.

Because at the end of the day, you want to test the product, once it's manufactured and deployed to the client in the same way, that they're actually going to use the product. And a lot of times it's overlooked. And so, what you'll see is, high first pass failures. So, arrival, I think it's called dead on arrival.

Etienne Nichols: Okay.

Shawnnah Monterrey: We would call it DOAs. Yeah.

Etienne Nichols: Oh, man.

Shawnnah Monterrey: So, once you start shipping your device, you get DOAs, generally you can trace it back upstream. And so, it's not just, an oversight in manufacturing deployment, it could be just even an oversight in how you tested your product. You weren't simulating the real user environment appropriately, or you overlooked stuff that can happen in the shipping process.

Or things that could happen, like, when they first deploy the software.

Etienne Nichols: So, there's a couple things that it sounds like we're talking about. There's, well, two specifically. Ones are the issues that will get you when it comes time to submission.

That you maybe never thought about till the day you're submitting, but then the other thing is, after you're submitted, maybe you got cleared, and then this other thing is…you didn't think about is going to cause that dead on arrival situation. What are those things? Like, you got cleared, but you should have done this for the reality check, because I think that's one thing a lot of people aren't thinking about. What are some of those things?

Shawnnah Monterrey: Well, a lot of things dead on the rival. If you look at, like, software and instruments, a little bit different, right? But if you have software running on an instrument, then you have compounded you know, potential issues. A lot of things for dead on arrival is things that happen during shipping, like not doing appropriate shipping, reliability testing. Other things that happen are things when the device actually sits on the shelf.

So, like, shipping, for example, let's say, I don't know, you don't have enough torque on a screw, right? When you're doing an R&D, right, and then you transfer manufacturing, maybe you didn't specify the appropriate amount of torque that needs to be applied to the screw. Something as simple as that, then the screw gets dislodged, or loosened.

And then there's a failure because there's a component of the device that's loose or shaky, right, when the device is now being used. So, it could be things as trivial as a screw.

When we talk about instruments, and again, that has to do from an R&D perspective, transferring that knowledge into manufacturing in the areas that's important. And something like a screw you may not think about in terms of impacting essential performance, and it may not impact essential performance, but it causes a dead on arrival where you can't even use the device, potentially, you know, in a customer complaint.

Which are very costly to address and to fix when it happens in the field, right? So, they have this kind of philosophy where you…it's called, injection detection. I forget where it's derived from, but basically what you want to do when you're in R&D is you want to basically, any failure or bug, right, that gets injected into the device, you want to detect it as soon as possible. The later you actually detect it.

Is where you have a higher cost to correct.

Right? So, it's…Yeah, that makes sense find an issue out in the customer field, right, and it got injected during R&D when you were actually coding or designing the product, then you have to go all the way back to design.

If it happened, like, let's say in manufacturing, now you're just going to go back to manufacturing, right? So, the sooner it got injected, and the later you find it, the more expensive it's going to be. So, you want to tighten that up as close as possible.

Etienne Nichols: Well, what are some defects?

That show up again and again, that team. And actually, before we move on to that, I…when you mentioned that screw, I mean, I started getting some…a little bit of trauma coming back, because the…the worst CAPA situation I was ever involved in was with a screw and a helicoil. You know, it was like…it was almost superfluous, why did we even do this? But, but yeah, so anyway, there's…I totally agree, you know, just all of these…these things, like, 3 threads for 95% strength, you know, someone is gonna correct me on this, but it's been a while, but…

Shawnnah Monterrey: Oh, I'll get corrected as well, I'm sure.

Etienne Nichols: Yeah, but it's just so interesting, just the things that can fail that you just don't think, you know, I mean, there's so many…which is why it needs to be an iterative situation, where all that information coming in the field update…is updating your design controls and risk management, and actually causing improvements to the product.

Without going too far down that rabbit trail, although if you want to, we can, detection is interesting, because you read so many team submissions as a reviewer, and then you see them afterwards as well, so what defects show up again and again that teams just don't seem to be able to see in their own products?

Shawnnah Monterrey: Oh, gosh, well, I wouldn't say it's defect-specific, right? So, I would say it's more where the team under-antis…didn't anticipate a failure mode.

Etienne Nichols: Hmm.

Shawnnah Monterrey: Right? And so, they're generally going to be failure mode-related defects, and if you go back and trace back to see what happened, the gap's going to probably be in the DFMEA.

The requirements, and or the testing. So, you see in some areas, there's…it's under SPAC.

Right? Or if it was well SPAC’d and just not appropriately characterized and tested, right? Or it was just a misfailure mode.

You'll see a lot of defects that manifest themselves in software.

But they're actually generally not software bugs, they're usually electromechanical issues that the software didn't intend to catch, right? So, let's say it's reading a signal, high and low, and it gets a no signal, and maybe the software didn't consider what would happen if they didn't receive a…

Etienne Nichols: Hmm.

Shawnnah Monterrey: A signal, because it was always expecting to get a high or low signal.

Right? And you didn't do a proper DFMEA to determine what would happen, and let's say the software errors out. It's gonna manifest itself as a software defect. So, you'll see a lot of more defects in the industry that get reported that are software-related.

Because the software didn't catch an error mode that was generated by the electromechanics, right? And you can see this even with software when it's on a platform, right? If you're using third-party platform to deploy, let's say, AWS and whatnot, and you're not, accounting for the way that behaves.

And the way it interacts with your software product, then you could have unintended side effects.

Which will be reported as a bug, right? And so, it really requires you to understand how your product is actually built, how it's integrated into the rest of the system, and how those systems interact that could cause failures.

Right? And so, FDA's really big on, you know, risk analysis, right? But failure…and that's considered, like, a top-down approach, right? Looking at it as a system.

But really, where a lot of the issues manifest themselves is going to be at the subsystem level and the interaction between subsystems, so having a really sound DFMEA, even if it's not expected by the FDA, having this really solid bottoms-up approach, helps significantly in reducing unanticipated failure modes out in the field.

Etienne Nichols: Yeah, that makes a lot of sense. I've been through several iterations in the evolution of my knowledge of med tech, I guess. You know, only FMEA while I was in certain areas of the field, manufacturing or whatever, and then…really, when I got into product development, ISO4971 became a disciple of that, and said, forget the FDA, FMEA's dead. Well, come full where I recognize the power of both. You've got to go top-down, you've got to go bottom-up, so yeah.

Shawnnah Monterrey: Yeah. Yeah, exactly.

Etienne Nichols: When you talk about software defects and hardware defects, it makes sense that they behave differently and the interaction is important. How does detection change when the product is…is a…is code?

Any…on that a little bit more.

Shawnnah Monterrey: A bug, or the implementation of a detection mechanism?

Etienne Nichols: Well, either one. I don't know, whichever one's more interesting.

Shawnnah Monterrey: Okay, well, kind of, we talked about, like, how important, like, all the upstream work is important when you launch a product, right? So, if, for example, let's say you're submitting a device to the FDA, and your documentation level's basic, so you submit basic software documents, and you go out there so your code's not well documented, right? Maybe you don't have unit testing.

And you go out there and you find a high-risk, obscure bug.

Right, and so high risk means maybe it impacts patient safety, it could raise to the level of a reportable event by the FDA.

Right? So you need to do a root cause analysis. If you don't have your software well documented, it's going to be difficult to go back and to understand how the system works that can actually manifest this bug. And a lot of times when you find these bugs out in the field, they're not readily reproducible.

Because they're being used in a different environment, as I mentioned. That's why simulating the intended use environment's really important. And now you have this patient interaction with the device.

Right, so I had this, it was a Class 1 recall a long time ago when I was working at, Covidian, and it was a bug where when the patient would breathe a certain way using the ventilator, it would issue, like, a high-priority alarm, and it would shut down the ventilator. And when it shut down the ventilator, it would just basically vent to air. So basically, you're just…you know, you need the ventilator to breathe, it would vent to air, so it's not really helping the patient in that regard, right? Ended up getting escalated, and the team, for whatever it was worth, was trying really hard to reproduce it by breathing into the actual patient circuit and reproduce the problem.

And what ended up happening is it ended up being not a system bug, it was actually a legit software bug.

And so, it wasn't until the team had looked at the design documents to realize that there was a certain implementation that was flawed. It was intentional, but it had a flaw in it.

And that flaw ultimately got addressed, right? But it was very hard to troubleshoot and debug, because they were trying to reproduce it without actually looking at the actual code itself for reproducibility, and so that goes to lend itself to why unit testing is very important.

Right, because unit testing, if you have the unit testing attached to your software, you can actually simulate inputs.

Right? You don't have to simulate the…recreate the patient, right, interaction, because that's more challenging to do a lot of times, not all the times, but you basically would re-simulate, you know, and if the unit testing was there, the bug wouldn't have manifested itself to begin with, right?

Etienne Nichols: Yeah.

Shawnnah Monterrey: And so.

Etienne Nichols: It's hard to isolate your variables with a human being involved, yeah.

Shawnnah Monterrey: Correct, and software's very complex, right?

Etienne Nichols: Yeah.

Shawnnah Monterrey: And so, if you're under-documenting, because you don't need to document to that certain level for FDA, but there's a, you know, high-level risk out in the field, it becomes difficult to reproduce and ultimately fix.

Etienne Nichols: I love that example, because it makes…from the mechanical side of things, it makes me think of a tolerance analysis stack-up. You know, we used to call them T-stacks, tolerance stackups.

Where, you know, if you're gonna test everything together, you would test it a good part, with a good part, with a good part, and we may not go down to the nth level and say, well, something wandered slightly out, and what does it look like at that point? So, it all goes back to your…

Was that tolerance correct? Did we do the T-Stack right? You know, so there's lots of ways to find it in the paperwork that you might not be able to find it on the line, so that's interesting.

Shawnnah Monterrey: Correct, and understanding, like, the assumptions that were made during the development as well, right? Why was the algorithm…once we understood the algorithm was designed that way intentionally, that's where we knew where the problem could exist, and then that's where we started digging in to try to reproduce the issue.

Etienne Nichols: Interesting.

Shawnnah Monterrey: And we did reproduce it pretty quickly after we understood that, versus trying to mimic the patient's interaction and kind of try to guess by playing around with the system, because it wasn't a system problem, it was a software bug.

Etienne Nichols: Man, doesn't that feel good when you do, like, actually arrive there intellectually and, okay, now we got it? That just feels good.

Shawnnah Monterrey: Yeah, and I think it feels good understanding that the process is there for a reason and has value.

Right, and skipping it, it may buy you time now, but it's pay now or pay later, because when you find these issues out in the field, and you have, let's say, hundreds of units, in this case, I had over 80,000 units that were exhibiting this issue, potentially, right? We didn't know. That's a huge install base.

And that's a very expensive software update.

as we're talking, I think, you know, if you look at post-market surveillance, like, specifically around software, that's where, kind of, FDA's really focused on making sure that software is upgradable and patchable.

Right, because these issues that were occurring in the field back in the day with these devices that are 10, 20, maybe sometimes even 30 years old, and still in the field, they're not patchable.

Right? And so, when you have a high-priority issue, like, then you're negotiating with regulators on what the value benefit is of keeping the unit out in the field with a high-risk patient safety issue versus making the actual correction.

Etienne Nichols: Yeah. Yeah. I can see that being a tough…you know, you gotta think about the ROI, for sure.

Shawnnah Monterrey: And so, a lot of these firmware devices aren't really…even now, I'm seeing a lot of clients launching their devices, and they're not field upgradeable.

Etienne Nichols: And so, I mean, that's not necessarily a requirement, I suppose, but it's just…it's really shooting yourself in the foot down the road.

Shawnnah Monterrey: Yeah, and it's not a hard requirement. You can get around it by saying, we're going to do a recall, right, or the device is disposable, limited use. In some cases, it makes a little bit of business sense. Like, if it's cheaper to just replace the device, right, for some of these wearables, it's just cheaper to replace the device.

But again, to your point, you're shooting yourself in the foot if the devices are intended to use longer than you anticipate, or out in the field, and your install base gets large because you're widely successful, and there's an issue that happens, you know, down the line, you want to make sure you're able to address it quickly and promptly.

Etienne Nichols: Well, you mentioned the recall, and that seems like a tough alternative to choose, you know, just from thinking about how to be field upgradeable. So let's talk a little bit about complaints and how that works. Most teams, I would assume, they treat a complaint like a fire you gotta put out.

But when, when did a complaint actually change how you built the next version? Have you…I mean, you kind of mentioned one already about, you know, something coming in, but…does it change how you build the next version when you get those complaints? Like, not just, we're going to fix this, but we're also going to fix the process and how we build things overall, or what have you learned from that perspective?

Shawnnah Monterrey: Yeah, definitely. When you're looking at complaints, you always want to look at root cause analysis and impact, right? So, you can have a lot of defects that don't have patient safety impact but are nuisance.

Right? That manifests itself in terms of the quality of their perception of the product.

Right, so I always say patient safety is really important from an FDA regulatory perspective when building a medical device, but building any product quality is just as important, right? So, when you're looking at defects, you kind of want to look at common themes, so you want to look at, you know, people, process, and tools.

Right? Was this an issue with the tools, the development, tools that we were actually using? Is it an issue in our development process, or is it people? Like, it's in the tools, it's in the process, it just…we didn't have appropriate training.

Right, and really understanding what the root cause is, and just continuously learning. So, I think if you're looking at one version to the next, it is important that you take customer feedback to develop the next product, right? Now, you have some data points, the product's been out in the field, you understand how they're using it, so I think a lot of companies will spend a lot of time on new features, but you should be working with your complaint team.

Right? For smaller companies, it's, you know, easier to do. For larger organizations, it's a little bit more challenging. You need to be able to be…have access to the complaint files, so then you can have it as input into your R&D process to understand, okay, these are most prevalent issues that we're seeing, and these are things that we need to fix, you know, upstream. Upstream in terms of process, or even the implementation of the product…the product itself.

Etienne Nichols: So, root cause analysis, it's…it is a science, but it's also a bit of an art. Knowing when to stop, knowing how far to go. So, I'm curious about this, because you mentioned that. How do you tell…if you've got a complaint, how do you tell that complaint from one that's…really more systemic and structural, and do you wait for more to come in, or what's been your experience?

Shawnnah Monterrey: Well, I think some defects, you can kind of tell right away if they're manifesting themselves based on the instances of occurrence.

Right? You would look at, you know, how many times the issue or a related issue has occurred, right? So just pure instance of incurrence. Other things, like in terms of severity, like the ones that are higher patient severity, like, you obviously want to treat with just importance, right? So, I would look at the ones that are occurring the most.

And the ones that impact patient safety the most. The ones that are occurring the most don't necessarily have to impact patient safety. Those are the quality ones, right? And the ones that impact patient safety, you want to look at those as well. But when you look at them, you kind of want to understand, like again, like, how did they come to be, right? Was it an issue within the manufacturing process? Was it an issue in the development process? Right? Is it something because the user's using the device differently than expected?

Sometimes the user will set up certain configurations or settings or even use the device in unanticipated ways that are still legitimate. It's not outside your indication for use, but you just didn't anticipate it, right? So that could be a patient training issue, or that can be, like, a real flaw, a real flaw in the system.

Etienne Nichols: Yeah.

Shawnnah Monterrey: Sometimes, like, a training issue, you can address that through labeling or, you know, end-user communication, customer communication.

Etienne Nichols: I'm kind of building a flow chart in my brain as you're talking, because we talk about, complaints that come in that systemic versus one-off, then when you go down one layer deeper and say it could be manufacturing, could be how we built it or deployed it, then usability. And then usability there, you know, indications for use outside the indications for use, that may be, you know, that's another thing, but I'm also curious, like, one layer deeper, even, from that branch of this root…of this diagram in my head is, usability problems are interesting, because…

What if, what if they are using it in a way you weren't intended, and maybe outside your indications for use? It's not a bug, but a defect, but they're using it this way, and it's a bug in their mind. What about that?

Shawnnah Monterrey: Well, that has its own related issues, right? So, if they're veering off using it outside of your indications, and your indications were, addressing a different, completely different use case, then that's outside of your indications for use, which is problematic, but the liability is on the end user.

From that perspective, as it relates to that intended use, where it relates to the manufacturer is if they're promoting that alternative intended use. Sometimes sales reps will get involved and sell it a certain way, and that's actually problematic for the medical device manufacturer, right? You cannot sell and market your device outside the intended use.

You know, and despite the intended users using it inappropriately, right, that's where the liability comes in. And so, if you know that that's happening, you have basically an obligation.

To, to address it with proper patient education and customer education, right? Proper training material, updated.

Etienne Nichols: When it's outside the indication, or inside, or…

Shawnnah Monterrey: If they're using it outside, I think there is a responsibility if it's known and it's prevalent.

That it gets addressed, like re-communicating, looking at your labeling and indications for use to make sure that it's actually clear, and that it's not intended. So, tell us maybe you need to add a contraindication or something like that.

Etienne Nichols: Yeah, that was the other thing I was curious about, and not to play the devil's advocate, because I don't believe in doing that, I'm just now reading Master of Margarita, I'm not sure if you're familiar with that. It's a Russian novel about how the devil comes to Moscow, but whatever.

I'm like, I don't want to play advocate with that guy. But, so, we're talking about that contraindication is the risk mitigation, I guess, in my mind, because if that complaint comes in, even if it's outside, I can see two trains of thought coming here. Someone comes in.

or a complaint comes in, it's outside the indications use, you could say, well, our hands are clean, it's in the IFU, we've trained them, etc, but I can almost see a product liability attorney saying, you should have put some guardrails in place so it could not be used that way. Is that the case, or do you put, you know, what are your thoughts there?

Shawnnah Monterrey: I think it's worth evaluating it and determining the risk of doing something versus not, and going back and, again, that root cause analysis and evaluating everything. Like, did we miss this in usability? Right? Should we have addressed this? Could we have caught it then? Right? Is our labeling clear? Can we be clearer?

Right? Checking with sales, are they actually promoting this? Because why is it so predominant?

Right? Is it just word of mouth, or are sales reps actually encouraging this behavior, right? Making sure you do your due diligence to make sure that you're not encouraging it in any way.

Yeah.

Etienne Nichols: One of my favorite usability examples is someone has a flat blade screwdriver, and they open…what do you use that for? Well, what do you use a flat blade screwdriver for? You open it to…you use…open it to…you use it to open paint cans, right? And then what do you use the handle for? To smash it closed again. And if that handle's glass, oh man.

Shawnnah Monterrey: Yeah.

Etienne Nichols: Yeah, you have to think about those misuse cases that's.

Shawnnah Monterrey: Yeah, so there is a responsibility, like, I don't want to use the word.

Etienne Nichols: No, no.

Shawnnah Monterrey: Community, right? But I think there is a responsibility on the medical manufacturer to do proper root cause analysis to determine why this is actually happening.

On the flip side, you know, they could start looking at it as, is this a proper indication, and then go to the FDA and extend their indication, so if they're actually excited about it, like, they're opening up the market in a way they didn't anticipate, then they, again, should go and be responsible and go back to the FDA and then update their indications accordingly, based on how people are actually using the device.

Etienne Nichols: I love that, and I'll just throw out the name of a book, because there's a whole case that…just around this entire subject, by a guy named Howard Root. It was the book Cardiac Arrest, Five Years as a CEO on the Fed's Hit List, and his company was accused of being…doing things off-label, but it turned out, after 5 years, he was acquitted on all accounts. But anyway, it's worth a read. It's definitely something I think every medical device CEO should read.

So, if you're listening and you want, you know, you're interested, reach out to.

Shawnnah Monterrey: Yeah, that sounds like a great book.

Etienne Nichols: It's very interesting. Okay, so let's talk about those upstream needs and the downstream expectations. So, upstream needs, because you talked about when those defects come in, that tracing, that complaint should go back to the design decision that caused it, but how do you actually do that in practice?

Shawnnah Monterrey: In terms of, like, the team…structuring the teams, or the actual process of doing the analysis itself?

Etienne Nichols: Well, I'm also almost going back to the…to what you said, the people, process, tools. I mean, what do you recommend as far as all the different things that are required, whether it's people, or process, or…yeah.

Shawnnah Monterrey: Yeah, so I think one is to make sure there's an open channel with the complaint team and the R&D team and make sure the R&D team is getting the information that they need. So, a lot of times, the complaints, right, garbage in, garbage out, aren't being categorized appropriately, and so, hence, they're not getting disseminated to the team. And a lot of times, if there's a trend, the complaint team might just be managing that. They might have a patch.

Right? Or some sort of, yeah, I mean, here's a good one. This was at Illumina. They had a team that would do customization out in the field.

And so, they would do custom software in addition to the product that got sold. And a lot of times the custom software, I guess they thought it was capability and features, but a lot of times it was fixing issues within the product, and that information never got back to the R&D team.

Etienne Nichols: Oh.

Shawnnah Monterrey: Right, so having appropriate communication channels, number one, is super important. So, a good example of that is when I was at Covidian, I was, head of their, basically, like, post-market, all the, all the products that were out in the field.

And I was on that meeting, right? So, I think they had a weekly meeting, and then they had a monthly meeting, so I would actually see the trends for all the products that were in market, and then able, because we're always constantly making software updates, being able to factor those in.

That's one. Two, I would say even without…despite the analysis, I think there's a trend towards businesses to focus on features and capabilities, because they see that as driving revenue and not focus on defects.

Right, and so I think it's really important that management is aware and accounts for any sort of software release that's coming out, even if it's a feature release, to account for bug fixes. I think it's super important that that gets factored into the product, and a lot of times it doesn't, for budget reasons, timelines, a lot of times the engineering team will come back and say, you know, the software's so poorly architected, it needs to be refactored, and anytime management hears refactor, they're like, -oh, right?

But refactoring's just a common part of development. You should constantly, as you're adding in changes, be refactoring the code to make it more modern, more maintainable, and redesigning the areas when you actually put in a new feature where it makes sense to do the redesign, because eventually what you'll have is technical debt, and you'll have a very slow, archaic, hard-to-maintain system that ends up being out in the field for decades that nobody wants to touch, right?

Etienne Nichols: Oh, man, and I don't want to…we've gone along, what, 33 minutes here without talking about AI, but I can't.

Shawnnah Monterrey: Notebook.

Etienne Nichols: I'd ask you this. With the amount of code that's being written by AI, do you think that tech debt's gonna go up in the future, or what are your thoughts?

Shawnnah Monterrey: I think there's going to be higher demand for software engineers in the role, so I think people are thinking AI is going to replace software engineers, so I kind of made it equivalent to people that aren't software engineers. I said, did the 3D printer get rid of mechanical engineers?

Etienne Nichols: Oh.

Shawnnah Monterrey: I did not. That's good. And now we're like, oh my gosh, that prototype was 3D printed.

Right? So now it's a criticism. So, I don't…I'm not saying we're gonna go as far as, like, oh my god, that code was written by AI. But I think there's an aspect of that. I haven't seen AI-generated code to see how maintainable it is, but what I will tell you is, I think this…everyone can relate to this. If I ask my chat…I have a personal instance of ChatGPT, and a professional instance, sorry, of ChatGPT we use it for our business. If I ask it the same question 3 times in 3 separate chats, I'll get 3 different answers.

Etienne Nichols: Yeah.

Shawnnah Monterrey: They're not even the same. They structurally look different, they have a different feel, they even have a different end result. So, one could give me a high rating, like, this document's great, and the other one could say, you're missing a bunch of stuff, because the reason being is because there's no control. It's constantly learning, so it's not repeatable.

Etienne Nichols: Yeah.

Shawnnah Monterrey: But when you actually have to sustain something, what you need is something that's repeatable. So, let's say you have a bug, and you ask, okay, let's say you have a baseline of the software, you did it through AI, okay? The baseline, you launch a product that way. Now you have a bug.

And you go back and you feed it the AI to fix, assuming AI can fix it that code that it regenerates to fix it will be completely different. You won't even be able to do a comparison to look at the actual differences to do a proper regression analysis. So, guess what? If you had AI generate your code.

you better have appropriate unit test coverage. I would say very high unit test coverage, because you will have to retest the entire software for one bug change.

Etienne Nichols: Yeah.

Shawnnah Monterrey: But if I developed the code, or the code was maintainable, or AI created it, and now I'm gonna fix the bug, assuming I know how to read the code.

Right? It's easy to follow, navigate. Like I said, I'm not…I haven't seen the actual code that AI creates to know whether it's re…if it's well-structured, using optical design, I don't know all that, right?

Etienne Nichols: To your point, I can imagine Oh, sorry to interrupt, yeah.

Shawnnah Monterrey: No, go ahead, go ahead.

Etienne Nichols: One thing I could see it being very, very good at is adding those notes to the code. Anytime I wrote code, it was always like, okay, I don't, you know, the notes, who's actually going to read this? I just want the thing to work, but I could see the notes maybe being better, so anyway, yeah.

Shawnnah Monterrey: Yeah, I think AI is great for, like, reverse engineering design documentation from the code, that would be good and maintaining that documentation.

Etienne Nichols: Fathom and take…

Shawnnah Monterrey: Would be great. Maintaining the, you know, the tech stack and the tool automation, like, I think that'll be great, and reporting and whatnot.

But I'm not, you know, I think, you know, obviously I'm probably going to get critiqued for what I said, but I want to qualify. I'm a software engineer, I have a degree in computer science, and back in the day, we were trying to use MATLAB to create algorithms, and because we couldn't, maintain it.

And it wasn't reproducible. Every time it would auto-generate the code, we decided to create it from scratch, because we had more control over it.

Right, and that was our essential performance for our medical device.

Right? And maybe you can argue for the GUI you know, it's fine. But if the code gets regenerated each time you fix an issue, right, I'm making that assumption.

Etienne Nichols: Back to usability, yeah.

Shawnnah Monterrey: We're gonna have an issue with usability, maintainability, right? The code, the UI gets tweaked a little bit, the icons change, right? I mean, how many times have you tried to create a graphic using AI and just said, can you just change my logo, and the whole graphic's completely different?

Etienne Nichols: Yeah.

Shawnnah Monterrey: I just wanted to change the logo, right? So that's just what I'm seeing. So, I think it's going to actually increase, the demand for software engineers, because I think the job…their job is just gonna get harder. Yeah.

Etienne Nichols: Quality.

Shawnnah Monterrey: Troubleshooting, debugging, like, if they find performance issues that are just deep in the tech stack.

They're gonna be really challenged to debug and to kind of reverse engineer what happened and maybe have to re-architect some things that were developed through AI.

Etienne Nichols: Yeah. Oh, MATLAB. I was not good with MATLAB.

Shawnnah Monterrey: It's the same argument, right? Like, did 3D printing get rid of mechanical engineers? No. It just changed their role, so I think AI does have a place, probably a significant place in software, but I think it's just going to change the role of the software engineer.

Etienne Nichols: Well, you've…you talked about design controls and risk analysis, so where upstream do you think that loop usually breaks first, or is there a trend?

Shawnnah Monterrey: Requirements?

Etienne Nichols: Requirements, yeah.

Shawnnah Monterrey: Every time requirements not having a well enough spec'd device, right, or not. And then the second would be testing, superficial testing. The superficial testing usually goes hand-in-hand, the requirements aren't spec…specific enough.

Etienne Nichols: When you say you could…the AI could be good for reverse engineering those design documents from the code, that…that's an interesting thought to me, because you now have a product, and then you're gonna go back and update your design, so where…how does that work in your mind?

Shawnnah Monterrey: I'm envisioning the cases that we see more often than not that we shouldn't see, where the code's fully developed and there was no design documents developed.

Etienne Nichols: World. Yeah.

Shawnnah Monterrey: Assuming you're tracing your unit testing to the actual implementation, I think that's key, because that's where it's going to provide you a trace from the code. So, let's say I have to fix a bug.

and I've already reverse-engineered all my documents, or at least the AI is trained on my documentation, then what can happen is I fix the bug, and the chain will go all the way up, and then tag it all the way to the requirements, right? So, let's say I change an icon from green to yellow.

Right? It'll go back all the way up, potentially to the requirements, maybe to the design spec, depending on what level you're specifying that level of detail, right? And then your test cases would appropriately be updated, and then it can go through and do the full impact analysis. So, I see it being impactful

When you've developed the software and didn't do the proper documentation, so reverse engineering all that, so now you have your design documentation, and then when you actually have to then go and maintain the code, and going back and updating the design documents, and doing the full impact analysis so you know where the regression needs to happen.

Etienne Nichols: If…

Go back to our use case there with AI generating something, and, you know, it'd be interesting to see that play out. I know that's happening right now, where people just have their prompts. Would you say that is almost…the prompt is the requirement?

Alright.

Shawnnah Monterrey: What do you mean prompt? Can you give me an example?

Etienne Nichols: Well, if you're prompting an AI to generate code, you are essentially giving AI your user need, for lack of a better word, and then it goes and makes a bunch of assumptions, generates the code based on your prompt and the assumption it makes around your prompt with the context that's been provided.

Shawnnah Monterrey: Your prompt would probably go back to your…it would be, like, maybe your user story, or your user needs, user input. Sure. But the requirement probably manifests itself in the code, right? So, if you say, I need the user to be able to log in, log out.

Right, then it's gonna implement that, but it may implement other things, like, if it's an auto-log-out, yeah, if there's, like…it depends, I guess, on the specificity you have to feed into the AI. I haven't used it myself for that purpose, so I don't know. I'm imagining you don't need a high level of specificity, and it's gonna produce stuff.

And then what you can imagine the breakdown is, is let's say I'm not a software engineer, I'm gonna actually miss all the nuance requirements. So now I'm like, I have login, log out, but I didn't specify the password, requirements. It was gonna be a strong password required by FDA. I didn't specify how often they need to reset the password. I didn't specify if there's an auto-logout, right? I didn't specify, you know, what the username requirements are, if it needs to be unique.

you're missing all those things. So yeah, technically it might functionally work as a prototype, and prototypes are misleading, right? So, you see a lot of startups that are asking for a couple million dollars and showing a very mature prototype, and they're like, why do you need that? Oh, because it's a prototype. It looks like it works, right? But they know.

Etienne Nichols: Yeah.

Shawnnah Monterrey: That there's a lot behind the scenes, like, so it's kind of dog and pony show. So, I don't know if AI is building full, polished you know, applications, or if it's surface-level, you know, user interface type, you know, code. Because I know when I ask it to write a process document, it has a really hard time writing a full process without starting at the beginning, which means AI needs to, like, be almost prompted section by section, or create the outline first and section by section, right? And I'll get different levels of fidelity, depending on what the AI knows.

Right?

Etienne Nichols: Interesting, yeah.

Shawnnah Monterrey: I mean, there's a lot of third-party software now, like, Microsoft has, you know, user-authentic tech stacks that actually support all these key capabilities, right? So, if AI is integrated with these robust, you know, tech stacks, then maybe that's, you know, an area that can be explored in terms of, like, the robustness. Yeah.

Etienne Nichols: Well, listen.

Shawnnah Monterrey: But I am a skeptic, I am a skeptic.

Etienne Nichols: I bet.

Shawnnah Monterrey: I'm not trying to preserve my job, I don't code anymore, but I understand the work involved as a software engineer.

Etienne Nichols: Yeah.

Shawnnah Monterrey: That I…and I see the output being produced by AI, again, granted, it's not code, where I can see, you know, the problem directly manifest itself in the programming world.

Etienne Nichols: Well, if someone's out there, you are building a product purely with AI and that code. We'd love to hear from you, at least I would. You know, maybe Shana would too, we could, you know, kick that around, so I'd love to reach out and hear your experience.

Shawnnah Monterrey: Yeah, I'd love to see how it works, and the level of quality, and how they're actually, you know, maturing it as they move along the development cycle. That'd be awesome.

Etienne Nichols: So, we've got upstream and how you tie that back. What about the downstream? Because you've got patients, clinicians, attorneys, payers…whose expectation do you think is hardest to design for, and why?

Shawnnah Monterrey: Expectations hardest to design for the end user.

The reason being is because they're the ones that's using your device day in and day out, and a lot of times they're not even interviewed at the design and development process, right? So, you'll see a lot of products being developed, and the user needs are created in-house without even talking to a customer.

Etienne Nichols: Yeah.

Shawnnah Monterrey: A potential customer, right? They're the ones that are ultimately going to use it, so if it's not easy for them, regardless how well you did a usability study, usability studies are mocked.

Right? It's not using the final product, because if we had to wait for the final product, we'd never get a product out the door, right? So, it's usability, like, I think a lot of companies will skip because it's not required formative testing. Formative testing is super valuable. That's when you can go back and validate your user needs before you

Invest in significant development, and then once you develop is making sure when you're doing your summative testing, you're representing the prop…the users that you expect to use your device and in the environment.

To get the most feedback sooner than later. There's a lot of surveys that get done even before creating user needs, like voice of customer. I don't see that being done enough. And so, imagine going through this entire process, which is super expensive. You get it clear to the FDA, and you send it to the end user, and they don't find it to be useful.

Yeah.

-Oh.

Etienne Nichols: Go.

Shawnnah Monterrey: So, oftentimes you'll see that the, the voice of customer isn't done before drafting user needs. That should be direct input, and it's usually a marketing commercialization activity to define the user needs, and what will end up happening is you get a device.

That you can clear, right, and that's safe for patients, but imagine shipping that device, and your customers, your end users, either the clinicians or the patients, whoever your end users are, deem the product as being unusable.

Yeah. Then you went through all that effort, all that expense, with a product that you cannot commercialize.

Etienne Nichols: That basically is your usability test at that point, and you did it in the field versus in-house, so, yeah.

Shawnnah Monterrey: Exactly. Yeah, that's your final usability test. If you do everything properly upstream, including collecting voice of the customer, then you'll minimize the risk of shipping a patient…shipping a device that nobody finds useful.

Etienne Nichols: Well, we've covered a lot of different aspects of this, so what last piece of advice or calls to action do you have for those listening out there in the audience?

Shawnnah Monterrey: I feel like we touched a lot on, like, key inputs that are…and outputs that is really important for designing a high-quality product, and to minimize the risk of post-market issues, which you will have, right? But to minimize that, and to also make it easier to address those. And so, I think the key trends, I would say, would be having appropriately written

Product, specifications and requirements.

Right, software and system, to the appropriate level of fidelity to basically allow you to thoroughly test your product.

So that's one key takeaway. The other key takeaway is utilizing the DFMEA.

Right, although you don't necessarily have to submit it to the FDA, and I'm not saying you need 50,000 lines for the DFMEA repeating the same issue or resolution, but you should look at each of your subsystems and how they interact and how they could fail.

And put in appropriate mitigations, even if it potentially doesn't impact patient safety.

To help reduce the risk out in the field. So, I think those are the two main key takeaways today, among many, but…

Etienne Nichols: Awesome. And if there are more questions out there, if you're listening and you have just…we've opened up more questions in your mind than answered, feel free to reach out to Shawnna or myself, because Shawnna does this every day, and I'm sure she'd be more than happy to help and discuss whatever it is you're working on.

And if you have thoughts on other topics or other people you'd like me to interview, I'd love to hear that as well. I'm on LinkedIn pretty often, so although less and less lately, I'm trying to get back up and on top of things. It's hard to stay on top of that, but...

Shawnnah Monterrey: I know.

Etienne Nichols: Hit me up, yeah. Alright, we'll put the links in the show notes to both Shawnna and myself, and if you're interested in getting in touch, feel free. But, other than that, everybody, thanks so much for listening, take care, and we'll see you next time.

Shawnnah Monterrey: Thank you, have a wonderful day.

Etienne Nichols: Thanks for tuning in to the Global Medical Device Podcast. If you found value in today's conversation, please take a moment to rate, review and subscribe on your favorite podcast platform. If you've got thoughts or questions, we'd love to hear from you.

Email us at podcast@greenlight.guru.

Stay connected for more insights into the future of MedTech innovation. And if you're ready to take your product development to the next level. Visit us at www.greenlight.guru. Until next time, keep innovating and improving the quality of life.

About the Global Medical Device Podcast:

Untitled (8.5 × 3 in)

The Global Medical Device Podcast powered by Greenlight Guru is where today's brightest minds in the medical device industry go to get their most useful and actionable insider knowledge, direct from some of the world's leading medical device experts and companies.

Like this episode? Subscribe today on iTunes or Spotify.

Etienne Nichols

Etienne Nichols is the Head of Industry Insights & Education at Greenlight Guru. As a Mechanical Engineer and Medical Device Guru, he specializes in simplifying complex ideas, teaching system integration, and connecting industry leaders. While hosting the Global Medical Device Podcast, Etienne has led over 200...