5 ms·

How we develop FDA-compliant machine learning algorithms

58 points by yshrestha 6y ago

omarhaneef 6y ago
I looked at a few of these (FDA approved AI products) recently because I was curious as to what the instrument was allowed to do and sure enough it was more of a smart instrument that told you, for instance, how large a volume was. These products don't, at the moment, actually diagnose a condition or recommend a treatment which is interesting. One of the constraints for AI treatment has always been authority and responsibility. To put it another way, if the AI completely messes up and recommends something terrible, whom do you sue? I am hopeful that we (as a society) will sort out these issues with self-driving cars and apply some of those rules to medicine.
- superbcarrot 6y ago
  It's tough to assign blame and responsibility to systems that are inherently probabilistic. ML algorithms output probabilities so even as the algorithms get incredibly good, unlikely scenarios will inevitably occur in large enough samples. This isn't a problem when the task is to tag your Facebook photos but is a problem when driving a car in the real world.
  - OnlyOneCannolo 6y ago
    >It's tough to assign blame and responsibility to systems that are inherently probabilistic. I hear this all the time nowadays, but everything has always been probabilistic. Most of the time, people just don't realize that all problems are fundamentally the same from the perspective of how to approach risk.
    - superbcarrot 6y ago
      I'm not following. How is everything probabilistic (in a meaningful way)? Most other systems don't have built in uncertainty in the same way ML does.
      - OnlyOneCannolo 6y ago
        Physical systems are probabilistic. Environmental conditions are probabilistic. And we've been trusting our safety to physical products operating in the real world for a long time. Whether physical or ML or whatever ... There's always some process that you can't characterize to your liking. So you explore mitigation options until you can make a decision about whether to accept the risk. Standards and regulations enforce that you went through those steps to get a reasonably trustworthy result. Then liability law applies in the usual way. Actuaries and accountants tell you what it's going to cost you, and market research tells you how much it might sell for. Inability to control or understand what you built doesn't absolve you of any fault or responsibility. All it does is change your risk acceptance criteria.
      - TeMPOraL 6y ago
        All our models are inherently probabilistic. The way we perceive and reason about the world is probabilistic, too.
- Gatsky 6y ago
  I think it is a little wrong-headed to emphasise the litigation aspect. Something that I think people don't appreciate is that as a patient, you want a real person to take responsibility for your problem because that is the basis for a favourable outcome and a satisfying interaction. This applies to every service in the end, whether you are ordering food at a restaurant, trying to get a plumbing problem fixed or being treated for cancer. Any corporation, for-profit or not, generally seeks to obfuscate their responsibilities where possible (one could argue that this is the explicit purpose of a corporation - to be a virtual entity which cannot really be held accountable). Tobacco companies are the best and ongoing examples of this. Creating legal protection for AI such that accountability is somehow removed or diminished is not desirable in my opinion. It will lead inevitably to a situation where healthcare becomes analogous to trying to get technical support for your printer. Contrast this with the what happens currently, where you walk into a clinic or a hospital with a problem and meet a bunch of real people who listen to you. Of course there are massive problems with healthcare, and many people have unpleasant or dangerous experiences, but it can definitely be made worse if we aren't careful.
- curiousllama 6y ago
  I've done a fair amount of work in the area, and one of the biggest lessons is "even if you can diagnose, you shouldn't" We likely won't have to deal with the legal dilemmas for a while for a simple reason: doctors _hate_ when non-doctors diagnose, and you can't build a diagnostic tool without doctors. The tools simply won't get built to any real quality standard. There's an easier path, interestingly. It's much more effective to just give docs the info they need. A system that could do the following would be the holy grail: "Patient entered ER with [chest pain, shortness of breath], and has a history of [hypertension, diabetes]. They are on [X, Y medication], but have not filled their prescription in [Z weeks]." No diagnosis, just the right info at the right time.
  - omarhaneef 6y ago
    The expert system guys in the 80s -- and this is before my time -- came to a similar conclusion, but for the reasons I am laying out (responsibility). They also decided it was better to have a computer make a suggestion to a doctor than to give it unvarnished to a patience, sort of assisting the doctor. I won't disagree with experts (yourself, or them), but there is the issue of scaling it. Consider a poor person in a poor country for whom the alternative is not a delayed appointment with a real person but no advice at all.
king_magic 6y ago
Good read. Reminds me a lot of Andrej Karpathy's "A Recipe for Training Neural Networks", which has served me extremely well in my AI career. https://karpathy.github.io/2019/04/25/recipe/ https://karpathy.github.io/2019/04/25/recipe/
- yshrestha 6y ago
  Thank you for sharing. These are awesome AI nuggets. In particular, I can relate to the author's suggestion to "visualize everything."
  - king_magic 6y ago
    100%, it's completely changed how I approach DL projects.
ska 6y ago
It's an interesting time for AI/ML/pattern rec in medical devices. People have been shipping such systems since at least the mid 90s (e.g. CADe) but it's always been a bit of a difficult area from a regulatory point of view. If you go back a couple decades, or really even only one depending on the panel, it was fair to describe the FDA as an organization that understood hardware but didn't really understand software. On top of this, a lot of their approach was focused on controlled, reproducible results. So anything probabilistic or even worse, continually learning, becomes difficult to frame. The FDA guidance doc/ action plan (link is broken in article and on FDA site: it's here https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-software-medical-device https://www.fda.gov/medical-devices/software-medical-device-...) is actually pretty encouraging, because they took community feedback from the original request for feedback (https://www.fda.gov/media/122535/download https://www.fda.gov/media/122535/download) and seem to have handle that process reasonably. Many people I've talked to without direct experience seem to assume the 13485 process as a whole is needlessly bureaucratic, but in my experience it's mostly holding yourself to a higher engineering standard that some other industries, and works out ok. This willingness to engage with technologies can be frustratingly slow, but it's mostly reasonable.
- ealexhudson 6y ago
  13485 is relatively bureaucratic, but it's not required for FDA per se and there are things you can do. What is more problematic is IEC 62304 or similar, which embeds a very particular SDLC into the development approach. It doesn't really fit AI/ML, it barely fits modern software development anyway.
  - ska 6y ago
    You can avoid it for class I, but generally 13485 is becoming more harmonized and required. I don't think 62304 is too hard to adapt; i haven't done the process work in a while but unless the latest rev radically changed things it isn't very prescriptive, not nearly as much as you suggest; I've seen agile and Agile variants done. It is a sign of the slowness of the process that it's only recently become required, and reflects 15-years ago thinking, but it's fine. You are right that a lot of modern development processes need a fair bit of tweaking, but it's mostly because they aren't nearly careful enough for device use. This may be a bit of a burden on pure software devices, but it's probably the right trade off overall. This is why the big shift in thinking for most developers will be the 14971 (risk management) adoption, not 62304 itself (IEC 62304 punts to ISO 14971 for this part, but it's common with hardware dev)
jellyksong 6y ago
Two questions: 1. I wonder how many times the test set can be used on "incremental changes in future versions of the model" before losing statistical validity. 2. This article describes their process, but not the FDA's process. Are there specific regulatory requirements for ML models beyond their four types of reports?
- yshrestha 6y ago
  1) AFIAK there are no hard and set rules for this. I think this would have to be the manufacturer's judgement call. Good point though, with enough time you may end up just over-fitting to the test set. 2) The FDA's guidance on ML models is still in flux. Please see https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-software-medical-device https://www.fda.gov/medical-devices/software-medical-device-...
Ericson2314 6y ago
Really not looking to forward to living in a ideocracy dependent upon proprietary software and black-box models. One is enough.
grizzly_bear 6y ago
What is your tooling regarding CI/CD and documentation maintenance? We thought we’d use off the shelf software to do (COTS ;)) but it seems incredibly complex and rigid for a SaMD that’s just, in a way, a simple function.

58 points by yshrestha 6y ago

I looked at a few of these (FDA approved AI products) recently because I was curious as to what the instrument was allowed to do and sure enough it was more of a smart instrument that told you, for instance, how large a volume was. These products don't, at the moment, actually diagnose a condition or recommend a treatment which is interesting. One of the constraints for AI treatment has always been authority and responsibility. To put it another way, if the AI completely messes up and recommends something terrible, whom do you sue? I am hopeful that we (as a society) will sort out these issues with self-driving cars and apply some of those rules to medicine.

superbcarrot 6y ago
It's tough to assign blame and responsibility to systems that are inherently probabilistic. ML algorithms output probabilities so even as the algorithms get incredibly good, unlikely scenarios will inevitably occur in large enough samples. This isn't a problem when the task is to tag your Facebook photos but is a problem when driving a car in the real world.
- OnlyOneCannolo 6y ago
  >It's tough to assign blame and responsibility to systems that are inherently probabilistic. I hear this all the time nowadays, but everything has always been probabilistic. Most of the time, people just don't realize that all problems are fundamentally the same from the perspective of how to approach risk.
  - superbcarrot 6y ago
    I'm not following. How is everything probabilistic (in a meaningful way)? Most other systems don't have built in uncertainty in the same way ML does.
    - OnlyOneCannolo 6y ago
      Physical systems are probabilistic. Environmental conditions are probabilistic. And we've been trusting our safety to physical products operating in the real world for a long time. Whether physical or ML or whatever ... There's always some process that you can't characterize to your liking. So you explore mitigation options until you can make a decision about whether to accept the risk. Standards and regulations enforce that you went through those steps to get a reasonably trustworthy result. Then liability law applies in the usual way. Actuaries and accountants tell you what it's going to cost you, and market research tells you how much it might sell for. Inability to control or understand what you built doesn't absolve you of any fault or responsibility. All it does is change your risk acceptance criteria.
    - TeMPOraL 6y ago
      All our models are inherently probabilistic. The way we perceive and reason about the world is probabilistic, too.
Gatsky 6y ago
I think it is a little wrong-headed to emphasise the litigation aspect. Something that I think people don't appreciate is that as a patient, you want a real person to take responsibility for your problem because that is the basis for a favourable outcome and a satisfying interaction. This applies to every service in the end, whether you are ordering food at a restaurant, trying to get a plumbing problem fixed or being treated for cancer. Any corporation, for-profit or not, generally seeks to obfuscate their responsibilities where possible (one could argue that this is the explicit purpose of a corporation - to be a virtual entity which cannot really be held accountable). Tobacco companies are the best and ongoing examples of this. Creating legal protection for AI such that accountability is somehow removed or diminished is not desirable in my opinion. It will lead inevitably to a situation where healthcare becomes analogous to trying to get technical support for your printer. Contrast this with the what happens currently, where you walk into a clinic or a hospital with a problem and meet a bunch of real people who listen to you. Of course there are massive problems with healthcare, and many people have unpleasant or dangerous experiences, but it can definitely be made worse if we aren't careful.
curiousllama 6y ago
I've done a fair amount of work in the area, and one of the biggest lessons is "even if you can diagnose, you shouldn't" We likely won't have to deal with the legal dilemmas for a while for a simple reason: doctors _hate_ when non-doctors diagnose, and you can't build a diagnostic tool without doctors. The tools simply won't get built to any real quality standard. There's an easier path, interestingly. It's much more effective to just give docs the info they need. A system that could do the following would be the holy grail: "Patient entered ER with [chest pain, shortness of breath], and has a history of [hypertension, diabetes]. They are on [X, Y medication], but have not filled their prescription in [Z weeks]." No diagnosis, just the right info at the right time.
- omarhaneef 6y ago
  The expert system guys in the 80s -- and this is before my time -- came to a similar conclusion, but for the reasons I am laying out (responsibility). They also decided it was better to have a computer make a suggestion to a doctor than to give it unvarnished to a patience, sort of assisting the doctor. I won't disagree with experts (yourself, or them), but there is the issue of scaling it. Consider a poor person in a poor country for whom the alternative is not a delayed appointment with a real person but no advice at all.

king_magic 6y ago

Good read. Reminds me a lot of Andrej Karpathy's "A Recipe for Training Neural Networks", which has served me extremely well in my AI career. https://karpathy.github.io/2019/04/25/recipe/ https://karpathy.github.io/2019/04/25/recipe/

yshrestha 6y ago
Thank you for sharing. These are awesome AI nuggets. In particular, I can relate to the author's suggestion to "visualize everything."
- king_magic 6y ago
  100%, it's completely changed how I approach DL projects.

ska 6y ago

It's an interesting time for AI/ML/pattern rec in medical devices. People have been shipping such systems since at least the mid 90s (e.g. CADe) but it's always been a bit of a difficult area from a regulatory point of view. If you go back a couple decades, or really even only one depending on the panel, it was fair to describe the FDA as an organization that understood hardware but didn't really understand software. On top of this, a lot of their approach was focused on controlled, reproducible results. So anything probabilistic or even worse, continually learning, becomes difficult to frame. The FDA guidance doc/ action plan (link is broken in article and on FDA site: it's here https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-software-medical-device https://www.fda.gov/medical-devices/software-medical-device-...) is actually pretty encouraging, because they took community feedback from the original request for feedback (https://www.fda.gov/media/122535/download https://www.fda.gov/media/122535/download) and seem to have handle that process reasonably. Many people I've talked to without direct experience seem to assume the 13485 process as a whole is needlessly bureaucratic, but in my experience it's mostly holding yourself to a higher engineering standard that some other industries, and works out ok. This willingness to engage with technologies can be frustratingly slow, but it's mostly reasonable.

ealexhudson 6y ago
13485 is relatively bureaucratic, but it's not required for FDA per se and there are things you can do. What is more problematic is IEC 62304 or similar, which embeds a very particular SDLC into the development approach. It doesn't really fit AI/ML, it barely fits modern software development anyway.
- ska 6y ago
  You can avoid it for class I, but generally 13485 is becoming more harmonized and required. I don't think 62304 is too hard to adapt; i haven't done the process work in a while but unless the latest rev radically changed things it isn't very prescriptive, not nearly as much as you suggest; I've seen agile and Agile variants done. It is a sign of the slowness of the process that it's only recently become required, and reflects 15-years ago thinking, but it's fine. You are right that a lot of modern development processes need a fair bit of tweaking, but it's mostly because they aren't nearly careful enough for device use. This may be a bit of a burden on pure software devices, but it's probably the right trade off overall. This is why the big shift in thinking for most developers will be the 14971 (risk management) adoption, not 62304 itself (IEC 62304 punts to ISO 14971 for this part, but it's common with hardware dev)

jellyksong 6y ago

Two questions: 1. I wonder how many times the test set can be used on "incremental changes in future versions of the model" before losing statistical validity. 2. This article describes their process, but not the FDA's process. Are there specific regulatory requirements for ML models beyond their four types of reports?

yshrestha 6y ago
1) AFIAK there are no hard and set rules for this. I think this would have to be the manufacturer's judgement call. Good point though, with enough time you may end up just over-fitting to the test set. 2) The FDA's guidance on ML models is still in flux. Please see https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-software-medical-device https://www.fda.gov/medical-devices/software-medical-device-...

Ericson2314 6y ago

Really not looking to forward to living in a ideocracy dependent upon proprietary software and black-box models. One is enough.

grizzly_bear 6y ago

What is your tooling regarding CI/CD and documentation maintenance? We thought we’d use off the shelf software to do (COTS ;)) but it seems incredibly complex and rigid for a SaMD that’s just, in a way, a simple function.