7 ms·

AI is making critical health care decisions, but is unregulated

35 points by isaacfrond 3y ago

macilacilove 3y ago
In the EU if a software is influencing diagnosis decisions of a doctor in any way it needs to be registered as a medical device software. Probably a similar regulation can be expected in the United States too.
- amelius 3y ago
  People already use ChatGPT to determine if a doctor's visit is necessary, so AI is already in the decision chain.
- JTyQZSnP3cQGa8B 3y ago
  From my own experience, most medical software is sold in the USA and must be certified by the FDA which is equivalent to the CE process.
huijzer 3y ago
Recently my girlfriend was sick and after seeing multiple doctors in Germany who all had long delays, unfriendly assistants, little time, and conflicting conclusions, I have high hopes for AI. It’s easy to improve the situation from where it is now.
- smcl 3y ago
  I'm sorry but surely it is even easier to see AI worsening the situation? Would you be happier if your girlfriend received the wrong treatment from an unaccountable system based on decisions made for unknowable reasons, as long at was in a timely manner?
  - huijzer 3y ago
    Can't the system be held accountable? Unknowable reasons holds for doctors too, by the way.
    - smcl 3y ago
      Well no they're quite famously unaccountable. IBM even had a mantra that stated "A computer can never be held accountable, therefore a computer must never make a management decision". Re the second, if you are prescribed a certain treatment by your doctor, it is entirely possible to find out why that treatment is proposed.
jeanloolz 3y ago
It is regulated in Europe via the AI Act, effective in 2026 (iirc).
- pintxo 3y ago
  It’s regulated everywhere with sensible health regulation. If you intent to have your software to make decisions about diagnosis or treatment. You are a medical device with all that follows from that.
pintxo 3y ago
This seems to be a rather superficial article: > When the FDA approves drugs and medical devices, it doesn’t need to keep tabs on how they evolve. The FDA clearly keeps an eye on ongoing developments. If your company produces an FDA approved drug, be aware you might get a visit by them to check on you, especially if you reported - oh yes, that’s mandatory - an unexpected deviation from your normal FDA approved processes. And from the reactions I have seen by such companies, those FDA visits are no fun thing to look forward to either, even here in Europe.
mittermayr 3y ago
I sometimes feel people overestimate the ability of doctors as a final decision maker. I used to make voting software for a large medical conference, and spent over a decade or so attending these events. Doctors would often present two (nearly) identical scenarios, i.e. a severe condition a patient has, and then proceed to try two different approaches in treatment. The follow-ups on this are presented at these conferences, to educate others and continue figuring out which procedures may be better in certain scenarios. The most shocking part about this all, to me, was the fact that they often just have their "own opinion" about the best treatment, only to get lectured (if not roasted) shortly after (in the Q&As) about why the method was long outdated, or not a good choice because of the patient showing X, Y and Z, etc. — all while two photos are up on the projector, of two children, one with "less than ideal results" and one with "perfect recovery" — two stories that could have been identical, but weren't, because we humans need to experiment and figure things out, slowly. And two other doctors present afterwards, doing the exact same thing, slightly differently, and thanking the first person about sharing it so they can also try it next time. Long story short - it felt SUPER ancient and error prone, at the cost of human life (and quality of life), and I often found myself thinking about what would be possible if we could let computers work with collective data and suggest based on probability rather than "winging it based on an (educated) hunch" the doctor has, just to have something to present next year. Side note: they were all absolute experts in their fields, of course, I am painting this a bit more black than it may be, but I remain quite shocked when I learned how doctors come to decision making in medical procedures.
- pintxo 3y ago
  I had similar experiences working in health tech. I am still shocked to have learned that it can take decades for new (proven) treatment procedures to make it into general application in healthcare.
  - smabie 3y ago
    tbh the same probably is true of tech and any other industry? People generally risk adverse (No one gets fired for buying IBM)
ylluminate 3y ago
And this is a good thing. I know MANY doctors who are getting out of healthcare and going into business in the USA due to the many problems that range from regulation to BU problems to - well, many other issues. People should be able to take risks as long as they know what the risks are and regulatory clamps need to be released.
- RandomLensman 3y ago
  How do you make sure people understand the risks in a more "free for all" world and who pays for when those risks realize?
  - ylluminate 3y ago
    There are many ways to do this - AI naturally facilitates this very thing in a robust manner. Frankly I have used it to create various "studies" with incredible success when you go deep enough with it. What we really need is MedGPT and not regulations - a ChatGPT that is simply highly specialized and gives warnings out the wahoo.
    - RandomLensman 3y ago
      How exactly? How did you test the results from your studies? Even for experts not easy to understand or articulate the risks in a lot of situations. Listing a long set of warnings isn't usually helpful in making an good decision.
      - ylluminate 3y ago
        Navigating the complexities of AI in healthcare isn't about embracing regulation or throwing it to the wind per se; it's about smart integration. "MedGPT" wouldn't just be a tool—it's a leap towards democratizing medical knowledge, with the power to sift through data and present nuanced insights that can guide decisions. There are definitely ways to do this that go far outside the scope of this format of discussion. When it comes to understanding risks, the clarity and depth AI can offer are unparalleled. By leveraging AI, we're not just throwing caution to the wind; we're arming ourselves with a precision tool. Testing? It’s through iterative, real-world application and feedback, refining our approach as we learn. This isn't about replacing human judgment; it's about enhancing it with comprehensive, data-driven insights. Let’s focus on how AI can transform healthcare, driven by innovation and guided by wisdom. The ONLY advantage of going to doctors or large hospitals is that they tend to have more data, but they (the doctors) admit that their hands are tied by so many regulations that they are being prevented from healing people. This has created a very toxic ecosystem for healthcare globally that is not driven by sincere and true interests of healing people. My grandfather convinced me to get out of healthcare due to his many decades in the industry and seeing it just completely disintegrate from circa 1950-2000. He implored me to go into a different field so as to not be caught up in an industry that does not truly cure. I'm grateful for his wisdom. His entire family were doctors and owned a hospital even. He had a very deep and well aged perspective of the development we have seen over the past century.
yungporko 3y ago
i'd genuinely rather take my chances with an unregulated AI than what is currently available in the UK.
- rickdeckard 3y ago
  Frankly, looking at the trajectory of UK healthcare and the huge lack of skilled and well-paid staff, I wouldn't be surprised if the government embraces AI already tomorrow and creates favorable laws for the supplying companies around that, just to score a quick win with the public and press... Something along the lines of: #1: If an AI creates a diagnosis and the AI-model was audited by a doctor, each diagnosis shall be treated like it was made by a doctor. #2: If an AI creates a diagnosis, the issuing doctor's office is eligible for 10% of the contractually agreed insurance compensation-fee.
jpalomaki 3y ago
Should we set a requirement that AI models used in certain decision making have to be "static" and "deterministic"? Meaning it would be possible to inspect and re-run cases through the specific version of model that was in use at certain point in past? You could take the model that was in use 2024-19-02T10:54:00Z, feed in data and get always the same results. Then tinker with the inputs and observe the results to find out for example potential biases or errors.
- nl 3y ago
  Why? Surely the key question is "Is it correct?", not if the same words were used?
  - RandomLensman 3y ago
    In very few situations you get a "yes" or "no" to that kind of question or you have a lot of things that are correct but not all of equal quality etc.
  - bilbo0s 3y ago
    Yep. Far more important to, 1 - get a result consistent with both reasonable diagnosis and standard of care for the condition in question. And, 2 - get the same result given the same input every time. The FDA doesn’t like when you say “amputate the left leg” one day, and then say “amputate the right leg” the next day. Particularly when that happens after you’ve already amputated the left leg and there was nothing wrong with it. Believe it or not, that amputation scenario has happened before. It wasn’t with AI software, but rather imaging software that hosed an orthonormal conversion on a DICOM dataset. But the same rules should obviously apply to AI.
    - RandomLensman 3y ago
      This sounds a bit like using a standard manual (not the wrong leg part, though).
      - bilbo0s 3y ago
hermannj314 3y ago
As long as unregulated AI doesn't spend a decade causing opioid addictions, collude to reduce labor supply of doctors, and cause a decline in lifespan while increasing costs then I'm willing to give it a shot over what regulation has given us.
- sverhagen 3y ago
  Or... the reduction in regulation could rather increase the problems? I'm not definitively saying that'd be the case, but there have been cases in other industries that I've at least heard about, such as the correlation, perhaps causation, between decreased railway regulation and large train crashes and disasters.
- rickdeckard 3y ago
  The beauty of this is, such "unregulated AI" would most likely be created by the companies making huge sums of money in the current flawed regulated market. Why on earth would they fund the development of an AI which serves against their own interest of securing and further growing their business and revenue...? It's more likely that they are such experts already in how these pesky regulations are hindering their prospects that they would jump on the opportunity of using AI to further redefining the rules in their favor...
zb3 3y ago
People are making critical health care decisions, but are unregulated.. Regulate them!
- weikju 3y ago
  That’s right. AI should not be regulated, people and organizations using it should be
  - rickdeckard 3y ago
    I'm sure companies would love that. Business model of big-corp X: "We deploy an AI model to make 100 million health care decisions per day and the AI-model is supervised by a independent certified doctor we contracted. Legally it should be treated like that doctor made those decisions, since he supervised the model. In case of any regulatory breach, the doctor who was hired for the task should be prosecuted, we replace that doctor, tweak the model and keep going. This way we can charge 100 million units of health diagnosis, and based on custom parameters we can decide i.e. which brand of painkiller should be prescribed. If that's not legally abstracted enough: Let's license the model to doctors based on a per-decision-license, allowing them to boost their daily insurance-claims for a small revenue-share, but requiring them to approve each diagnosis in some kind of UI"
    - from-nibly 3y ago
      Just because you hired A doctor doesn't mean your company isn't liable. Otherwise companies would pull this any time they get sued. And in your second case insurance companies wouldn't insure doctors that did that if they were liable. Assuming the diagnosis comes out bad.
      - rickdeckard 3y ago
        In a world where "AI should not be regulated", there are lots of possible corporate constructs where the "people and organizations using them" are decoupled from the entity which owns the AI, securing it from ever becoming liable for its actions. Already today it's really hard to prove liability for a car company providing self-driving services and not just the driver using them. And that area IS REGULATED.
wslh 3y ago
This is an interesting case to analyze. Health web pages has a disclaimer saying something alike: "The information provided on this website is for general informational and educational purposes only and is not a substitute for professional medical advice, diagnosis, or treatment. Always seek the advice of your physician or other qualified health provider with any questions you may have regarding a medical condition. Never disregard professional medical advice or delay in seeking it because of something you have read on this website.". It doesn't matter if it is AI or any other technology the point is about relying on computers instead of humans and breaking the rule number #0. Not saying that AI couldn't be better than humans in the future but saying that you cannot jump over rule #0 easily.
- gambiting 3y ago
  Sure, but even with that disclaimer the website still has to display at least an attempt at not providing you with blatantly false information. If it literally said "in case of a cold, drink a glass of bleach" then no disclaimer would save them in the resulting criminal case(or at least I hope that it wouldn't).
  - wslh 3y ago
    Not saying that the disclaimer is enough, I tried to express the usin AI is a concern because you are bypassing doctors.
drewcoo 3y ago
> Doctors are already using unregulated artificial intelligence tools such as note-taking virtual assistants and predictive software that helps them diagnose and treat diseases. That sounds to me like the doctors are making the decisions. And they are regulated. If they are concerned about malpractice suits due to AI dependence, I'm sure they'll curtail its use or make sure the AI-makers are liable. And that liability is what causes meaningful regulations. That kind of liability is also what separates actual engineering professions from most "software engineering."

35 points by isaacfrond 3y ago

In the EU if a software is influencing diagnosis decisions of a doctor in any way it needs to be registered as a medical device software. Probably a similar regulation can be expected in the United States too.

amelius 3y ago
People already use ChatGPT to determine if a doctor's visit is necessary, so AI is already in the decision chain.
JTyQZSnP3cQGa8B 3y ago
From my own experience, most medical software is sold in the USA and must be certified by the FDA which is equivalent to the CE process.

huijzer 3y ago

Recently my girlfriend was sick and after seeing multiple doctors in Germany who all had long delays, unfriendly assistants, little time, and conflicting conclusions, I have high hopes for AI. It’s easy to improve the situation from where it is now.

smcl 3y ago
I'm sorry but surely it is even easier to see AI worsening the situation? Would you be happier if your girlfriend received the wrong treatment from an unaccountable system based on decisions made for unknowable reasons, as long at was in a timely manner?
- huijzer 3y ago
  Can't the system be held accountable? Unknowable reasons holds for doctors too, by the way.
  - smcl 3y ago
    Well no they're quite famously unaccountable. IBM even had a mantra that stated "A computer can never be held accountable, therefore a computer must never make a management decision". Re the second, if you are prescribed a certain treatment by your doctor, it is entirely possible to find out why that treatment is proposed.

jeanloolz 3y ago

It is regulated in Europe via the AI Act, effective in 2026 (iirc).

pintxo 3y ago
It’s regulated everywhere with sensible health regulation. If you intent to have your software to make decisions about diagnosis or treatment. You are a medical device with all that follows from that.

pintxo 3y ago

This seems to be a rather superficial article: > When the FDA approves drugs and medical devices, it doesn’t need to keep tabs on how they evolve. The FDA clearly keeps an eye on ongoing developments. If your company produces an FDA approved drug, be aware you might get a visit by them to check on you, especially if you reported - oh yes, that’s mandatory - an unexpected deviation from your normal FDA approved processes. And from the reactions I have seen by such companies, those FDA visits are no fun thing to look forward to either, even here in Europe.

mittermayr 3y ago

I sometimes feel people overestimate the ability of doctors as a final decision maker. I used to make voting software for a large medical conference, and spent over a decade or so attending these events. Doctors would often present two (nearly) identical scenarios, i.e. a severe condition a patient has, and then proceed to try two different approaches in treatment. The follow-ups on this are presented at these conferences, to educate others and continue figuring out which procedures may be better in certain scenarios. The most shocking part about this all, to me, was the fact that they often just have their "own opinion" about the best treatment, only to get lectured (if not roasted) shortly after (in the Q&As) about why the method was long outdated, or not a good choice because of the patient showing X, Y and Z, etc. — all while two photos are up on the projector, of two children, one with "less than ideal results" and one with "perfect recovery" — two stories that could have been identical, but weren't, because we humans need to experiment and figure things out, slowly. And two other doctors present afterwards, doing the exact same thing, slightly differently, and thanking the first person about sharing it so they can also try it next time. Long story short - it felt SUPER ancient and error prone, at the cost of human life (and quality of life), and I often found myself thinking about what would be possible if we could let computers work with collective data and suggest based on probability rather than "winging it based on an (educated) hunch" the doctor has, just to have something to present next year. Side note: they were all absolute experts in their fields, of course, I am painting this a bit more black than it may be, but I remain quite shocked when I learned how doctors come to decision making in medical procedures.

pintxo 3y ago
I had similar experiences working in health tech. I am still shocked to have learned that it can take decades for new (proven) treatment procedures to make it into general application in healthcare.
- smabie 3y ago
  tbh the same probably is true of tech and any other industry? People generally risk adverse (No one gets fired for buying IBM)

ylluminate 3y ago

And this is a good thing. I know MANY doctors who are getting out of healthcare and going into business in the USA due to the many problems that range from regulation to BU problems to - well, many other issues. People should be able to take risks as long as they know what the risks are and regulatory clamps need to be released.

RandomLensman 3y ago
How do you make sure people understand the risks in a more "free for all" world and who pays for when those risks realize?
- ylluminate 3y ago
  There are many ways to do this - AI naturally facilitates this very thing in a robust manner. Frankly I have used it to create various "studies" with incredible success when you go deep enough with it. What we really need is MedGPT and not regulations - a ChatGPT that is simply highly specialized and gives warnings out the wahoo.
  - RandomLensman 3y ago
    How exactly? How did you test the results from your studies? Even for experts not easy to understand or articulate the risks in a lot of situations. Listing a long set of warnings isn't usually helpful in making an good decision.
    - ylluminate 3y ago
      Navigating the complexities of AI in healthcare isn't about embracing regulation or throwing it to the wind per se; it's about smart integration. "MedGPT" wouldn't just be a tool—it's a leap towards democratizing medical knowledge, with the power to sift through data and present nuanced insights that can guide decisions. There are definitely ways to do this that go far outside the scope of this format of discussion. When it comes to understanding risks, the clarity and depth AI can offer are unparalleled. By leveraging AI, we're not just throwing caution to the wind; we're arming ourselves with a precision tool. Testing? It’s through iterative, real-world application and feedback, refining our approach as we learn. This isn't about replacing human judgment; it's about enhancing it with comprehensive, data-driven insights. Let’s focus on how AI can transform healthcare, driven by innovation and guided by wisdom. The ONLY advantage of going to doctors or large hospitals is that they tend to have more data, but they (the doctors) admit that their hands are tied by so many regulations that they are being prevented from healing people. This has created a very toxic ecosystem for healthcare globally that is not driven by sincere and true interests of healing people. My grandfather convinced me to get out of healthcare due to his many decades in the industry and seeing it just completely disintegrate from circa 1950-2000. He implored me to go into a different field so as to not be caught up in an industry that does not truly cure. I'm grateful for his wisdom. His entire family were doctors and owned a hospital even. He had a very deep and well aged perspective of the development we have seen over the past century.

yungporko 3y ago

i'd genuinely rather take my chances with an unregulated AI than what is currently available in the UK.

rickdeckard 3y ago
Frankly, looking at the trajectory of UK healthcare and the huge lack of skilled and well-paid staff, I wouldn't be surprised if the government embraces AI already tomorrow and creates favorable laws for the supplying companies around that, just to score a quick win with the public and press... Something along the lines of: #1: If an AI creates a diagnosis and the AI-model was audited by a doctor, each diagnosis shall be treated like it was made by a doctor. #2: If an AI creates a diagnosis, the issuing doctor's office is eligible for 10% of the contractually agreed insurance compensation-fee.

jpalomaki 3y ago

Should we set a requirement that AI models used in certain decision making have to be "static" and "deterministic"? Meaning it would be possible to inspect and re-run cases through the specific version of model that was in use at certain point in past? You could take the model that was in use 2024-19-02T10:54:00Z, feed in data and get always the same results. Then tinker with the inputs and observe the results to find out for example potential biases or errors.

nl 3y ago
Why? Surely the key question is "Is it correct?", not if the same words were used?
- RandomLensman 3y ago
  In very few situations you get a "yes" or "no" to that kind of question or you have a lot of things that are correct but not all of equal quality etc.
- bilbo0s 3y ago
  Yep. Far more important to, 1 - get a result consistent with both reasonable diagnosis and standard of care for the condition in question. And, 2 - get the same result given the same input every time. The FDA doesn’t like when you say “amputate the left leg” one day, and then say “amputate the right leg” the next day. Particularly when that happens after you’ve already amputated the left leg and there was nothing wrong with it. Believe it or not, that amputation scenario has happened before. It wasn’t with AI software, but rather imaging software that hosed an orthonormal conversion on a DICOM dataset. But the same rules should obviously apply to AI.
  - RandomLensman 3y ago
    This sounds a bit like using a standard manual (not the wrong leg part, though).
    - bilbo0s 3y ago

hermannj314 3y ago

As long as unregulated AI doesn't spend a decade causing opioid addictions, collude to reduce labor supply of doctors, and cause a decline in lifespan while increasing costs then I'm willing to give it a shot over what regulation has given us.

sverhagen 3y ago
Or... the reduction in regulation could rather increase the problems? I'm not definitively saying that'd be the case, but there have been cases in other industries that I've at least heard about, such as the correlation, perhaps causation, between decreased railway regulation and large train crashes and disasters.
rickdeckard 3y ago
The beauty of this is, such "unregulated AI" would most likely be created by the companies making huge sums of money in the current flawed regulated market. Why on earth would they fund the development of an AI which serves against their own interest of securing and further growing their business and revenue...? It's more likely that they are such experts already in how these pesky regulations are hindering their prospects that they would jump on the opportunity of using AI to further redefining the rules in their favor...

zb3 3y ago

People are making critical health care decisions, but are unregulated.. Regulate them!

weikju 3y ago
That’s right. AI should not be regulated, people and organizations using it should be
- rickdeckard 3y ago
  I'm sure companies would love that. Business model of big-corp X: "We deploy an AI model to make 100 million health care decisions per day and the AI-model is supervised by a independent certified doctor we contracted. Legally it should be treated like that doctor made those decisions, since he supervised the model. In case of any regulatory breach, the doctor who was hired for the task should be prosecuted, we replace that doctor, tweak the model and keep going. This way we can charge 100 million units of health diagnosis, and based on custom parameters we can decide i.e. which brand of painkiller should be prescribed. If that's not legally abstracted enough: Let's license the model to doctors based on a per-decision-license, allowing them to boost their daily insurance-claims for a small revenue-share, but requiring them to approve each diagnosis in some kind of UI"
  - from-nibly 3y ago
    Just because you hired A doctor doesn't mean your company isn't liable. Otherwise companies would pull this any time they get sued. And in your second case insurance companies wouldn't insure doctors that did that if they were liable. Assuming the diagnosis comes out bad.
    - rickdeckard 3y ago
      In a world where "AI should not be regulated", there are lots of possible corporate constructs where the "people and organizations using them" are decoupled from the entity which owns the AI, securing it from ever becoming liable for its actions. Already today it's really hard to prove liability for a car company providing self-driving services and not just the driver using them. And that area IS REGULATED.

wslh 3y ago

This is an interesting case to analyze. Health web pages has a disclaimer saying something alike: "The information provided on this website is for general informational and educational purposes only and is not a substitute for professional medical advice, diagnosis, or treatment. Always seek the advice of your physician or other qualified health provider with any questions you may have regarding a medical condition. Never disregard professional medical advice or delay in seeking it because of something you have read on this website.". It doesn't matter if it is AI or any other technology the point is about relying on computers instead of humans and breaking the rule number #0. Not saying that AI couldn't be better than humans in the future but saying that you cannot jump over rule #0 easily.

gambiting 3y ago
Sure, but even with that disclaimer the website still has to display at least an attempt at not providing you with blatantly false information. If it literally said "in case of a cold, drink a glass of bleach" then no disclaimer would save them in the resulting criminal case(or at least I hope that it wouldn't).
- wslh 3y ago
  Not saying that the disclaimer is enough, I tried to express the usin AI is a concern because you are bypassing doctors.

drewcoo 3y ago

> Doctors are already using unregulated artificial intelligence tools such as note-taking virtual assistants and predictive software that helps them diagnose and treat diseases. That sounds to me like the doctors are making the decisions. And they are regulated. If they are concerned about malpractice suits due to AI dependence, I'm sure they'll curtail its use or make sure the AI-makers are liable. And that liability is what causes meaningful regulations. That kind of liability is also what separates actual engineering professions from most "software engineering."