6 ms·

Beware explanations from AI in health care

53 points by pcrh 5y ago

MisterTea 5y ago
This AI thing worries me not because of the application but because of how it will make society dumber. There is a lot of potential to help diagnose patients, picking up on patterns humans might not be able to parse. Every human has a different perspective and approaches problems differently. The old adage "two heads are better than one" comes to mind. With AI there is the possibility that just "one head" will make millions of decisions using a single algorithm. Think about that. No team cooperation. No learning from experience. No brain storming. Who's second guessing the AI? Where is the alternate perspective? Where can you go to get a second opinion when the computer has replaced humans? Imagine being misdiagnosed by a computer and the clueless human health care workers don't or won't question it because the computer said so. You'll be one the phone talking to an AI which is looking at records generated by AI full of data compiled by AI. Why pay costly humans when you can replace them with a computer. All of this is going to dehumanize a very human industry (along with everything else.)
- haswell 5y ago
  I have a lot of concerns about AI moving into spaces that involve human life and death. With that said, I think you've highlighted one possible outcome, but it makes a few assumptions that might play out differently depending on how the tech is implemented. Imagine, for sake of argument, that the AI is just "part of the team". It doesn't replace a doctor, it doesn't have authority over doctors, it's just another signal. If applied in this way, then you can still: - Ask for a second opinion - Go to a doctor/team that doesn't rely on AI - Consult a different / (better?) AI I realize you're focusing on an outcome where AI replaces humans in the field, but I'd hope humanity can find an implementation that can both leverage the benefits of the tech while not throwing out the core disciplines of medicine in the process. It's reasonable to be pessimistic about this given the tech-run-amok issues prevalent in 2021, but I still have some hope. Also imagine the possibilities of AIs sharing knowledge with each other. This has its own semi-terrifying implications, but I don't necessarily think we should conclude that the only outcome is a single AI to rule all AIs (and doctors). Of course, I could be wrong as well. But I don't think all roads lead to doom.
- potatoman22 5y ago
  I don't think AI will make us dumber, it will allow us to work faster and more abstractly. Doctors have no incentive to regurgitate an AI prediction; they're the ones liable for care. Doctors will continue to second guess AI while they use its predictions as one of many data points for their decision.
  - Verdex 5y ago
    This is actually kind of what I'm hoping for. The saying is "when you hear hoofbeats think horses not zebras." But that's boring and it turns the doctor into a bored and checked out professional who just constantly sees the same symptoms day in and out and prescribes the same treatment. When something interesting does show up, they're too used to the status quo to actually address the issue. Let the AI prescribe all of the horse treatments. The doctor can then spend their time imagining more fanciful diagnosis that we're centuries away from teaching the AI how to deal with. Then when the primary treatment fails, instead of trying to convince an overworked and checked out doctor that you've got some other problem, they'll actually already have a list of possible alternative issues that they were really hoping they would get to investigate this time around.
- kibblesalad 5y ago
  James Bridle wrote a rather good book titled "New Dark Age" that covers the continual increase in reliance on algorithms and often non-human-verified data in our decision making processes in all aspects of our lives. This impacts scenarios from the individual level when people drive off roads and into hazards following their GPS blindly, all the way to global markets behaving in unexpected ways due to trade algorithms being unable to account for never before seen changes to conditions that don't fit within their existing parameters.
  - Frost1x 5y ago
    There's some similar work that frames some of this around cognitive artifacts. There are processes people perform that create an artifact in their mind they can use for other purposes of problem solving. The model leaves a sort of precense on their consciousness, an artifact,, and allows them to reason in ways aided by those artifacts. When you stop creating and using those artifacts and depend on technology to act in place of them, you lose some creative aspects you would normally be able to connect or utilize in other mental models. For some things I think it's good for technology to act as black boxes we don't have to understand, just know that we can understand it if we need to. Other things are good to understand even if you can automate it. How you choose and differentiate which of those is an incredible challenge in my mind. What do I commit to memory and integrate into my mental model of the world and what do I just reference and use when I need it.
- Verdex 5y ago
  I have a friend and his wife had some weird stomach problems that the local doctors couldn't figure out. It sounds like the doctors basically ignored that a problem existed because they couldn't think of anything. Eventually they had to go to the nearest big city (city they're in now is 60K people; city they went to has 900K people). The doctors there figured it out pretty quickly. And they also did not have good things to say about the local doctors. (paraphrase) "I'm not surprised they didn't find anything. They look for the simple solution they know and if that doesn't work they give up." Additionally, I went to a doctor for a problem in the same big city and he read (and printed out) a webmd article for me. Anyway. So, today people are stuck with human doctors who have no idea how to fix anything but the simple obvious problems. Personally, I wouldn't mind replacing those people with an AI as I'm not sure it's going to make any difference. There's always going to be specialists and if the AI doctor fails you then you can go to the specialist. But just like today, you're going to have to be your own medical researcher / advocate if the service you get doesn't actually fix your problem.
  - disabled 5y ago
    Wait until you have a rare disease before you say something like you want AI help diagnose you (or make decisions about your healthcare for that matter—the US is going to be a mess if they go this route). 7% of the general population collectively has a rare disease, so it’s not uncommon to have a rare disease. Most of them are classified as very rare. I personally have 2 rare diseases affecting my peripheral nervous system, and one of them is very rare. The very rare disease is believed to have also caused my type 1 diabetes. The wild part? The very rare disease, which I had prior to my type 1 diabetes diagnosis, which was also undiagnosed at the time, was blamed as “type 1 diabetes related complications” specifically autonomic neuropathy by human doctors. Do you think AI would get this right? The answer is an obvious no. Also, I am in my early 30s. My endocrinologist was surprised that I was 30 and was making a big deal about it (as in “you survived and you shouldn’t be alive” sense). It was never statistically likely that I would be alive, which is why AI should never be used. So do you think AI should be making health decisions in a healthcare system that already has a ton of human error, and is trained on those errors? I mean, the third leading cause of death in the US is believed to be preventable medical errors: https://www.bmj.com/content/353/bmj.i2139 https://www.bmj.com/content/353/bmj.i2139
- jazzkingrt 5y ago
  I think that a lot, perhaps most, AI diagnoses would involve a feedback loop that tracks the patient's health outcomes. Here are some examples of feedback: - The AI misses a cancer that gets worse and is later detected on a blood test. - The AI detects a cancer on an x-ray but a (more accurate) CT or biopsy indicates that there is no cancer These kinds of feedback loops should allow the clinical performance of AI to be measured and hopefully improved over time.
  - ska 5y ago
    This has been going on for 25ish years now, but the iteration is quite slow and there are systemic barriers to speeding it up.
- civilized 5y ago
  You could say all these same things about any time in human history where some process got automated, standardized, mechanized, or computerized. Indeed, many people did. They're called Luddites. People always get scared that automation will completely remove the human touch from a process. They forget that people will still design, understand, troubleshoot, maintain, improve, be inspired by, learn from, and innovatively transform their new automated devices and processes. The future has always ended up being people and machines working together, and I think it will continue to be that way for a long time. There will not be an AGI that replaces human thinking in our lifetime.
  - UnFleshedOne 5y ago
    Instead of a pile of machines, imagine a vast opaque bureaucracy, where any individual participant doesn't have any decision-making power and must follow a prescribed set of steps. I think this is partially what fuels nightmares of your average modern luddite. If you follow a happy path in such a system, everything is fine. But if you don't, it gets Kafkaesque. Check out a first hand account [0] for flavor. [0] https://slatestarcodex.com/2017/08/29/my-irb-nightmare/ https://slatestarcodex.com/2017/08/29/my-irb-nightmare/
    - civilized 5y ago
      Heartless, rule-based, unappealable decisions have been a feature of human society for as long as there has been organized bureaucratic society. It is not the case that soft-hearted human bureaucrats are all that stands in the way of the Kafkaesque nightmare. If we want better social and political decisions, we have to engage in society and politics. The AI will follow what we decide, as it always does.
- MonaroVXR 5y ago
  >? Imagine being misdiagnosed by a computer and the clueless human health care workers don't or won't question it because the computer said so. You'll be one the phone talking to an AI which is looking at records generated by AI full of data compiled by AI. Why pay costly humans when you can replace them with a computer. Has already been happening.
  - potatoman22 5y ago
    I highly doubt this. Source: work in the industry.
    - MonaroVXR 5y ago
      I see, I reacted on a other comment and I can't delete this one.
andy99 5y ago
> overstates the benefits and undercounts the drawbacks of requiring black-box algorithms to be explainable. I wrote the essay linked below [0] a few months ago. It is very relevant here. I argue that asking ML for explanations forces you to get a dumbed down version of the result, just like asking any expert to explain all the subtlety of what they are doing. Asking for explanations is a kind of micromanagement. There are instances where explanations are important (like research), but much less so in model deployment. The better way is to focus on the results the models provide, and confidence that the model is making supported decisions (i.e. is not extrapolating or predicting on out of distribution data). This is how we would use other kinds of experts - validate their expertise and trust them when they are working in their area. [0] https://www.willows.ai/blog/getting-more https://www.willows.ai/blog/getting-more
- madsbuch 5y ago
  When I meet a new person, it could be in a hiring situation, I would question that person to know how she thinks -- how she arrives at her conclusions. Whenever she has earned the trust of her collaborators, she can work mostly of her intuition. Why should that be different for ML models? I would expect to be able to switch between a result based (intuition) and a more thorough explaining mode (XAI) to assess soundness of reasoning. And then I am also fully aware that complexity is increased when I turn on explanations.
  - andy99 5y ago
    Well I'd argue that if you take this approach to hiring, you can only hire people who share your area of expertise, that thinks the way you do. This is not generally the case with how ML models work. You could create an explainable model that shows you the things you want to see as part of it's decisions, but (a) that won't take advantage of the full strengths on ML, it's more like a hard coded model (b) it just pushes the problem down a level - great, it said it's a dog because it has teeth, how does it know what teeth are, (c) it still is subject failure on out of distribution or extrapolated sample data. Like I say, there is a place for explanations, especially for the ML scientist working on building a model. But for "managing" the model as I imagine one would in a healthcare setting, I think explainability as commonly construed (feature attribution) is the wrong focus
6gvONxR4sf7o 5y ago
> We argue that this consensus, at least as applied to health care, both overstates the benefits and undercounts the drawbacks of requiring black-box algorithms to be explainable. I wouldn't describe this as a drawback of requiring explainability. I'd describe this as drawbacks of today's models and today's explainability methods. Today's explainability tools are pretty nascent and our usual deep learning models know nothing about causation, which handicaps any "why" question. But I'm still pretty strongly in favor of requiring an interrogable answer to "why did you diagnose that," even if that isn't something we really know how to do right yet. The challenge seems to be that we know we want "why" answers, but we don't know how to describe/codify what constitutes an acceptable "why" answer.
- potatoman22 5y ago
  The doctor comes up with the why, the AI serves as one data point for coming to that solution.
- aabaker99 5y ago
  I worked on making a more explainable AI in bio for while. Probably a lot of the unexplainable AI that is being discusses is about deep learning. Some of the biggest advances in deep learning can also be said to have made the models more explainable. Take computer vision, for example. We have great success with convolutional neural networks because these move the models away from unexplainable fully connected layers where the mathematicals naturally produces many neurons with small effect sizes and towards detecting larger coherent structures in the convolutional layers. I agree there is more work to be done and making good mathematical decisions when designing the layers seems like a promising way to go.
neaden 5y ago
Whenver I see something about a black box algo in healthcare it makes me think of this study: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0141357 https://journals.plos.org/plosone/article?id=10.1371/journal... of pigeons trained to read radiology images. Obviously if a doctor was to tell you that the pigeons thought you had cancer you would think it is ridiculous, but if an algorithm said the same thing and was just as likely to be right it comes off as much more credible, yet it might not be any more understandable or effective.
- ninjinxo 5y ago
  Some cultures would take a dog-smell diagnosis over an algorithm; and others would trust a pigeon before a dog, even more if you called them doves.
- aabaker99 5y ago
  Another classic of the genre is the ol' dead fish fMRI analysis: http://prefrontal.org/files/posters/Bennett-Salmon-2009.jpg http://prefrontal.org/files/posters/Bennett-Salmon-2009.jpg
armchairhacker 5y ago
Idk, people have the same weaknesses that we always blame on AI. People have bias, people lack empathy, people make incorrect moral judgements. Doctors have bias. Doctors sometimes get apathetic. Not because they have bad intentions, in fact I bet most doctors really care and really try. But because they're overworked, and they have to care for many patients who they don't personally know, and some of whom get very sick. And because they're only human. In fact, I believe the way most doctor's appointments work is that the doctor gets a list of symptoms, then matches it against a checklist of what they were taught / experience to form a diagnosis. Very similar to an AI algorithm. Of course this isn't as easy as it seems, you can't just Google the symptoms yourself. Doctors' knowledge and experience is very deep, and takes many years to develop. But still, it's not something an AI couldn't eventually automate. Right now I would be skeptical of an AI doctor, just because I don't think current AI has that capability. But seriously, if ML gets better I would trust it better than any real doctor.
- avgDev 5y ago
  The issues with doctors is that many of them are operating on knowledge that is not completely up to date. Doctors have an extremely hard time dealing with complex disorders or even diagnosing them. People go years without a proper diagnosis. I would love for doctors to just take all the information during first visit and then use some kind of search engine/AI that would go and analyze current literature, new research and suggest things to rule out in a cost effective manner while ruling out things that would need immediate diagnosis/treatment such as cancers. Patients also need to be tracked if they agree to it, especially if they have a serious disabling condition. Their sleeping habits, diet and everything they do needs to reviewed in detail but just isn't done. Most of these things are self reported and that can be problematic.
  - bick_nyers 5y ago
    Interesting, so the AI will filter out irrelevant information that the doctor can then focus on. I think from a social perspective that might be hard, because the patient/doctor has to be freely comfortable with the idea that the doctor was not an expert in that sub-area before, and had to recently learn/come up to speed to be able to help. To me it seems like doctors don't want to freely admit that they don't know something (that they can figure out), and patients don't want to think that is the case. Slightly related but there's a website somewhere that you can upload your DNA sequence (in a .txt file!) to and it shows you relevant scientific literature. About 50% of those articles for me told me that I was probably going to develop male pattern baldness in my thirties/forties.
onecommentman 5y ago
I suspect the middle-term role of AI in diagnosis and treatment is as a low-cost consult. The pressure point for docs to use the AI will be legal. During a malpractice case, if there is no documentary evidence the doctor(s) used the standard-of-care AI system during diagnosis and treatment, and the AI system would have recommended an improved diagnosis and treatment with the same patient information, it will not go well for the hospital/clinic/doctor. AI will be a CYA (Cover Your A*s) in their day-to-day work that would hopefully cheaply and effectively save lives while keeping human doctors in the loop. Sort of like chess engines evaluating positions during tournaments…except you’re allowed to cheat and use them in the game itself. The problem then becomes taking a good history, using imaging and tests effectively, and charting all relevant observations. AI and IT can help here too. With the same CYA motivation… The real problem today seems to be that docs need amanuensis-es. Entering the medical data even in electronic records systems seems to be a real challenge to docs. No data…not much an AI can do.

53 points by pcrh 5y ago

This AI thing worries me not because of the application but because of how it will make society dumber. There is a lot of potential to help diagnose patients, picking up on patterns humans might not be able to parse. Every human has a different perspective and approaches problems differently. The old adage "two heads are better than one" comes to mind. With AI there is the possibility that just "one head" will make millions of decisions using a single algorithm. Think about that. No team cooperation. No learning from experience. No brain storming. Who's second guessing the AI? Where is the alternate perspective? Where can you go to get a second opinion when the computer has replaced humans? Imagine being misdiagnosed by a computer and the clueless human health care workers don't or won't question it because the computer said so. You'll be one the phone talking to an AI which is looking at records generated by AI full of data compiled by AI. Why pay costly humans when you can replace them with a computer. All of this is going to dehumanize a very human industry (along with everything else.)

haswell 5y ago
I have a lot of concerns about AI moving into spaces that involve human life and death. With that said, I think you've highlighted one possible outcome, but it makes a few assumptions that might play out differently depending on how the tech is implemented. Imagine, for sake of argument, that the AI is just "part of the team". It doesn't replace a doctor, it doesn't have authority over doctors, it's just another signal. If applied in this way, then you can still: - Ask for a second opinion - Go to a doctor/team that doesn't rely on AI - Consult a different / (better?) AI I realize you're focusing on an outcome where AI replaces humans in the field, but I'd hope humanity can find an implementation that can both leverage the benefits of the tech while not throwing out the core disciplines of medicine in the process. It's reasonable to be pessimistic about this given the tech-run-amok issues prevalent in 2021, but I still have some hope. Also imagine the possibilities of AIs sharing knowledge with each other. This has its own semi-terrifying implications, but I don't necessarily think we should conclude that the only outcome is a single AI to rule all AIs (and doctors). Of course, I could be wrong as well. But I don't think all roads lead to doom.
potatoman22 5y ago
I don't think AI will make us dumber, it will allow us to work faster and more abstractly. Doctors have no incentive to regurgitate an AI prediction; they're the ones liable for care. Doctors will continue to second guess AI while they use its predictions as one of many data points for their decision.
- Verdex 5y ago
  This is actually kind of what I'm hoping for. The saying is "when you hear hoofbeats think horses not zebras." But that's boring and it turns the doctor into a bored and checked out professional who just constantly sees the same symptoms day in and out and prescribes the same treatment. When something interesting does show up, they're too used to the status quo to actually address the issue. Let the AI prescribe all of the horse treatments. The doctor can then spend their time imagining more fanciful diagnosis that we're centuries away from teaching the AI how to deal with. Then when the primary treatment fails, instead of trying to convince an overworked and checked out doctor that you've got some other problem, they'll actually already have a list of possible alternative issues that they were really hoping they would get to investigate this time around.
kibblesalad 5y ago
James Bridle wrote a rather good book titled "New Dark Age" that covers the continual increase in reliance on algorithms and often non-human-verified data in our decision making processes in all aspects of our lives. This impacts scenarios from the individual level when people drive off roads and into hazards following their GPS blindly, all the way to global markets behaving in unexpected ways due to trade algorithms being unable to account for never before seen changes to conditions that don't fit within their existing parameters.
- Frost1x 5y ago
  There's some similar work that frames some of this around cognitive artifacts. There are processes people perform that create an artifact in their mind they can use for other purposes of problem solving. The model leaves a sort of precense on their consciousness, an artifact,, and allows them to reason in ways aided by those artifacts. When you stop creating and using those artifacts and depend on technology to act in place of them, you lose some creative aspects you would normally be able to connect or utilize in other mental models. For some things I think it's good for technology to act as black boxes we don't have to understand, just know that we can understand it if we need to. Other things are good to understand even if you can automate it. How you choose and differentiate which of those is an incredible challenge in my mind. What do I commit to memory and integrate into my mental model of the world and what do I just reference and use when I need it.
Verdex 5y ago
I have a friend and his wife had some weird stomach problems that the local doctors couldn't figure out. It sounds like the doctors basically ignored that a problem existed because they couldn't think of anything. Eventually they had to go to the nearest big city (city they're in now is 60K people; city they went to has 900K people). The doctors there figured it out pretty quickly. And they also did not have good things to say about the local doctors. (paraphrase) "I'm not surprised they didn't find anything. They look for the simple solution they know and if that doesn't work they give up." Additionally, I went to a doctor for a problem in the same big city and he read (and printed out) a webmd article for me. Anyway. So, today people are stuck with human doctors who have no idea how to fix anything but the simple obvious problems. Personally, I wouldn't mind replacing those people with an AI as I'm not sure it's going to make any difference. There's always going to be specialists and if the AI doctor fails you then you can go to the specialist. But just like today, you're going to have to be your own medical researcher / advocate if the service you get doesn't actually fix your problem.
- disabled 5y ago
  Wait until you have a rare disease before you say something like you want AI help diagnose you (or make decisions about your healthcare for that matter—the US is going to be a mess if they go this route). 7% of the general population collectively has a rare disease, so it’s not uncommon to have a rare disease. Most of them are classified as very rare. I personally have 2 rare diseases affecting my peripheral nervous system, and one of them is very rare. The very rare disease is believed to have also caused my type 1 diabetes. The wild part? The very rare disease, which I had prior to my type 1 diabetes diagnosis, which was also undiagnosed at the time, was blamed as “type 1 diabetes related complications” specifically autonomic neuropathy by human doctors. Do you think AI would get this right? The answer is an obvious no. Also, I am in my early 30s. My endocrinologist was surprised that I was 30 and was making a big deal about it (as in “you survived and you shouldn’t be alive” sense). It was never statistically likely that I would be alive, which is why AI should never be used. So do you think AI should be making health decisions in a healthcare system that already has a ton of human error, and is trained on those errors? I mean, the third leading cause of death in the US is believed to be preventable medical errors: https://www.bmj.com/content/353/bmj.i2139 https://www.bmj.com/content/353/bmj.i2139
jazzkingrt 5y ago
I think that a lot, perhaps most, AI diagnoses would involve a feedback loop that tracks the patient's health outcomes. Here are some examples of feedback: - The AI misses a cancer that gets worse and is later detected on a blood test. - The AI detects a cancer on an x-ray but a (more accurate) CT or biopsy indicates that there is no cancer These kinds of feedback loops should allow the clinical performance of AI to be measured and hopefully improved over time.
- ska 5y ago
  This has been going on for 25ish years now, but the iteration is quite slow and there are systemic barriers to speeding it up.
civilized 5y ago
You could say all these same things about any time in human history where some process got automated, standardized, mechanized, or computerized. Indeed, many people did. They're called Luddites. People always get scared that automation will completely remove the human touch from a process. They forget that people will still design, understand, troubleshoot, maintain, improve, be inspired by, learn from, and innovatively transform their new automated devices and processes. The future has always ended up being people and machines working together, and I think it will continue to be that way for a long time. There will not be an AGI that replaces human thinking in our lifetime.
- UnFleshedOne 5y ago
  Instead of a pile of machines, imagine a vast opaque bureaucracy, where any individual participant doesn't have any decision-making power and must follow a prescribed set of steps. I think this is partially what fuels nightmares of your average modern luddite. If you follow a happy path in such a system, everything is fine. But if you don't, it gets Kafkaesque. Check out a first hand account [0] for flavor. [0] https://slatestarcodex.com/2017/08/29/my-irb-nightmare/ https://slatestarcodex.com/2017/08/29/my-irb-nightmare/
  - civilized 5y ago
    Heartless, rule-based, unappealable decisions have been a feature of human society for as long as there has been organized bureaucratic society. It is not the case that soft-hearted human bureaucrats are all that stands in the way of the Kafkaesque nightmare. If we want better social and political decisions, we have to engage in society and politics. The AI will follow what we decide, as it always does.
MonaroVXR 5y ago
>? Imagine being misdiagnosed by a computer and the clueless human health care workers don't or won't question it because the computer said so. You'll be one the phone talking to an AI which is looking at records generated by AI full of data compiled by AI. Why pay costly humans when you can replace them with a computer. Has already been happening.
- potatoman22 5y ago
  I highly doubt this. Source: work in the industry.
  - MonaroVXR 5y ago
    I see, I reacted on a other comment and I can't delete this one.

andy99 5y ago

> overstates the benefits and undercounts the drawbacks of requiring black-box algorithms to be explainable. I wrote the essay linked below [0] a few months ago. It is very relevant here. I argue that asking ML for explanations forces you to get a dumbed down version of the result, just like asking any expert to explain all the subtlety of what they are doing. Asking for explanations is a kind of micromanagement. There are instances where explanations are important (like research), but much less so in model deployment. The better way is to focus on the results the models provide, and confidence that the model is making supported decisions (i.e. is not extrapolating or predicting on out of distribution data). This is how we would use other kinds of experts - validate their expertise and trust them when they are working in their area. [0] https://www.willows.ai/blog/getting-more https://www.willows.ai/blog/getting-more

madsbuch 5y ago
When I meet a new person, it could be in a hiring situation, I would question that person to know how she thinks -- how she arrives at her conclusions. Whenever she has earned the trust of her collaborators, she can work mostly of her intuition. Why should that be different for ML models? I would expect to be able to switch between a result based (intuition) and a more thorough explaining mode (XAI) to assess soundness of reasoning. And then I am also fully aware that complexity is increased when I turn on explanations.
- andy99 5y ago
  Well I'd argue that if you take this approach to hiring, you can only hire people who share your area of expertise, that thinks the way you do. This is not generally the case with how ML models work. You could create an explainable model that shows you the things you want to see as part of it's decisions, but (a) that won't take advantage of the full strengths on ML, it's more like a hard coded model (b) it just pushes the problem down a level - great, it said it's a dog because it has teeth, how does it know what teeth are, (c) it still is subject failure on out of distribution or extrapolated sample data. Like I say, there is a place for explanations, especially for the ML scientist working on building a model. But for "managing" the model as I imagine one would in a healthcare setting, I think explainability as commonly construed (feature attribution) is the wrong focus

6gvONxR4sf7o 5y ago

> We argue that this consensus, at least as applied to health care, both overstates the benefits and undercounts the drawbacks of requiring black-box algorithms to be explainable. I wouldn't describe this as a drawback of requiring explainability. I'd describe this as drawbacks of today's models and today's explainability methods. Today's explainability tools are pretty nascent and our usual deep learning models know nothing about causation, which handicaps any "why" question. But I'm still pretty strongly in favor of requiring an interrogable answer to "why did you diagnose that," even if that isn't something we really know how to do right yet. The challenge seems to be that we know we want "why" answers, but we don't know how to describe/codify what constitutes an acceptable "why" answer.

potatoman22 5y ago
The doctor comes up with the why, the AI serves as one data point for coming to that solution.
aabaker99 5y ago
I worked on making a more explainable AI in bio for while. Probably a lot of the unexplainable AI that is being discusses is about deep learning. Some of the biggest advances in deep learning can also be said to have made the models more explainable. Take computer vision, for example. We have great success with convolutional neural networks because these move the models away from unexplainable fully connected layers where the mathematicals naturally produces many neurons with small effect sizes and towards detecting larger coherent structures in the convolutional layers. I agree there is more work to be done and making good mathematical decisions when designing the layers seems like a promising way to go.

neaden 5y ago

Whenver I see something about a black box algo in healthcare it makes me think of this study: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0141357 https://journals.plos.org/plosone/article?id=10.1371/journal... of pigeons trained to read radiology images. Obviously if a doctor was to tell you that the pigeons thought you had cancer you would think it is ridiculous, but if an algorithm said the same thing and was just as likely to be right it comes off as much more credible, yet it might not be any more understandable or effective.

ninjinxo 5y ago
Some cultures would take a dog-smell diagnosis over an algorithm; and others would trust a pigeon before a dog, even more if you called them doves.
aabaker99 5y ago
Another classic of the genre is the ol' dead fish fMRI analysis: http://prefrontal.org/files/posters/Bennett-Salmon-2009.jpg http://prefrontal.org/files/posters/Bennett-Salmon-2009.jpg

armchairhacker 5y ago

Idk, people have the same weaknesses that we always blame on AI. People have bias, people lack empathy, people make incorrect moral judgements. Doctors have bias. Doctors sometimes get apathetic. Not because they have bad intentions, in fact I bet most doctors really care and really try. But because they're overworked, and they have to care for many patients who they don't personally know, and some of whom get very sick. And because they're only human. In fact, I believe the way most doctor's appointments work is that the doctor gets a list of symptoms, then matches it against a checklist of what they were taught / experience to form a diagnosis. Very similar to an AI algorithm. Of course this isn't as easy as it seems, you can't just Google the symptoms yourself. Doctors' knowledge and experience is very deep, and takes many years to develop. But still, it's not something an AI couldn't eventually automate. Right now I would be skeptical of an AI doctor, just because I don't think current AI has that capability. But seriously, if ML gets better I would trust it better than any real doctor.

avgDev 5y ago
The issues with doctors is that many of them are operating on knowledge that is not completely up to date. Doctors have an extremely hard time dealing with complex disorders or even diagnosing them. People go years without a proper diagnosis. I would love for doctors to just take all the information during first visit and then use some kind of search engine/AI that would go and analyze current literature, new research and suggest things to rule out in a cost effective manner while ruling out things that would need immediate diagnosis/treatment such as cancers. Patients also need to be tracked if they agree to it, especially if they have a serious disabling condition. Their sleeping habits, diet and everything they do needs to reviewed in detail but just isn't done. Most of these things are self reported and that can be problematic.
- bick_nyers 5y ago
  Interesting, so the AI will filter out irrelevant information that the doctor can then focus on. I think from a social perspective that might be hard, because the patient/doctor has to be freely comfortable with the idea that the doctor was not an expert in that sub-area before, and had to recently learn/come up to speed to be able to help. To me it seems like doctors don't want to freely admit that they don't know something (that they can figure out), and patients don't want to think that is the case. Slightly related but there's a website somewhere that you can upload your DNA sequence (in a .txt file!) to and it shows you relevant scientific literature. About 50% of those articles for me told me that I was probably going to develop male pattern baldness in my thirties/forties.

onecommentman 5y ago

I suspect the middle-term role of AI in diagnosis and treatment is as a low-cost consult. The pressure point for docs to use the AI will be legal. During a malpractice case, if there is no documentary evidence the doctor(s) used the standard-of-care AI system during diagnosis and treatment, and the AI system would have recommended an improved diagnosis and treatment with the same patient information, it will not go well for the hospital/clinic/doctor. AI will be a CYA (Cover Your A*s) in their day-to-day work that would hopefully cheaply and effectively save lives while keeping human doctors in the loop. Sort of like chess engines evaluating positions during tournaments…except you’re allowed to cheat and use them in the game itself. The problem then becomes taking a good history, using imaging and tests effectively, and charting all relevant observations. AI and IT can help here too. With the same CYA motivation… The real problem today seems to be that docs need amanuensis-es. Entering the medical data even in electronic records systems seems to be a real challenge to docs. No data…not much an AI can do.