4 ms·

We were talking not about about clear signs but about small signs that could be artifacts or actual issues. In that case you are going to have to do some compli

We were talking not about about clear signs but about small signs that could be artifacts or actual issues. In that case you are going to have to do some complimentary or repeated tests to verify, so the ML output saying something is an artifact will have no effect on those procedures unless it's 100% accurate. But in general, these are matters of life and death, literally. You're still going to have someone looking at the images and verifying the output. That limits a lot the potential cost benefits of these applications.

klodolph 5y ago
I still don’t understand why decisions about life or death have to involve 100% certainty or 100% correctness. I would think that it’s the opposite, that when the stakes are high, you want anything that improves your odds. If your life were on the line, you’d accept treatments or diagnoses with much less than 100% certainty.
- gjulianm 5y ago
  See my comment below for the same discussion: https://news.ycombinator.com/item?id=27433368 https://news.ycombinator.com/item?id=27433368 My point isn't that you need 100% accuracy. It's that a diagnosis is a process, not a single test. If your model applies to a single step of the process, and if it doesn't remove enough uncertainty about that step, you're still going to continue the diagnostic process and the model is not going to change anything really.
  - klodolph 5y ago
    You said ML will have “no effect on those procedures unless it's 100% accurate.” I disagree with that, and think it’s a bit absurd—the 100% figure is not possible anyways, so you might as well stick some other impossible predicate in there and the statement is equally insightful. E.g., “it will have no effect on outcomes until pigs fly.” There will always be uncertainty, and uncertainty isn’t the only relevant parameter.
    - gjulianm 5y ago
      > There will always be uncertainty, and uncertainty isn’t the only relevant parameter. But it's a really important one and a lot of medical ML research doesn't seem to address in the proper sense. A very simple example: a ML model that classifies lung nodules as benign or malign, with 95% accuracy vs 70% accuracy of regular radiologists. Very good, right? But for actual, real world results, you need to see how the patient outcomes change. If the patients where the model and doctors disagree were going to have extra tests or followup regardless of what the model says, the model is not actually offering anything new despite the increase in accuracy. So no, it's not absurd to say that models that work for a single type of test need to be very, very accurate to actually bring changes to the procedures that are worth the investment. What matters is whether they actually change patient outcomes, and sometimes it seems like the ML researchers barely consider it.

klodolph 5y ago

I still don’t understand why decisions about life or death have to involve 100% certainty or 100% correctness. I would think that it’s the opposite, that when the stakes are high, you want anything that improves your odds. If your life were on the line, you’d accept treatments or diagnoses with much less than 100% certainty.

gjulianm 5y ago
See my comment below for the same discussion: https://news.ycombinator.com/item?id=27433368 https://news.ycombinator.com/item?id=27433368 My point isn't that you need 100% accuracy. It's that a diagnosis is a process, not a single test. If your model applies to a single step of the process, and if it doesn't remove enough uncertainty about that step, you're still going to continue the diagnostic process and the model is not going to change anything really.
- klodolph 5y ago
  You said ML will have “no effect on those procedures unless it's 100% accurate.” I disagree with that, and think it’s a bit absurd—the 100% figure is not possible anyways, so you might as well stick some other impossible predicate in there and the statement is equally insightful. E.g., “it will have no effect on outcomes until pigs fly.” There will always be uncertainty, and uncertainty isn’t the only relevant parameter.
  - gjulianm 5y ago
    > There will always be uncertainty, and uncertainty isn’t the only relevant parameter. But it's a really important one and a lot of medical ML research doesn't seem to address in the proper sense. A very simple example: a ML model that classifies lung nodules as benign or malign, with 95% accuracy vs 70% accuracy of regular radiologists. Very good, right? But for actual, real world results, you need to see how the patient outcomes change. If the patients where the model and doctors disagree were going to have extra tests or followup regardless of what the model says, the model is not actually offering anything new despite the increase in accuracy. So no, it's not absurd to say that models that work for a single type of test need to be very, very accurate to actually bring changes to the procedures that are worth the investment. What matters is whether they actually change patient outcomes, and sometimes it seems like the ML researchers barely consider it.