4 ms·

Great question! We did our trials at two overseas locations in parallel with doctors. All uses cases were diagnostic for immigration purposes (e.g., detecting T

Great question! We did our trials at two overseas locations in parallel with doctors. All uses cases were diagnostic for immigration purposes (e.g., detecting Tuberculosis and other chest infections at border points of entry). Given the non-medical use -- no liability insurance. No coverage diagnosis. Also given everything was run in parallel, double-blind with doctors also doing reads, no blame had to be absorbed. Once we got out of parallel, still we wouldn't need liability. The importance here was demonstrating efficacy, which we did fantastically well. Once we prove efficacy for multiple use cases, we can at least remove the "oh you computer scientists dont get it" argument and can have adult conversations about how to progress state of the art rather than continue to bleed patients dry. I'll admit there are definitely barriers like what you mention. But those barriers are not some impenetrable force once we break down real issues and deal with them separately and start treating the problem as one we can solve as a society.

readee456 5y ago
I can't help but think some of the barriers here involved proving the software in a situation decidedly different than a clinical setting. I would not be surprised if an immigration medical officer developed different views about diseases than a GP or ER doctor. They're not treating the person, they're not in a doctor-patient relationship with the person, they're not really even "diagnosing" the person, they're just deciding whether they're "too sick" to come into the country. Maybe if the person looks messed up in some other way, their chest x-ray gets interpreted a little more strictly.
- TuringNYC 5y ago
  >> I can't help but think some of the barriers here involved proving the software in a situation decidedly different than a clinical setting. Totally agree. But science moves in baby steps and progress builds on progress. We started ML by doing linear regression. Then we moved onto recognizing digits. Then we moved onto recognizing cats. Suddenly, Google Photos can find a friend of mine from 1994 in images it appears to have automatically sucked up. That is amazing progress. Similarly, our viewpoint as co-founders in the space was to solve a single use-case amazingly well and prove AUC and cost/value metrics. The field wont be moved by me or you, it will be moved by dozens of teams building upon each other.
- dragonwriter 5y ago
  But AI theater being good enough to replace no-stakes (because no one is liable to anyone for any errors, in either direction) medical theater is a step, just not as big a step or relevant to any use case of any importance as being sold upthread
rscho 5y ago
> Once we prove efficacy for multiple use cases, we can at least remove the "oh you computer scientists dont get it" No, you can't. Stating this is a clear proof that you don't understand what you're dealing with. In medical ML/AI, efficacy is not the issue. What you are detecting is not relevant. That's the issue. But I know I won't convince you.
- aeternum 5y ago
  From where does the efficacy come if what you are detecting is irrelevant?
  - rscho 5y ago
    They are detecting what they are testing for. But that's in most cases irrelevant regarding what happens to the patient afterwards, because it's lacking major connexions to the clinical situation that will have to be filled up by a human expert. So it does in fact work. Unfortunately, only in trivial cases.
    - aeternum 5y ago
      Maybe, but then the problem isn't an issue with AI/ML, it's that humans just suck at math. We're terrible at bayesian logic. Especially when it comes to medical tests, and doctors are very guilty of this also, we ignore priors and take what should just be a Bayes factor as the final truth.
      - rscho 5y ago
        We're terrible at bayesian logic all right, but still better than machines lacking most of the data picture. That's why the priority is not to push lab model efficiency but to push for policy changes that encourage sensible gathering of data. And that's _far_ more difficult than theorizing about model efficiency vs. humans.
glitchc 5y ago
So the answer is: zero. Not surprising. Why does it surprise you then that doctors and patients don't take the solution seriously? You don't have any skin in the game! Whereas the doctors are on the hook for malpractice, and as for the patient, well it's life or death for them.

readee456 5y ago

I can't help but think some of the barriers here involved proving the software in a situation decidedly different than a clinical setting. I would not be surprised if an immigration medical officer developed different views about diseases than a GP or ER doctor. They're not treating the person, they're not in a doctor-patient relationship with the person, they're not really even "diagnosing" the person, they're just deciding whether they're "too sick" to come into the country. Maybe if the person looks messed up in some other way, their chest x-ray gets interpreted a little more strictly.

TuringNYC 5y ago
>> I can't help but think some of the barriers here involved proving the software in a situation decidedly different than a clinical setting. Totally agree. But science moves in baby steps and progress builds on progress. We started ML by doing linear regression. Then we moved onto recognizing digits. Then we moved onto recognizing cats. Suddenly, Google Photos can find a friend of mine from 1994 in images it appears to have automatically sucked up. That is amazing progress. Similarly, our viewpoint as co-founders in the space was to solve a single use-case amazingly well and prove AUC and cost/value metrics. The field wont be moved by me or you, it will be moved by dozens of teams building upon each other.
dragonwriter 5y ago
But AI theater being good enough to replace no-stakes (because no one is liable to anyone for any errors, in either direction) medical theater is a step, just not as big a step or relevant to any use case of any importance as being sold upthread

rscho 5y ago

> Once we prove efficacy for multiple use cases, we can at least remove the "oh you computer scientists dont get it" No, you can't. Stating this is a clear proof that you don't understand what you're dealing with. In medical ML/AI, efficacy is not the issue. What you are detecting is not relevant. That's the issue. But I know I won't convince you.

aeternum 5y ago
From where does the efficacy come if what you are detecting is irrelevant?
- rscho 5y ago
  They are detecting what they are testing for. But that's in most cases irrelevant regarding what happens to the patient afterwards, because it's lacking major connexions to the clinical situation that will have to be filled up by a human expert. So it does in fact work. Unfortunately, only in trivial cases.
  - aeternum 5y ago
    Maybe, but then the problem isn't an issue with AI/ML, it's that humans just suck at math. We're terrible at bayesian logic. Especially when it comes to medical tests, and doctors are very guilty of this also, we ignore priors and take what should just be a Bayes factor as the final truth.
    - rscho 5y ago
      We're terrible at bayesian logic all right, but still better than machines lacking most of the data picture. That's why the priority is not to push lab model efficiency but to push for policy changes that encourage sensible gathering of data. And that's _far_ more difficult than theorizing about model efficiency vs. humans.

glitchc 5y ago

So the answer is: zero. Not surprising. Why does it surprise you then that doctors and patients don't take the solution seriously? You don't have any skin in the game! Whereas the doctors are on the hook for malpractice, and as for the patient, well it's life or death for them.