3 ms·

> On a recent morning, the results for a 73-year-old lung cancer patient were underwhelming: Watson recommended a chemotherapy regimen the oncologists had alrea

by yread 9y ago

> On a recent morning, the results for a 73-year-old lung cancer patient were underwhelming: Watson recommended a chemotherapy regimen the oncologists had already flagged. > “It’s fine,” Dr. Sujal Shah, a medical oncologist, said of Watson’s treatment suggestion while discussing the case with colleagues. > He said later that the background information Watson provided, including medical journal articles, was helpful, giving him more confidence that using a specific chemotherapy was a sound idea. But the system did not directly help him make that decision, nor did it tell him anything he didn’t already know. I don't know, that doesn't sound bad at all. Perhaps it's just a case of overhyping. But if a computer can do recommend the same treatment as an expensive oncologist with years of training that's actually pretty good! On the other hand it is quite baffling why doesn't FDA require a clinical trial or any independent review of its capabilities before allowing its use in clinic. It's a bit like Boeing certifying its own planes, I guess.

jessriedel 9y ago
> why doesn't FDA require a clinical trial or any independent review of its capabilities before allowing its use in clinic. Because it's a tool for informing doctors, not a medical treatment. We don't have a clinical trial before PubMed rolls out a new search form either.
- dekhn 9y ago
  It's definitely part of medical treatment. Laws are changing to include "software medical devices" in health regulation. The FDA recently recognized this as a gap. BTW I've worked with regulated software and it's not pretty. Just to update the firmware you have to fill out a ton of paperwork (because somebody died once after a firmware update and the firmware update was found to be the root cause). Regulation and review doesn't really make things tremendously safer, it just provides an audit chain so you can point at something when it does go pear-shape
  - jessriedel 9y ago
    I can't tell if you're arguing that it is currently regulated, that it will be, or that it fits some platonic notion of 'medical treatment' and therefore should be (even if it's not beneficial).
  - ska 9y ago
    Laws are changing to include "software medical devices" in health regulation. The FDA recently recognized this as a gap. This isn't quite correct - FDA has always regulated software medical devices. It is true however that the FDA is looking a lot closer at software than they did in previous decades, and is releasing guidance (e.g. recently on cybersecurity) that signals intentions moving forward. They definitely have signaled that they don't believe they were focused enough on software. It's important understand that FDA approval to market is always done in terms of indications for use. You don't approve a device full stop, you approve it (and evaluate it) in terms of the indications. Only in this context can you determine if you have a 510k path (i.e. short cut) or need a PMA (i.e. clinical trial). It is also important to realize that different panels in the FDA (i.e. the type of device you are making) have different evaluation criteria. So "Watson" is unlikely to ever be cleared per se (it is a tech platform) but you are likely to see individual Watson enabled systems file 510ks for particular indications. If I recall correctly this has already happened.
    - ltowel 9y ago
      Do you remember what it's already happened for? Do you know what devices they're equivalent to? While I get the impression FDA and industry prefer 510ks as much as possible, personally I don't see how this is close enough to anything out there right now for that to pass the smell test. My guess would be that some `Watson` equivalent stuff will be 510k'd in as part of an MRI, CT or DBT imaging system or a CAD tool working on these systems. I think it'd take a while though, I don't think the regulatory science is there yet for actually showing value.
      - ska 9y ago
        Off the top of my head, I don't, but I'll have a look if I have a chance later. There have been approved CAD systems separate from imaging systems since the 90s. "Value" is a nebulous term. FDA looks specifically for "safety & efficacy", and the criterion for each are rather dependent on the intended use.
- deleted 9y ago
  [deleted]
mindcrime 9y ago
I've seen my doctor consulting Wikipedia before, along with other online resources. I don't think the FDA regulates that type of thing because it's just information, which the doctor is free to use or discard. I would think Watson or any other kind of medical diagnosis system would fall into the same category, as long as the human doctor is making the final decisions.

by yread 9y ago

jessriedel 9y ago

> why doesn't FDA require a clinical trial or any independent review of its capabilities before allowing its use in clinic. Because it's a tool for informing doctors, not a medical treatment. We don't have a clinical trial before PubMed rolls out a new search form either.

dekhn 9y ago
It's definitely part of medical treatment. Laws are changing to include "software medical devices" in health regulation. The FDA recently recognized this as a gap. BTW I've worked with regulated software and it's not pretty. Just to update the firmware you have to fill out a ton of paperwork (because somebody died once after a firmware update and the firmware update was found to be the root cause). Regulation and review doesn't really make things tremendously safer, it just provides an audit chain so you can point at something when it does go pear-shape
- jessriedel 9y ago
  I can't tell if you're arguing that it is currently regulated, that it will be, or that it fits some platonic notion of 'medical treatment' and therefore should be (even if it's not beneficial).
- ska 9y ago
  Laws are changing to include "software medical devices" in health regulation. The FDA recently recognized this as a gap. This isn't quite correct - FDA has always regulated software medical devices. It is true however that the FDA is looking a lot closer at software than they did in previous decades, and is releasing guidance (e.g. recently on cybersecurity) that signals intentions moving forward. They definitely have signaled that they don't believe they were focused enough on software. It's important understand that FDA approval to market is always done in terms of indications for use. You don't approve a device full stop, you approve it (and evaluate it) in terms of the indications. Only in this context can you determine if you have a 510k path (i.e. short cut) or need a PMA (i.e. clinical trial). It is also important to realize that different panels in the FDA (i.e. the type of device you are making) have different evaluation criteria. So "Watson" is unlikely to ever be cleared per se (it is a tech platform) but you are likely to see individual Watson enabled systems file 510ks for particular indications. If I recall correctly this has already happened.
  - ltowel 9y ago
    Do you remember what it's already happened for? Do you know what devices they're equivalent to? While I get the impression FDA and industry prefer 510ks as much as possible, personally I don't see how this is close enough to anything out there right now for that to pass the smell test. My guess would be that some `Watson` equivalent stuff will be 510k'd in as part of an MRI, CT or DBT imaging system or a CAD tool working on these systems. I think it'd take a while though, I don't think the regulatory science is there yet for actually showing value.
    - ska 9y ago
      Off the top of my head, I don't, but I'll have a look if I have a chance later. There have been approved CAD systems separate from imaging systems since the 90s. "Value" is a nebulous term. FDA looks specifically for "safety & efficacy", and the criterion for each are rather dependent on the intended use.
deleted 9y ago
[deleted]

mindcrime 9y ago

I've seen my doctor consulting Wikipedia before, along with other online resources. I don't think the FDA regulates that type of thing because it's just information, which the doctor is free to use or discard. I would think Watson or any other kind of medical diagnosis system would fall into the same category, as long as the human doctor is making the final decisions.