3 ms·

That's not a great definition of parity. We'd want accuracy and specificity numbers linked to outcomes, not concurrence. The times when Watson agrees with docto

That's not a great definition of parity. We'd want accuracy and specificity numbers linked to outcomes, not concurrence. The times when Watson agrees with doctors is effectively irrelevant - results would be the same whether or not he was added. We need to highlight whether, given a disagreement, Watson was better or worse for outcomes.

philipodonnell 8y ago
The trick is whether the cases of disagreement were themselves predictable! Given a fixed number of oncologists and deploying Watson only to support those oncologists, yes, you're correct that its only useful if it outperforms them. But I think of it more like Watson is a single hive-mind team of like a thousand med students near the end of residency: they get most things right but there are a few places where more experienced doctors will be better, though the scale with the hive mind is far higher. You have one of two reactions to that. 1) Hire fewer senior oncologists and have them focus on the more difficult cases and leave the hive-mind to deal with thousands of routine cases, or 2) ignore the hive mind until its literally better than a typical senior oncologist. The medial profession seems to repeat this cycle of "only full doctors can do anything because even seemingly routine cases might be hiding something more serious" to "maybe some routine things can be done by people with less training and full doctors should focus on the more difficult cases". See nurse practitioners, dental assistants, and, in my mind at least where we are going with things like Watson.
- Bartweiss 8y ago
  I think part of the problem for Watson is that it needs someone to gather the data, which is usually a doctor. So if you're pairing each patient with an oncologist for intake anyway, it's not clear that "examination plus enter all data into Watson" is a benefit over "examination plus make a decision". I guess the ideal outcome for Watson (if it doesn't outpace expert oncologists) would be something like "experience nurse practitioner does an exam, and enters data into Watson" or maybe even "special oncology-trained NP does an exam with Watson". The other part I don't know is what oncology accuracy rates look like. If the reason to not majorly expand screening is cost and availability, Watson could be huge. If it's false positives from our existing rate, there's a lot less value.
  - freeone3000 8y ago
    Let's look specifically at mammograms. (Stats from CTFPHC, a division of PHAC, part of the Canadian government.) https://canadiantaskforce.ca/tools-resources/breast-cancer-2/breast-cancer-risks-benefits-age-40-49/ https://canadiantaskforce.ca/tools-resources/breast-cancer-2... Wider screening isn't great. Essentially, people with medical problems self-select pretty alright already, and wide early-screening initiatives for most cancers introduces as much or more false positives (that persist through screening!) than actual cases of cancer it catches - and moreover, it's not even clear that the early screening is effective - the false negative rate is high enough that the overall incidence of advanced cancer is unchanged even with early screening. (Bleyer and Welch, NEJM, 2012) Basically, we need better screening, not more screening at our current levels, and it's not clear whether watson can provide that.
    - Bartweiss 8y ago
      Precisely - since mammography is the standard "counterintuitive Bayes rule" primer, I remembered those numbers don't support wider screening. I'm not discounting the possibility of a useful role for Watson in wider screening, but it's not clear to me where it would be. If it happens after any kind of extensive examination, doctor-hours are being committed regardless and there's little gain. If it happens at a population-level screen like mammograms and colonoscopies, "almost as good as oncologists" isn't enough to add any value.
Dylan16807 8y ago
> The times when Watson agrees with doctors is effectively irrelevant If your question is "can Watson improve treatment over an oncologist?" (ignoring issues of expense and availability), then it doesn't matter. When you're talking about how close it is to "parity", it matters a lot. If Watson exactly matched the oncologist in all but one out of a million cases, and in that millionth case caused the patient to explode, that would be extremely impressive, and Watson would be extremely useful. It would be nonsense to exclude all the matching cases.
- Bartweiss 8y ago
  > When you're talking about how close it is to "parity", it matters a lot. I guess the question there is how valuable 'close to parity' is. If you can treat more patients, or treat them faster, or even much cheaper, then close to parity is a big deal. If examining patients and avoiding destructive false positives are the limiting factors, it's not clear that parity-level decisions are a significant benefit.

philipodonnell 8y ago

The trick is whether the cases of disagreement were themselves predictable! Given a fixed number of oncologists and deploying Watson only to support those oncologists, yes, you're correct that its only useful if it outperforms them. But I think of it more like Watson is a single hive-mind team of like a thousand med students near the end of residency: they get most things right but there are a few places where more experienced doctors will be better, though the scale with the hive mind is far higher. You have one of two reactions to that. 1) Hire fewer senior oncologists and have them focus on the more difficult cases and leave the hive-mind to deal with thousands of routine cases, or 2) ignore the hive mind until its literally better than a typical senior oncologist. The medial profession seems to repeat this cycle of "only full doctors can do anything because even seemingly routine cases might be hiding something more serious" to "maybe some routine things can be done by people with less training and full doctors should focus on the more difficult cases". See nurse practitioners, dental assistants, and, in my mind at least where we are going with things like Watson.

Bartweiss 8y ago
I think part of the problem for Watson is that it needs someone to gather the data, which is usually a doctor. So if you're pairing each patient with an oncologist for intake anyway, it's not clear that "examination plus enter all data into Watson" is a benefit over "examination plus make a decision". I guess the ideal outcome for Watson (if it doesn't outpace expert oncologists) would be something like "experience nurse practitioner does an exam, and enters data into Watson" or maybe even "special oncology-trained NP does an exam with Watson". The other part I don't know is what oncology accuracy rates look like. If the reason to not majorly expand screening is cost and availability, Watson could be huge. If it's false positives from our existing rate, there's a lot less value.
- freeone3000 8y ago
  Let's look specifically at mammograms. (Stats from CTFPHC, a division of PHAC, part of the Canadian government.) https://canadiantaskforce.ca/tools-resources/breast-cancer-2/breast-cancer-risks-benefits-age-40-49/ https://canadiantaskforce.ca/tools-resources/breast-cancer-2... Wider screening isn't great. Essentially, people with medical problems self-select pretty alright already, and wide early-screening initiatives for most cancers introduces as much or more false positives (that persist through screening!) than actual cases of cancer it catches - and moreover, it's not even clear that the early screening is effective - the false negative rate is high enough that the overall incidence of advanced cancer is unchanged even with early screening. (Bleyer and Welch, NEJM, 2012) Basically, we need better screening, not more screening at our current levels, and it's not clear whether watson can provide that.
  - Bartweiss 8y ago
    Precisely - since mammography is the standard "counterintuitive Bayes rule" primer, I remembered those numbers don't support wider screening. I'm not discounting the possibility of a useful role for Watson in wider screening, but it's not clear to me where it would be. If it happens after any kind of extensive examination, doctor-hours are being committed regardless and there's little gain. If it happens at a population-level screen like mammograms and colonoscopies, "almost as good as oncologists" isn't enough to add any value.

Dylan16807 8y ago

> The times when Watson agrees with doctors is effectively irrelevant If your question is "can Watson improve treatment over an oncologist?" (ignoring issues of expense and availability), then it doesn't matter. When you're talking about how close it is to "parity", it matters a lot. If Watson exactly matched the oncologist in all but one out of a million cases, and in that millionth case caused the patient to explode, that would be extremely impressive, and Watson would be extremely useful. It would be nonsense to exclude all the matching cases.

Bartweiss 8y ago
> When you're talking about how close it is to "parity", it matters a lot. I guess the question there is how valuable 'close to parity' is. If you can treat more patients, or treat them faster, or even much cheaper, then close to parity is a big deal. If examining patients and avoiding destructive false positives are the limiting factors, it's not clear that parity-level decisions are a significant benefit.