5 ms·

>nothing about any kind of controls or whether or not this was a one-off performance or if it would work at scale. I thought the 20 million oncology studies Wa

>nothing about any kind of controls or whether or not this was a one-off performance or if it would work at scale. I thought the 20 million oncology studies Watson cross referenced was the control and I arrived at the opposite conclusion as you about the usefulness at scale. For example, I immediately thought that under the Affordable Care Act, the number of patient Electronic Health Records the Government collects eclipses what Watson had to work with in this case so at scale the potential is huge. Admittedly, until reading your comment I did not consider there were likely a number of times Watson misdiagnosed (which seems obvious now). Therefore, it would be interesting to rerun against the Electronic Health Records the Government has collected so far and test if Watson could diagnosis correctly in less attempts than with the 20 million record it used (or what I call the control). Though at that point, I think it would be the patient that is the control.

Noseshine 10y ago
The "control" for the actual (machine) learning may be the test set derived from the available data, but the control of a medical study must be real life (new data) and not old data used for the machine learning algorithm.
- fennecfoxen 10y ago
  You don't need 100% new data to get an idea of performance. It's meaningful (indeed, common) to select some arbitrary percentage of the real-world data as a "training set" then measure the performance of the algorithm on the rest. It's not proof against overfitting, of course, but it helps.
  - apathy 10y ago
    right, but there's a reason we run clinical trials before accepting that something is superior to standard of care. It is really, really hard to manipulate a preregistered clinical trial (not that people don't try) and incredibly easy to manipulate train/test/validate results.

Noseshine 10y ago

The "control" for the actual (machine) learning may be the test set derived from the available data, but the control of a medical study must be real life (new data) and not old data used for the machine learning algorithm.

fennecfoxen 10y ago
You don't need 100% new data to get an idea of performance. It's meaningful (indeed, common) to select some arbitrary percentage of the real-world data as a "training set" then measure the performance of the algorithm on the rest. It's not proof against overfitting, of course, but it helps.
- apathy 10y ago
  right, but there's a reason we run clinical trials before accepting that something is superior to standard of care. It is really, really hard to manipulate a preregistered clinical trial (not that people don't try) and incredibly easy to manipulate train/test/validate results.