8 ms·

AI Models Predict Breast Cancer with Radiologist-Level Accuracy

146 points by tiagobraw 7y ago

mtgx 7y ago
What's the False Positive Rate?
- ska 7y ago
  FP = system (or person) flags this as true when it is not TP = ... flags as true and it is FN = ... flags as false but it is true TN = ... flags as false and it is false To turn these into rates, you normalize them. e.g. TPR = TP/P = TP/(TP + FN) = 1 - FNR etc. These are characteristics of a classification system You will also hear sensitivity (TPR) vs. specificity (TNR) often, particularly in medical contexts. In other contexts you'll hear Type I (FP) vs. Type II (FN) error. In most cases you a set of trade offs in your algorithm, and will need to pick a balance between sensitivity and specificity. c.f. ROC: https://en.wikipedia.org/wiki/Receiver_operating_characteristic https://en.wikipedia.org/wiki/Receiver_operating_characteris...
  - adyavanapalli 7y ago
    I think the OP is asking about the _value_ of the FPR instead of the definition.
    - RosanaAnaDana 7y ago
      The value of the false positive rate is that it lets you know the probability of a true-miss. Depending on the classification exercise, you may be concerned with false positives, where the consequence of a missed call is significantly greater than an unwarranted checkup from a human doctor.
      - shawnz 7y ago
        The numeric value
    - ska 7y ago
      Ah, if I misread then from the results section of the linked paper: For the malignancy prediction objective, the algorithm obtained an area under the receiver operating characteristic curve (AUC) of 0.91 (95% CI: 0.89, 0.93), with specificity of 77.3% (95% CI: 69.2%, 85.4%) at a sensitivity of 87%. I haven't read the papers methods, but the data set size is small-ish for this sort of analysis.
- TuringNYC 7y ago
  from the paper: https://pubs.rsna.org/doi/10.1148/radiol.2019182622 https://pubs.rsna.org/doi/10.1148/radiol.2019182622 AUC is 0.78 Sensitivity-Specificity Graph is here: http://images.rsna.org/index.html?doi=10.1148/radiol.2019182622&fig=fig3 http://images.rsna.org/index.html?doi=10.1148/radiol.2019182...
  - mdorazio 7y ago
    Thanks for linking. If I'm reading that correctly, it's pretty bad in comparison to an average human radiologist at ~6% false positive rate [1]. There's probably a bias factor in there, however, where humans are hesitant to predict potential cancer due to the cost/time involved in follow-up screening. [1] https://www.ncbi.nlm.nih.gov/pubmed/21643887 https://www.ncbi.nlm.nih.gov/pubmed/21643887
NikolaeVarius 7y ago
Is this new AI Model under Watson?
- layoutIfNeeded 7y ago
  There’s no such thing as “Watson”. IBM have put the Watson name on basically everything, to the point where its information content was reduced to zero bits. Watson for IBM is like the i-prefix for Apple.
- noelsusman 7y ago
  Watson is a brand, so that doesn't really mean anything. If Watson refers to anything it would be the NLP functionality that IBM sells, and that's not relevant here.
btilly 7y ago
This is not exactly new. I remember seeing models that did really well many years ago. And again caught many that humans had miss. The problem is that they fail differently than humans do, in a way that humans wind up not trusting the results. It turns out that there are parts of the breast that are easy to spot tumors in, and parts that are hard. A human scans quickly over the easy areas, and focuses on the hard. The result is that humans make careless errors on the easy areas, and catch hard tumors. Computers make no careless errors, but can't catch the hard ones. Thus when a human sees what the computer caught that the human did not, the mistake is easily dismissed. But when the human sees the ones that the computer missed, it becomes, "It doesn't know how to do the real work." Ideally the two would be used together for better results than either alone. But humans wind up resenting the computer...
- hathawsh 7y ago
  That's a helpful perspective. Would it be possible for IBM to create a service that allows patients to submit their own pictures for scanning?
  - lostlogin 7y ago
    We are at a tricky stage with this. Images are too big to be emailed and many people no longer have CD drives. There is increasing use of tomographic imaging in examinations too, and the files are pretty big.
    - hathawsh 7y ago
      On the other hand, cheap micro SD cards and thumb drives hold far more than CDs or DVDs.
      - lostlogin 7y ago
        We prefer to send the images to the PACS the patient wants them on, as for most imaging that’s all that’s wanted. Obstetric imaging is one massive exception.
deleted 7y ago
[deleted]
rayuela 7y ago
Anything related to AI coming out of IBM should be viewed with a huge dose of skepticism. They're honestly one of the worst offenders in overselling the capabilities of their products, bordering out outright fraud. There is certainly a lot of promise to the application of recent computer vision algos on medical imaging data, but I wouldn't bet much on IBM being anything close to a leader in this space.
- Myrmornis 7y ago
  I don't doubt what you say. Just want to point out that this is published in a peer-reviewed journal, so hopefully the academic community will judge it objectively. https://pubs.rsna.org/doi/10.1148/radiol.2019182622 https://pubs.rsna.org/doi/10.1148/radiol.2019182622
  - dragandj 7y ago
    OTOH, do even radiologists (or anyone else) can predict cancer at all before it happens? I thought that radiologists diagnose cancer once it is already there.
    - the8472 7y ago
      not all tumors are malignant
    - thaumasiotes 7y ago
      > do even radiologists (or anyone else) can predict cancer at all before it happens? Sure, some of the time it's easy. Let's all recall the words of my mother's medical school instructor, "There's a bit of cancer in everyone's prostate". (The context was a lab exercise in which medical students were supposed to find which of a set of slides of prostate tissue was cancerous. The reminder was necessary because many of the slides were cancerous, just not at levels high enough to be considered medically alarming.) Predicting that a man will develop prostate cancer is basically the same thing as predicting that he'll experience old age.
nkurz 7y ago
What makes this an "AI Model" instead of just a "Model"? That is, in what way does it have "artificial intelligence"?
- amelius 7y ago
  Probably because it was obtained using some kind of machine learning.
- TuringNYC 7y ago
  "Model" ---> [[marketing department]] --> "AI Model"
  - raxxorrax 7y ago
    Can I call myself an AI specialist if I successfully fed a plain support vector machine once or twice for diagnostic support? Feels like driving an old timer here...
  - avgDev 7y ago
    [Student in school] Implemented MinMax algorithm in checkers ---> [student looking for work] Implemented state of the art AI algorithm, which successfully will beat the human opponent EVERY time. ---> [HR/Marketing dept at some corp] Wow you are HIRED!!!!!!!! ---> [Lead dev] Oof this guy can't program for shit.
    - TeMPOraL 7y ago
      ---> [student about to get fired from work] Why on Earth did they put "AI expertise" in the job requirements if all they want me to do is to shovel CSS and JS, and the closest thing to AI they have in the office is a 1960s thermostat?
- onemoresoop 7y ago
  Yes, the AI part is marketing/fluff
- Myrmornis 7y ago
dontreact 7y ago
The reason I'm skeptical of this is that there is no actual comparison to human level performance. I.E. they didn't have radiologist actually read their images to compare against the model. Notice that the title of the paper is "Predicting Breast Cancer by Applying Deep Learning to Linked Health Records and Mammograms" it's only in the press release that they seem to imply a comparison to radiologists was actually done.
- thatcantbeit 7y ago
  https://pubs.rsna.org/doi/full/10.1148/radiol.2016161174 https://pubs.rsna.org/doi/full/10.1148/radiol.2016161174 This is their comparison point for actual radiologists. Citation number 6. It doesn't look comparable, though. Radiologists are around 90% specificity and sensitivity, which varies a good amount from the model's 77.3% and 87%, respectively.
  - dontreact 7y ago
    This is not on this dataset though (right?), so not really a solid comparison point. Plus lik you mentioned, they seem to be doing worse than this benchmark.
baybal2 7y ago
I was lucky to date a girl who was into math, and who was coding those "machine learning" algorithms for a radiology startup here in Shenzhen. She had a lot of scepticism for what she did. One of biggest showstoppers she said was the unpredictability of errors. An algo can catch 99% tumors, including tiny ones, bur can randomly pass over very obvious ones which a human radiologist will spot with his eyes closed. They had a demo day with radiologists, and them throwing tricky edge case xrays at the computer. Edge cases were all ok, but one radiologist pulled his own xray from his bag, with a 100% obvious, terminal stage tumor, and to company's embarrassment, the algo failed to detect it no mater how they twisted and scaled the xray. The guy then just walked out.
- WalterBright 7y ago
  It seems to me the use case should be to have the radiologist look at a scan for tumors. Then, the algo should look. If they disagree, then the radiologist should look at the difference. It'll be the best of both. And in the scans where the algo is wrong, have the scan added to the machine learning database of the algo.
  - ttlei 7y ago
    If the radiologist has to look at and double check every scan that algo looked at, then what is the point of the algo? Seems like a useless middleman that get in the way.
    - navigatesol 7y ago
      >then what is the point of the algo? The point is that the algorithm can improve results. This isn't ad placement, it's peoples' lives. Checking and double checking should be the norm.
    - bradstewart 7y ago
      Because the scan check by the radiologist becomes a _double_ check.
    - ska 7y ago
mlcrypto 7y ago
Doctors will be some of the first to be replaced by AI. My physicians walk around with a computer already checking all the boxes for symptoms and seeing what it says. I wish I could find one with a true intuition for medicine
- caraffle 7y ago
  Apparently you're not familiar with the documentation burden in the medical field. EHR's don't diagnose for you. There is no "true" intuition in medicine, just years of study and practice leading to quick recognition of common problems like any other field.
Myrmornis 7y ago
@moderators: would it make sense to change the link to the journal article rather than IBM's article? It's free access. https://pubs.rsna.org/doi/10.1148/radiol.2019182622 https://pubs.rsna.org/doi/10.1148/radiol.2019182622
professorTuring 7y ago
The real problem here is when the society will allow a machine to diagnose them and if the society is ready to believe that most diagnostics are probabilistically made. Up to date we allow humans to be at a 70% error level without problems, but we ask machines to be 100% effective. The very same happens with autopilot, the big numbers say they drive better than humans but...
- petschge 7y ago
  I remember seeing a statistical analysis here on HN that said the numbers for Tesla autopilots are neither great compared to drivers of Teslas nor do they seem to be fair. (They found a case where a human driver had a crash in what would have been counted as 0 miles in the analysis, indicating that something is inflating the "crashes per miles" metric)
  - AstralStorm 7y ago
    Using autopilot in parking?
- anthony_doan 7y ago
  These type of algorithms don't give a confidence interval for their predictions so I don't believe these diagnostics are base on probability at all. Having a confusion matrix for what the model predict correct or not is not the same as having a CI for the model's prediction.
  - zone411 7y ago
    Isn't it possible to derive a CI from the confusion matrix? https://stats.stackexchange.com/questions/363382/confidence-interval-of-precision-recall-and-f1-score https://stats.stackexchange.com/questions/363382/confidence-... or using bootstrap? The authors of this research also provide CIs.
- stubish 7y ago
  This isn't about group think. It is when individuals will allow diagnosis. If you give a woman two options, diagnosis by machine with a record of 80% or a human with a record of 70%, it is a really easy decision to make. The desire to not suffer cancer is strong enough to override almost all emotional arguments. And if you can afford it, you will likely choose both or get a second opinion.
blueyes 7y ago
Old news from a major source of AI hype. Here's some previous results https://med.stanford.edu/news/all-news/2018/11/ai-outperformed-radiologists-in-screening-x-rays-for-certain-diseases.html https://med.stanford.edu/news/all-news/2018/11/ai-outperform...
michaelhoffman 7y ago
No positive predictive value reported, imbalanced test data, IBM. Garbage.

146 points by tiagobraw 7y ago

mtgx 7y ago

What's the False Positive Rate?

ska 7y ago
FP = system (or person) flags this as true when it is not TP = ... flags as true and it is FN = ... flags as false but it is true TN = ... flags as false and it is false To turn these into rates, you normalize them. e.g. TPR = TP/P = TP/(TP + FN) = 1 - FNR etc. These are characteristics of a classification system You will also hear sensitivity (TPR) vs. specificity (TNR) often, particularly in medical contexts. In other contexts you'll hear Type I (FP) vs. Type II (FN) error. In most cases you a set of trade offs in your algorithm, and will need to pick a balance between sensitivity and specificity. c.f. ROC: https://en.wikipedia.org/wiki/Receiver_operating_characteristic https://en.wikipedia.org/wiki/Receiver_operating_characteris...
- adyavanapalli 7y ago
  I think the OP is asking about the _value_ of the FPR instead of the definition.
  - RosanaAnaDana 7y ago
    The value of the false positive rate is that it lets you know the probability of a true-miss. Depending on the classification exercise, you may be concerned with false positives, where the consequence of a missed call is significantly greater than an unwarranted checkup from a human doctor.
    - shawnz 7y ago
      The numeric value
  - ska 7y ago
    Ah, if I misread then from the results section of the linked paper: For the malignancy prediction objective, the algorithm obtained an area under the receiver operating characteristic curve (AUC) of 0.91 (95% CI: 0.89, 0.93), with specificity of 77.3% (95% CI: 69.2%, 85.4%) at a sensitivity of 87%. I haven't read the papers methods, but the data set size is small-ish for this sort of analysis.
TuringNYC 7y ago
from the paper: https://pubs.rsna.org/doi/10.1148/radiol.2019182622 https://pubs.rsna.org/doi/10.1148/radiol.2019182622 AUC is 0.78 Sensitivity-Specificity Graph is here: http://images.rsna.org/index.html?doi=10.1148/radiol.2019182622&fig=fig3 http://images.rsna.org/index.html?doi=10.1148/radiol.2019182...
- mdorazio 7y ago
  Thanks for linking. If I'm reading that correctly, it's pretty bad in comparison to an average human radiologist at ~6% false positive rate [1]. There's probably a bias factor in there, however, where humans are hesitant to predict potential cancer due to the cost/time involved in follow-up screening. [1] https://www.ncbi.nlm.nih.gov/pubmed/21643887 https://www.ncbi.nlm.nih.gov/pubmed/21643887

NikolaeVarius 7y ago

Is this new AI Model under Watson?

layoutIfNeeded 7y ago
There’s no such thing as “Watson”. IBM have put the Watson name on basically everything, to the point where its information content was reduced to zero bits. Watson for IBM is like the i-prefix for Apple.
noelsusman 7y ago
Watson is a brand, so that doesn't really mean anything. If Watson refers to anything it would be the NLP functionality that IBM sells, and that's not relevant here.

btilly 7y ago

This is not exactly new. I remember seeing models that did really well many years ago. And again caught many that humans had miss. The problem is that they fail differently than humans do, in a way that humans wind up not trusting the results. It turns out that there are parts of the breast that are easy to spot tumors in, and parts that are hard. A human scans quickly over the easy areas, and focuses on the hard. The result is that humans make careless errors on the easy areas, and catch hard tumors. Computers make no careless errors, but can't catch the hard ones. Thus when a human sees what the computer caught that the human did not, the mistake is easily dismissed. But when the human sees the ones that the computer missed, it becomes, "It doesn't know how to do the real work." Ideally the two would be used together for better results than either alone. But humans wind up resenting the computer...

hathawsh 7y ago
That's a helpful perspective. Would it be possible for IBM to create a service that allows patients to submit their own pictures for scanning?
- lostlogin 7y ago
  We are at a tricky stage with this. Images are too big to be emailed and many people no longer have CD drives. There is increasing use of tomographic imaging in examinations too, and the files are pretty big.
  - hathawsh 7y ago
    On the other hand, cheap micro SD cards and thumb drives hold far more than CDs or DVDs.
    - lostlogin 7y ago
      We prefer to send the images to the PACS the patient wants them on, as for most imaging that’s all that’s wanted. Obstetric imaging is one massive exception.

deleted 7y ago

[deleted]

rayuela 7y ago

Anything related to AI coming out of IBM should be viewed with a huge dose of skepticism. They're honestly one of the worst offenders in overselling the capabilities of their products, bordering out outright fraud. There is certainly a lot of promise to the application of recent computer vision algos on medical imaging data, but I wouldn't bet much on IBM being anything close to a leader in this space.

Myrmornis 7y ago
I don't doubt what you say. Just want to point out that this is published in a peer-reviewed journal, so hopefully the academic community will judge it objectively. https://pubs.rsna.org/doi/10.1148/radiol.2019182622 https://pubs.rsna.org/doi/10.1148/radiol.2019182622
- dragandj 7y ago
  OTOH, do even radiologists (or anyone else) can predict cancer at all before it happens? I thought that radiologists diagnose cancer once it is already there.
  - the8472 7y ago
    not all tumors are malignant
  - thaumasiotes 7y ago
    > do even radiologists (or anyone else) can predict cancer at all before it happens? Sure, some of the time it's easy. Let's all recall the words of my mother's medical school instructor, "There's a bit of cancer in everyone's prostate". (The context was a lab exercise in which medical students were supposed to find which of a set of slides of prostate tissue was cancerous. The reminder was necessary because many of the slides were cancerous, just not at levels high enough to be considered medically alarming.) Predicting that a man will develop prostate cancer is basically the same thing as predicting that he'll experience old age.

nkurz 7y ago

What makes this an "AI Model" instead of just a "Model"? That is, in what way does it have "artificial intelligence"?

amelius 7y ago
Probably because it was obtained using some kind of machine learning.
TuringNYC 7y ago
"Model" ---> [[marketing department]] --> "AI Model"
- raxxorrax 7y ago
  Can I call myself an AI specialist if I successfully fed a plain support vector machine once or twice for diagnostic support? Feels like driving an old timer here...
- avgDev 7y ago
  [Student in school] Implemented MinMax algorithm in checkers ---> [student looking for work] Implemented state of the art AI algorithm, which successfully will beat the human opponent EVERY time. ---> [HR/Marketing dept at some corp] Wow you are HIRED!!!!!!!! ---> [Lead dev] Oof this guy can't program for shit.
  - TeMPOraL 7y ago
    ---> [student about to get fired from work] Why on Earth did they put "AI expertise" in the job requirements if all they want me to do is to shovel CSS and JS, and the closest thing to AI they have in the office is a 1960s thermostat?
onemoresoop 7y ago
Yes, the AI part is marketing/fluff
Myrmornis 7y ago

dontreact 7y ago

The reason I'm skeptical of this is that there is no actual comparison to human level performance. I.E. they didn't have radiologist actually read their images to compare against the model. Notice that the title of the paper is "Predicting Breast Cancer by Applying Deep Learning to Linked Health Records and Mammograms" it's only in the press release that they seem to imply a comparison to radiologists was actually done.

thatcantbeit 7y ago
https://pubs.rsna.org/doi/full/10.1148/radiol.2016161174 https://pubs.rsna.org/doi/full/10.1148/radiol.2016161174 This is their comparison point for actual radiologists. Citation number 6. It doesn't look comparable, though. Radiologists are around 90% specificity and sensitivity, which varies a good amount from the model's 77.3% and 87%, respectively.
- dontreact 7y ago
  This is not on this dataset though (right?), so not really a solid comparison point. Plus lik you mentioned, they seem to be doing worse than this benchmark.

baybal2 7y ago

I was lucky to date a girl who was into math, and who was coding those "machine learning" algorithms for a radiology startup here in Shenzhen. She had a lot of scepticism for what she did. One of biggest showstoppers she said was the unpredictability of errors. An algo can catch 99% tumors, including tiny ones, bur can randomly pass over very obvious ones which a human radiologist will spot with his eyes closed. They had a demo day with radiologists, and them throwing tricky edge case xrays at the computer. Edge cases were all ok, but one radiologist pulled his own xray from his bag, with a 100% obvious, terminal stage tumor, and to company's embarrassment, the algo failed to detect it no mater how they twisted and scaled the xray. The guy then just walked out.

WalterBright 7y ago
It seems to me the use case should be to have the radiologist look at a scan for tumors. Then, the algo should look. If they disagree, then the radiologist should look at the difference. It'll be the best of both. And in the scans where the algo is wrong, have the scan added to the machine learning database of the algo.
- ttlei 7y ago
  If the radiologist has to look at and double check every scan that algo looked at, then what is the point of the algo? Seems like a useless middleman that get in the way.
  - navigatesol 7y ago
    >then what is the point of the algo? The point is that the algorithm can improve results. This isn't ad placement, it's peoples' lives. Checking and double checking should be the norm.
  - bradstewart 7y ago
    Because the scan check by the radiologist becomes a _double_ check.
  - ska 7y ago

mlcrypto 7y ago

Doctors will be some of the first to be replaced by AI. My physicians walk around with a computer already checking all the boxes for symptoms and seeing what it says. I wish I could find one with a true intuition for medicine

caraffle 7y ago
Apparently you're not familiar with the documentation burden in the medical field. EHR's don't diagnose for you. There is no "true" intuition in medicine, just years of study and practice leading to quick recognition of common problems like any other field.

Myrmornis 7y ago

@moderators: would it make sense to change the link to the journal article rather than IBM's article? It's free access. https://pubs.rsna.org/doi/10.1148/radiol.2019182622 https://pubs.rsna.org/doi/10.1148/radiol.2019182622

professorTuring 7y ago

The real problem here is when the society will allow a machine to diagnose them and if the society is ready to believe that most diagnostics are probabilistically made. Up to date we allow humans to be at a 70% error level without problems, but we ask machines to be 100% effective. The very same happens with autopilot, the big numbers say they drive better than humans but...

petschge 7y ago
I remember seeing a statistical analysis here on HN that said the numbers for Tesla autopilots are neither great compared to drivers of Teslas nor do they seem to be fair. (They found a case where a human driver had a crash in what would have been counted as 0 miles in the analysis, indicating that something is inflating the "crashes per miles" metric)
- AstralStorm 7y ago
  Using autopilot in parking?
anthony_doan 7y ago
These type of algorithms don't give a confidence interval for their predictions so I don't believe these diagnostics are base on probability at all. Having a confusion matrix for what the model predict correct or not is not the same as having a CI for the model's prediction.
- zone411 7y ago
  Isn't it possible to derive a CI from the confusion matrix? https://stats.stackexchange.com/questions/363382/confidence-interval-of-precision-recall-and-f1-score https://stats.stackexchange.com/questions/363382/confidence-... or using bootstrap? The authors of this research also provide CIs.
stubish 7y ago
This isn't about group think. It is when individuals will allow diagnosis. If you give a woman two options, diagnosis by machine with a record of 80% or a human with a record of 70%, it is a really easy decision to make. The desire to not suffer cancer is strong enough to override almost all emotional arguments. And if you can afford it, you will likely choose both or get a second opinion.

blueyes 7y ago

Old news from a major source of AI hype. Here's some previous results https://med.stanford.edu/news/all-news/2018/11/ai-outperformed-radiologists-in-screening-x-rays-for-certain-diseases.html https://med.stanford.edu/news/all-news/2018/11/ai-outperform...

michaelhoffman 7y ago

No positive predictive value reported, imbalanced test data, IBM. Garbage.