6 ms·

Stanford computer diagnoses breast cancer more accurately than human doctor

146 points by mrsebastian 15y ago

euroclydon 15y ago
Reminds me of a comment by _delirium a few days ago, in the "AI is killing jobs" thread: dermatologists are fairly worried that "upload a photo and we'll diagnose your mole", either via software or outsourcing, will cut out a significant percentage of their business. http://news.ycombinator.com/item?id=3204466 http://news.ycombinator.com/item?id=3204466
- AndrewDucker 15y ago
  Also announced today: http://news.techeye.net/science/ai-used-to-hunt-dinosaur-bones http://news.techeye.net/science/ai-used-to-hunt-dinosaur-bon... those darn robots are everywhere!
  - artmageddon 15y ago
    "We may never have to dig again" -Dr. Grant, Jurassic Park
- deleted 15y ago
  [deleted]
allenp 15y ago
"...but imagine if you had a C-Path machine in every town or city." How near-sighted of the author. An article about computer vision, AI, machine learning, and no thoughts of tele-pathology or remote reading?
- mrsebastian 15y ago
  hat tip My thought was that such such a machine would basically have a built-in, automated biopsy thing. Like, the machine really would do everything from taking a biopsy, looking at the sample, and then diagnosing. Tele-pathology is cool, but you still need to get the tissue sample somehow -- which I imagine is hard in a town/village without a hospital. You could have some kind of automated biopsy unit that then sends data back to a central C-Path server, I guess.
  - ippisl 15y ago
    There's an indian company selling biopsy helper robots. it robots simplify the process and enable technicians to take the biopsy instead of doctors, and the whole process is faster. see : http://www.perfinthealthcare.com/Procedure_Videos.html http://www.perfinthealthcare.com/Procedure_Videos.html Looking at the videos , it seems that the whole process can be done by robot, But maybe patients and other stakeholders feel this way is safer/more economical.
melling 15y ago
You can learn from some of the experts in the field: http://ai-class.com http://ai-class.com - Peter Norvig and Sebastian Thrun http://ml-class.com http://ml-class.com - Andrew Ng I signed up 4 weeks into Andrew's class. Both of these are excellent. Btw, the technique in the article is a classification problem, right? :-)
- signa11 15y ago
  a multiclass-classification but yes. i am enrolled as well ;)
- barry-cotter 15y ago
  Is there any reason to watch the videos if one has AIMA? Could one do the assignments without without ever watching a video?
dave_sullivan 15y ago
They kind of hint at it in the article, but sounds like automatic feature detection really helped here. That's really the future (or present): learning algos discovering better features automatically, significantly cutting down the labor involved in feature engineering. Pretty neat. Here's a video talking about current research into this at Stanford: http://www.youtube.com/watch?v=ZmNOAtZIgIk&feature=youtube_gdata_player http://www.youtube.com/watch?v=ZmNOAtZIgIk&feature=youtu...
pigs 15y ago
Although he's not listed as an author, I find it interesting that Andrew Ng has been using "malignant or benign tumor" as an example of a machine learning classification problem at http://www.ml-class.org http://www.ml-class.org
davidhansen 15y ago
This does not address the issue of feature detection, but it's interesting nonetheless, inasmuch as well-trained human doctors are quite bad at probability, with the specific case of breast cancer diagnoses. The first 1/8th or so of Eliezer's layman's Bayes document: http://yudkowsky.net/rational/bayes http://yudkowsky.net/rational/bayes
kenjackson 15y ago
One thing the article doesn't point out is that AI methods in the past did a poor job of detecting breast cancer. This is an important new result and big step for the field.
- _delirium 15y ago
  Unfortunately this still doesn't look like it's detecting breast cancer; it's scoring the severity of tumors, given prior knowledge that it's being presented tumors to analyze (but tumors of unknown severity).
separated 15y ago
The funny thing is, most rads physicians would probably welcome this, as mammography is often the least preferred sub-specialty (my wife is a radiologist and feels this way, and I've encountered many radiologists with the same outlook).
- roentgen 15y ago
  The system in this article is not interpreting mammograms, which is something that radiologists do. This is looking at tissue samples on slides, the domain of pathologists.
- hugh3 15y ago
  mammography is often the least preferred sub-specialty (my wife is a radiologist and feels this way, and I've encountered many radiologists with the same outlook). Why? Because breast cancer is hard to diagnose? Because nobody likes looking at floppy old-lady boobs all day?
bh42222 15y ago
This should not be surprising for anyone who understand software. And software identifying breast and skin and other cancers is going to continue getting better. Now the bad news is that human medicine moves super slowly due to very strict regulations, and I expect doctors to resist this. I hate to say this, but I think health insurance companies are our best hope to push this technology into greater use.
- tryitnow 15y ago
  I agree. A lot of these technologies will lead to improved outcomes at lower costs (for certain sets of basic problems). Guild professions (like doctors) are inclined to keep doing even the basic simple-minded aspects of their job because they get paid "economic rents" for doing so thanks to the regulations that insist even basic tasks must be done by someone with 10+ years of education. Insurance companies on the other hand ultimately have to respond to employer demands for lower premiums (unfortunately this process is slow and HR departments are usually horrible at keeping costs under control). Insurers and to an extent employers are going to be the impetus for a lot of improvements in effectiveness and affordability. Read Clayton Christensen "The Innovators Prescription" for more on how this might play out.
2mur 15y ago
I'm a pathologist and an avocational programmer. This is pretty neat material and is very relevant to me, as I have been trying to bone up my math chops with Khan Academy videos so that I can tackle some computer vision related work in pathology. With regards to the study, I will just point out that the system is not diagnosing breast carcinoma, but rather is producing a score which reflects the prognosis as relates to overall survival of the patient (so it is not as impressive as the somewhat hyperbolic HN title makes it out to be... a better title would be 'Stanford computer analyzes breast cancer more accurately than human doctor' which is not surprising at all given the well-documented interobserver variability in breast cancer scoring, particularly in moderately-differentiated tumors). That is to say that the computer already knows the tissue that it is looking at is a tumor and not benign breast tissue. Furthermore, it is really only providing a histologic (or morphologic) score, which is to say, that it is attempting to predict how aggressive the tumor will behave based upon how well or not well it is differentiated (how ugly it is). These days, this score is actually less useful in clinical practice than other information such as whether or not the tumor cells are expressing estrogen/progesterone hormone receptors, or if it is over-expressing Her2-neu protein, as these are possible paths for cancer therapy (anti-estrogen drugs vs. Herceptin) in addition to being prognostic indicators (tumors which express ER/PR generally behave better than tumors which are ER/PR negative and express Her2), as well as the stage (or how far the cancer has already spread) at the time the patient is diagnosed. There are a bunch of companies which are getting FDA approval for computer vision related algorithms for scoring immunohistochemical assays for ER/PR/Her2 [1]. So I am actually far less concerned about a computer doing my job very well, which is actually looking at a piece of tissue on a slide and making a tumor versus not-tumor distinction. This is very hard to do and I think will continue to be even harder for computers/computer-vision/AI to do for a long time to come. I am far more concerned about molecular diagnostics. That is the true-future for making cancer determinations and may even eliminate the part of my job where I tell you if something is benign vs. malignant. [1] http://www.aperio.com/pathology-services/analyze-slides-automated-pathology.asp http://www.aperio.com/pathology-services/analyze-slides-auto...
- bh42222 15y ago
  So I am actually far less concerned about a computer doing my job very well, which is actually looking at a piece of tissue on a slide and making a tumor versus not-tumor distinction. This is very hard to do and I think will continue to be even harder for computers/computer-vision/AI to do for a long time to come. Can you elaborate?
danso 15y ago
This kind of development is not that new, conceptually, at least (obviously, implementing it is another thing). One of the most consistently verified themes of study is that human experts can be very inconsistent, and in some cases, expertise can be a detriment when it leads to overconfidence. Radiologists, for example, have been found 20% of the time to render two different verdicts when looking at the same X-ray at two different times. And of course there's the recent study showing how judges were consistently more likely to deny parole during hearings that happen in the afternoon: http://blogs.discovermagazine.com/notrocketscience/2011/04/11/justice-is-served-but-more-so-after-lunch-how-food-breaks-sway-the-decisions-of-judges/ http://blogs.discovermagazine.com/notrocketscience/2011/04/1...
maeon3 15y ago
Looks like weak AI is making inroads all over the medical industry. Here is predictive ai technology for heart failure that works better/faster than human diagnosis. http://en.m.wikipedia.org/wiki/Multifunction_cardiogram http://en.m.wikipedia.org/wiki/Multifunction_cardiogram
pak 15y ago
Please: there ought to be a rule on HN that instead of linking to a popsci re-interpretation of a scientific result, you instead link directly to the paper. http://stm.sciencemag.org/content/3/108/108ra113.full http://stm.sciencemag.org/content/3/108/108ra113.full If it is behind a paywall, as it probably is for this paper if you are not at a university, perhaps look around for the least hyperbolic re-interpretation of the paper, and link that instead. http://www.genengnews.com/gen-news-highlights/image-analysis-system-predicts-breast-cancer-survival-based-on-stromal-and-epithelial-features/81245934/ http://www.genengnews.com/gen-news-highlights/image-analysis... Even if you do not have access to the full paper because of the paywall, you should be able to still read the abstract and pick the popsci article that is fairest to what the paper actually says. Otherwise, this happens: http://www.phdcomics.com/comics.php?f=1174 http://www.phdcomics.com/comics.php?f=1174 and I really don't like it when HN blindly follows the hivemind in furthering that phenomenon. Let's examine what the authors actually say. To directly compare the performance of the C-Path system to pathological grading on the exact same set of images, we applied standard pathological grading criteria to the TMA images used in the C-Path analysis. [...] the pathologist grading the images was blinded from the survival data. Although the C-Path predictions on the NKI data set were strongly associated with survival, the pathologic grade derived from the same TMA images showed no significant association with survival (log-rank P = 0.4), highlighting the difficulty of obtaining accurate prognostic predictions from these small tumor samples. That's it. They make one remark about it, and do not focus on this at all elsewhere in the paper, because it was not the point of their study and the methodology for this little result is far from robust. Note they used one pathologist to run this little test. Also note that a high p-value is not evidence that the null hypothesis is true--it is quite possible that there is a relationship but the study is underpowered; this is a frequent point of confusion. Let's please keep scientific statements in context. The original paper says nothing tantamount to computers diagnosing breast cancer more accurately than doctors. It is, principally, about a new morphological feature that the researchers believe is tied more strongly to survival according to their computational model.
- nazgulnarsil 15y ago
  The cottage industry of drawing fanciful conclusions from tiny data sets needs to die in a fire. It is pure noise.
JabavuAdams 15y ago
Wow! The medicalese description (abstract of the original paper) seems completely unrelated to the computerese description. I need to learn medicalese ... or train some ML algos.
deleted 15y ago
[deleted]

146 points by mrsebastian 15y ago

Reminds me of a comment by _delirium a few days ago, in the "AI is killing jobs" thread: dermatologists are fairly worried that "upload a photo and we'll diagnose your mole", either via software or outsourcing, will cut out a significant percentage of their business. http://news.ycombinator.com/item?id=3204466 http://news.ycombinator.com/item?id=3204466

AndrewDucker 15y ago
Also announced today: http://news.techeye.net/science/ai-used-to-hunt-dinosaur-bones http://news.techeye.net/science/ai-used-to-hunt-dinosaur-bon... those darn robots are everywhere!
- artmageddon 15y ago
  "We may never have to dig again" -Dr. Grant, Jurassic Park
deleted 15y ago
[deleted]

allenp 15y ago

"...but imagine if you had a C-Path machine in every town or city." How near-sighted of the author. An article about computer vision, AI, machine learning, and no thoughts of tele-pathology or remote reading?

mrsebastian 15y ago
hat tip My thought was that such such a machine would basically have a built-in, automated biopsy thing. Like, the machine really would do everything from taking a biopsy, looking at the sample, and then diagnosing. Tele-pathology is cool, but you still need to get the tissue sample somehow -- which I imagine is hard in a town/village without a hospital. You could have some kind of automated biopsy unit that then sends data back to a central C-Path server, I guess.
- ippisl 15y ago
  There's an indian company selling biopsy helper robots. it robots simplify the process and enable technicians to take the biopsy instead of doctors, and the whole process is faster. see : http://www.perfinthealthcare.com/Procedure_Videos.html http://www.perfinthealthcare.com/Procedure_Videos.html Looking at the videos , it seems that the whole process can be done by robot, But maybe patients and other stakeholders feel this way is safer/more economical.

melling 15y ago

You can learn from some of the experts in the field: http://ai-class.com http://ai-class.com - Peter Norvig and Sebastian Thrun http://ml-class.com http://ml-class.com - Andrew Ng I signed up 4 weeks into Andrew's class. Both of these are excellent. Btw, the technique in the article is a classification problem, right? :-)

signa11 15y ago
a multiclass-classification but yes. i am enrolled as well ;)
barry-cotter 15y ago
Is there any reason to watch the videos if one has AIMA? Could one do the assignments without without ever watching a video?

dave_sullivan 15y ago

They kind of hint at it in the article, but sounds like automatic feature detection really helped here. That's really the future (or present): learning algos discovering better features automatically, significantly cutting down the labor involved in feature engineering. Pretty neat. Here's a video talking about current research into this at Stanford: http://www.youtube.com/watch?v=ZmNOAtZIgIk&feature=youtube_gdata_player http://www.youtube.com/watch?v=ZmNOAtZIgIk&feature=youtu...

pigs 15y ago

Although he's not listed as an author, I find it interesting that Andrew Ng has been using "malignant or benign tumor" as an example of a machine learning classification problem at http://www.ml-class.org http://www.ml-class.org

davidhansen 15y ago

This does not address the issue of feature detection, but it's interesting nonetheless, inasmuch as well-trained human doctors are quite bad at probability, with the specific case of breast cancer diagnoses. The first 1/8th or so of Eliezer's layman's Bayes document: http://yudkowsky.net/rational/bayes http://yudkowsky.net/rational/bayes

kenjackson 15y ago

One thing the article doesn't point out is that AI methods in the past did a poor job of detecting breast cancer. This is an important new result and big step for the field.

_delirium 15y ago
Unfortunately this still doesn't look like it's detecting breast cancer; it's scoring the severity of tumors, given prior knowledge that it's being presented tumors to analyze (but tumors of unknown severity).

separated 15y ago

The funny thing is, most rads physicians would probably welcome this, as mammography is often the least preferred sub-specialty (my wife is a radiologist and feels this way, and I've encountered many radiologists with the same outlook).

roentgen 15y ago
The system in this article is not interpreting mammograms, which is something that radiologists do. This is looking at tissue samples on slides, the domain of pathologists.
hugh3 15y ago
mammography is often the least preferred sub-specialty (my wife is a radiologist and feels this way, and I've encountered many radiologists with the same outlook). Why? Because breast cancer is hard to diagnose? Because nobody likes looking at floppy old-lady boobs all day?

bh42222 15y ago

This should not be surprising for anyone who understand software. And software identifying breast and skin and other cancers is going to continue getting better. Now the bad news is that human medicine moves super slowly due to very strict regulations, and I expect doctors to resist this. I hate to say this, but I think health insurance companies are our best hope to push this technology into greater use.

tryitnow 15y ago
I agree. A lot of these technologies will lead to improved outcomes at lower costs (for certain sets of basic problems). Guild professions (like doctors) are inclined to keep doing even the basic simple-minded aspects of their job because they get paid "economic rents" for doing so thanks to the regulations that insist even basic tasks must be done by someone with 10+ years of education. Insurance companies on the other hand ultimately have to respond to employer demands for lower premiums (unfortunately this process is slow and HR departments are usually horrible at keeping costs under control). Insurers and to an extent employers are going to be the impetus for a lot of improvements in effectiveness and affordability. Read Clayton Christensen "The Innovators Prescription" for more on how this might play out.

2mur 15y ago

I'm a pathologist and an avocational programmer. This is pretty neat material and is very relevant to me, as I have been trying to bone up my math chops with Khan Academy videos so that I can tackle some computer vision related work in pathology. With regards to the study, I will just point out that the system is not diagnosing breast carcinoma, but rather is producing a score which reflects the prognosis as relates to overall survival of the patient (so it is not as impressive as the somewhat hyperbolic HN title makes it out to be... a better title would be 'Stanford computer analyzes breast cancer more accurately than human doctor' which is not surprising at all given the well-documented interobserver variability in breast cancer scoring, particularly in moderately-differentiated tumors). That is to say that the computer already knows the tissue that it is looking at is a tumor and not benign breast tissue. Furthermore, it is really only providing a histologic (or morphologic) score, which is to say, that it is attempting to predict how aggressive the tumor will behave based upon how well or not well it is differentiated (how ugly it is). These days, this score is actually less useful in clinical practice than other information such as whether or not the tumor cells are expressing estrogen/progesterone hormone receptors, or if it is over-expressing Her2-neu protein, as these are possible paths for cancer therapy (anti-estrogen drugs vs. Herceptin) in addition to being prognostic indicators (tumors which express ER/PR generally behave better than tumors which are ER/PR negative and express Her2), as well as the stage (or how far the cancer has already spread) at the time the patient is diagnosed. There are a bunch of companies which are getting FDA approval for computer vision related algorithms for scoring immunohistochemical assays for ER/PR/Her2 [1]. So I am actually far less concerned about a computer doing my job very well, which is actually looking at a piece of tissue on a slide and making a tumor versus not-tumor distinction. This is very hard to do and I think will continue to be even harder for computers/computer-vision/AI to do for a long time to come. I am far more concerned about molecular diagnostics. That is the true-future for making cancer determinations and may even eliminate the part of my job where I tell you if something is benign vs. malignant. [1] http://www.aperio.com/pathology-services/analyze-slides-automated-pathology.asp http://www.aperio.com/pathology-services/analyze-slides-auto...

bh42222 15y ago
So I am actually far less concerned about a computer doing my job very well, which is actually looking at a piece of tissue on a slide and making a tumor versus not-tumor distinction. This is very hard to do and I think will continue to be even harder for computers/computer-vision/AI to do for a long time to come. Can you elaborate?

danso 15y ago

This kind of development is not that new, conceptually, at least (obviously, implementing it is another thing). One of the most consistently verified themes of study is that human experts can be very inconsistent, and in some cases, expertise can be a detriment when it leads to overconfidence. Radiologists, for example, have been found 20% of the time to render two different verdicts when looking at the same X-ray at two different times. And of course there's the recent study showing how judges were consistently more likely to deny parole during hearings that happen in the afternoon: http://blogs.discovermagazine.com/notrocketscience/2011/04/11/justice-is-served-but-more-so-after-lunch-how-food-breaks-sway-the-decisions-of-judges/ http://blogs.discovermagazine.com/notrocketscience/2011/04/1...

maeon3 15y ago

Looks like weak AI is making inroads all over the medical industry. Here is predictive ai technology for heart failure that works better/faster than human diagnosis. http://en.m.wikipedia.org/wiki/Multifunction_cardiogram http://en.m.wikipedia.org/wiki/Multifunction_cardiogram

pak 15y ago

Please: there ought to be a rule on HN that instead of linking to a popsci re-interpretation of a scientific result, you instead link directly to the paper. http://stm.sciencemag.org/content/3/108/108ra113.full http://stm.sciencemag.org/content/3/108/108ra113.full If it is behind a paywall, as it probably is for this paper if you are not at a university, perhaps look around for the least hyperbolic re-interpretation of the paper, and link that instead. http://www.genengnews.com/gen-news-highlights/image-analysis-system-predicts-breast-cancer-survival-based-on-stromal-and-epithelial-features/81245934/ http://www.genengnews.com/gen-news-highlights/image-analysis... Even if you do not have access to the full paper because of the paywall, you should be able to still read the abstract and pick the popsci article that is fairest to what the paper actually says. Otherwise, this happens: http://www.phdcomics.com/comics.php?f=1174 http://www.phdcomics.com/comics.php?f=1174 and I really don't like it when HN blindly follows the hivemind in furthering that phenomenon. Let's examine what the authors actually say. To directly compare the performance of the C-Path system to pathological grading on the exact same set of images, we applied standard pathological grading criteria to the TMA images used in the C-Path analysis. [...] the pathologist grading the images was blinded from the survival data. Although the C-Path predictions on the NKI data set were strongly associated with survival, the pathologic grade derived from the same TMA images showed no significant association with survival (log-rank P = 0.4), highlighting the difficulty of obtaining accurate prognostic predictions from these small tumor samples. That's it. They make one remark about it, and do not focus on this at all elsewhere in the paper, because it was not the point of their study and the methodology for this little result is far from robust. Note they used one pathologist to run this little test. Also note that a high p-value is not evidence that the null hypothesis is true--it is quite possible that there is a relationship but the study is underpowered; this is a frequent point of confusion. Let's please keep scientific statements in context. The original paper says nothing tantamount to computers diagnosing breast cancer more accurately than doctors. It is, principally, about a new morphological feature that the researchers believe is tied more strongly to survival according to their computational model.

nazgulnarsil 15y ago
The cottage industry of drawing fanciful conclusions from tiny data sets needs to die in a fire. It is pure noise.

JabavuAdams 15y ago

Wow! The medicalese description (abstract of the original paper) seems completely unrelated to the computerese description. I need to learn medicalese ... or train some ML algos.

deleted 15y ago

[deleted]