5 ms·

Emergency doctor here, I have to say that diagnosing a sick child is one of the hardest things you can do in medicine (aside from figuring out why someone is fe

Emergency doctor here, I have to say that diagnosing a sick child is one of the hardest things you can do in medicine (aside from figuring out why someone is feeling 'dizzy'). The limiting step here is that we can't do blood tests and x-ray/CT on everyone that walks in the door (which itself can lead to harm: https://emergencymedicinecases.com/overinvestigation-emergency-medicine/ https://emergencymedicinecases.com/overinvestigation-emergen...) I work in one of the busiest ER's in Canada, and maybe one child out of hundreds will have something hidden... Many illnesses, particularly meningitis, are so rapid in killing the patient that you may miss the early symptoms (which are often mild). What many posters have eluded to about 'gut feeling' is well described in medical education literature, where 'system 1' is well thought out, algorithmic reasoning, and 'system 2' is pattern recognition and based on experience. Many cognitive biases can affect both meeting (described will in this paper: https://www.ncbi.nlm.nih.gov/m/pubmed/12915363/ https://www.ncbi.nlm.nih.gov/m/pubmed/12915363/). Your gut feeling can be affected from anything from unusual vital signs, to the way a patient talks, to beads of sweat on their forehead, and a million other things that take years and years of training to catch. Anyone who can build an ML model to catch these clues is going to make billions.

philipodonnell 8y ago
> Anyone who can build an ML model to catch these clues is going to make billions. You'd be surprised. IBM poured billions into Watson and appears to have been pretty successful in nearly reaching parity with a certified oncologist, but the results were dismissed because it didn't outperform them. > At first, Manipal used Watson to recommend treatment options for all cancer patients, said oncologist S.P. Somashekhar. It found the software agreed with doctors most of the time, so Manipal stopped using Watson on every patient, he said. https://www.wsj.com/articles/ibm-bet-billions-that-watson-could-improve-cancer-treatment-it-hasnt-worked-1533961147 https://www.wsj.com/articles/ibm-bet-billions-that-watson-co...
- j16sdiz 8y ago
  It is used to recommend treatment options, not diagnosis
  - VLM 8y ago
    Treatment oriented medicine is a problem in and of itself. There is no treatment for medically diagnosed Celliac disease other than don't eat wheat for the rest of your life; its hard to profit off that in the medical biz, therefore, there being no treatment, mental gymnastics to disprove anything treatable are mandatory across several pediatricians and gastroenterologists and oncologists we talked to for a year or so WRT our son. Finally after ruling out child abuse and cancer, almost out of desperation, they did a quick blood antibody test leading to a stomach cilia biopsy and they're like "sorry its not good news like treatable cancer, its untreatable gluten allergy". Somehow the poor little guy must have given blood for a hundred tests before the antibody test came back with higher levels than the ENT had ever seen in a test result, and the biopsy confirmed it. Remove wheat from his diet and in about a week he was the healthiest kid ever. The insane part is about one percent of the population is somewhere along the Celiac spectrum of symptoms; my kid somewhat along the extreme edge, most not so bad. But docs will do anything to avoid a simple blood test / biopsy diagnosis because there's no profitable treatment or pill to push. Somehow, almost accidentally or unintentionally, western medicine in the USA can provide good outcomes, but the average citizen playing the game doesn't understand that its primarily a legal CYA game secondarily a treatment commissioned salesmen game, also a money making racket, and only a distant minor goal is actually helping people live better lives. If people are helped by the medical establishment its almost by accident as a side effect, helping people is definitely not the primary purpose of the establishment.
  - opportune 8y ago
    That’s just liability-related verbiage
- freeone3000 8y ago
  That's not a great definition of parity. We'd want accuracy and specificity numbers linked to outcomes, not concurrence. The times when Watson agrees with doctors is effectively irrelevant - results would be the same whether or not he was added. We need to highlight whether, given a disagreement, Watson was better or worse for outcomes.
  - philipodonnell 8y ago
    The trick is whether the cases of disagreement were themselves predictable! Given a fixed number of oncologists and deploying Watson only to support those oncologists, yes, you're correct that its only useful if it outperforms them. But I think of it more like Watson is a single hive-mind team of like a thousand med students near the end of residency: they get most things right but there are a few places where more experienced doctors will be better, though the scale with the hive mind is far higher. You have one of two reactions to that. 1) Hire fewer senior oncologists and have them focus on the more difficult cases and leave the hive-mind to deal with thousands of routine cases, or 2) ignore the hive mind until its literally better than a typical senior oncologist. The medial profession seems to repeat this cycle of "only full doctors can do anything because even seemingly routine cases might be hiding something more serious" to "maybe some routine things can be done by people with less training and full doctors should focus on the more difficult cases". See nurse practitioners, dental assistants, and, in my mind at least where we are going with things like Watson.
    - Bartweiss 8y ago
      I think part of the problem for Watson is that it needs someone to gather the data, which is usually a doctor. So if you're pairing each patient with an oncologist for intake anyway, it's not clear that "examination plus enter all data into Watson" is a benefit over "examination plus make a decision". I guess the ideal outcome for Watson (if it doesn't outpace expert oncologists) would be something like "experience nurse practitioner does an exam, and enters data into Watson" or maybe even "special oncology-trained NP does an exam with Watson". The other part I don't know is what oncology accuracy rates look like. If the reason to not majorly expand screening is cost and availability, Watson could be huge. If it's false positives from our existing rate, there's a lot less value.
- kashprime 8y ago
  I think better diagnostic testing is one way to solve this. Oncology in some ways is easy since the data input is usually a set of biomarkers particular to that tumour (i.e. estrogen receptor, her2/neu, etc). It's probably the most rapidly advancing field in medicine. Same with radiology and dermatology where the input can be images which can be standardized. Dealing with analog humans is another challenge entirely! Better biomarkers for acute illnesses with fewer false positives would help you better test 'accuracy' versus an ML model.
- ggm 8y ago
  A recent post here suggested Watson was recommending highly disadvantageous treatments and MDs became reluctant to continue because it wasn't working.
deleted 8y ago
[deleted]
epmaybe 8y ago
Someone else in the comments made a good point about capturing the common words patients tell providers and being able to parse that. If you could transform what a patient is telling you from their words into a concise set of standardized symptoms, then you have essentially done what any other provider does when they obtain a focused history. That could allow a model to prompt the specific questions needed, or request the provider to look for pertinent physical exam findings. On the other hand (maybe easier?), I think we could also improve the use of some of our less costly and non-harmful techniques, such as ultrasound, to improve our screening.
opo 8y ago
>...Anyone who can build an ML model to catch these clues is going to make billions. Yea you would think so... Unfortunately, in medicine, just because a computer program is better then a Dr at diagnosing a patient is no guarantee it will be used. The classic example here was the MYCIN expert system developed in the 1970s. MYCIN was shown to outperform infectious disease experts by 1979 in a blind test: >... Eight independent evaluators with special expertise in the management of meningitis compared MYCIN's choice of antimicrobials with the choices of nine human prescribers for ten test cases of meningitis. MYCIN received an acceptability rating of 65% by the evaluators; the corresponding ratings for acceptability of the regimen prescribed by the five faculty specialists ranged from 42.5% to 62.5%. The system never failed to cover a treatable pathogen while demonstrating efficiency in minimizing the number of antimicrobials prescribed. https://jamanetwork.com/journals/jama/article-abstract/366606 https://jamanetwork.com/journals/jama/article-abstract/36660... https://en.wikipedia.org/wiki/Mycin https://en.wikipedia.org/wiki/Mycin If MYCIN hadn't been rejected by the medical profession, I am sure by now that we would have developed software to assist Doctors in all areas of medicine...

philipodonnell 8y ago

> Anyone who can build an ML model to catch these clues is going to make billions. You'd be surprised. IBM poured billions into Watson and appears to have been pretty successful in nearly reaching parity with a certified oncologist, but the results were dismissed because it didn't outperform them. > At first, Manipal used Watson to recommend treatment options for all cancer patients, said oncologist S.P. Somashekhar. It found the software agreed with doctors most of the time, so Manipal stopped using Watson on every patient, he said. https://www.wsj.com/articles/ibm-bet-billions-that-watson-could-improve-cancer-treatment-it-hasnt-worked-1533961147 https://www.wsj.com/articles/ibm-bet-billions-that-watson-co...

j16sdiz 8y ago
It is used to recommend treatment options, not diagnosis
- VLM 8y ago
  Treatment oriented medicine is a problem in and of itself. There is no treatment for medically diagnosed Celliac disease other than don't eat wheat for the rest of your life; its hard to profit off that in the medical biz, therefore, there being no treatment, mental gymnastics to disprove anything treatable are mandatory across several pediatricians and gastroenterologists and oncologists we talked to for a year or so WRT our son. Finally after ruling out child abuse and cancer, almost out of desperation, they did a quick blood antibody test leading to a stomach cilia biopsy and they're like "sorry its not good news like treatable cancer, its untreatable gluten allergy". Somehow the poor little guy must have given blood for a hundred tests before the antibody test came back with higher levels than the ENT had ever seen in a test result, and the biopsy confirmed it. Remove wheat from his diet and in about a week he was the healthiest kid ever. The insane part is about one percent of the population is somewhere along the Celiac spectrum of symptoms; my kid somewhat along the extreme edge, most not so bad. But docs will do anything to avoid a simple blood test / biopsy diagnosis because there's no profitable treatment or pill to push. Somehow, almost accidentally or unintentionally, western medicine in the USA can provide good outcomes, but the average citizen playing the game doesn't understand that its primarily a legal CYA game secondarily a treatment commissioned salesmen game, also a money making racket, and only a distant minor goal is actually helping people live better lives. If people are helped by the medical establishment its almost by accident as a side effect, helping people is definitely not the primary purpose of the establishment.
- opportune 8y ago
  That’s just liability-related verbiage
freeone3000 8y ago
That's not a great definition of parity. We'd want accuracy and specificity numbers linked to outcomes, not concurrence. The times when Watson agrees with doctors is effectively irrelevant - results would be the same whether or not he was added. We need to highlight whether, given a disagreement, Watson was better or worse for outcomes.
- philipodonnell 8y ago
  The trick is whether the cases of disagreement were themselves predictable! Given a fixed number of oncologists and deploying Watson only to support those oncologists, yes, you're correct that its only useful if it outperforms them. But I think of it more like Watson is a single hive-mind team of like a thousand med students near the end of residency: they get most things right but there are a few places where more experienced doctors will be better, though the scale with the hive mind is far higher. You have one of two reactions to that. 1) Hire fewer senior oncologists and have them focus on the more difficult cases and leave the hive-mind to deal with thousands of routine cases, or 2) ignore the hive mind until its literally better than a typical senior oncologist. The medial profession seems to repeat this cycle of "only full doctors can do anything because even seemingly routine cases might be hiding something more serious" to "maybe some routine things can be done by people with less training and full doctors should focus on the more difficult cases". See nurse practitioners, dental assistants, and, in my mind at least where we are going with things like Watson.
  - Bartweiss 8y ago
    I think part of the problem for Watson is that it needs someone to gather the data, which is usually a doctor. So if you're pairing each patient with an oncologist for intake anyway, it's not clear that "examination plus enter all data into Watson" is a benefit over "examination plus make a decision". I guess the ideal outcome for Watson (if it doesn't outpace expert oncologists) would be something like "experience nurse practitioner does an exam, and enters data into Watson" or maybe even "special oncology-trained NP does an exam with Watson". The other part I don't know is what oncology accuracy rates look like. If the reason to not majorly expand screening is cost and availability, Watson could be huge. If it's false positives from our existing rate, there's a lot less value.
kashprime 8y ago
I think better diagnostic testing is one way to solve this. Oncology in some ways is easy since the data input is usually a set of biomarkers particular to that tumour (i.e. estrogen receptor, her2/neu, etc). It's probably the most rapidly advancing field in medicine. Same with radiology and dermatology where the input can be images which can be standardized. Dealing with analog humans is another challenge entirely! Better biomarkers for acute illnesses with fewer false positives would help you better test 'accuracy' versus an ML model.
ggm 8y ago
A recent post here suggested Watson was recommending highly disadvantageous treatments and MDs became reluctant to continue because it wasn't working.

deleted 8y ago

[deleted]

epmaybe 8y ago

Someone else in the comments made a good point about capturing the common words patients tell providers and being able to parse that. If you could transform what a patient is telling you from their words into a concise set of standardized symptoms, then you have essentially done what any other provider does when they obtain a focused history. That could allow a model to prompt the specific questions needed, or request the provider to look for pertinent physical exam findings. On the other hand (maybe easier?), I think we could also improve the use of some of our less costly and non-harmful techniques, such as ultrasound, to improve our screening.

opo 8y ago

>...Anyone who can build an ML model to catch these clues is going to make billions. Yea you would think so... Unfortunately, in medicine, just because a computer program is better then a Dr at diagnosing a patient is no guarantee it will be used. The classic example here was the MYCIN expert system developed in the 1970s. MYCIN was shown to outperform infectious disease experts by 1979 in a blind test: >... Eight independent evaluators with special expertise in the management of meningitis compared MYCIN's choice of antimicrobials with the choices of nine human prescribers for ten test cases of meningitis. MYCIN received an acceptability rating of 65% by the evaluators; the corresponding ratings for acceptability of the regimen prescribed by the five faculty specialists ranged from 42.5% to 62.5%. The system never failed to cover a treatable pathogen while demonstrating efficiency in minimizing the number of antimicrobials prescribed. https://jamanetwork.com/journals/jama/article-abstract/366606 https://jamanetwork.com/journals/jama/article-abstract/36660... https://en.wikipedia.org/wiki/Mycin https://en.wikipedia.org/wiki/Mycin If MYCIN hadn't been rejected by the medical profession, I am sure by now that we would have developed software to assist Doctors in all areas of medicine...