6 ms·

Yes, it's absolutely very difficult to do rigorous statistical analysis. Genetics is oversimplified to non-physicians. It's cool that we can diagnose and predi

by sxg 5y ago

Yes, it's absolutely very difficult to do rigorous statistical analysis. Genetics is oversimplified to non-physicians. It's cool that we can diagnose and predict the likelihood of getting Huntington's disease using our knowledge of genetics, but extremely few diseases are this simple. There are huge swaths of the human genome that we don't understand but are likely playing some important role in the regulation of other genes and diseases. We are nowhere close to being able to look at a patient's genome to predict anything useful outside of a handful of exceptions. Patient histories are honestly often garbage—I say that as a physician. I look through dozens of patients' charts every day, and there are constantly errors, incomplete documentation, and fragmented records across multiple institutions. Just last week I read a chart for a patient who had a documented hysterectomy from years ago. The brand new CT scan I saw showed a perfectly normal uterus. Once something goes in a patient's history it's nearly impossible to correct or remove. If some doctor from ages ago said the patient is allergic to medication X, but the patient denies it, what do I do? Usually, we opt to leave the allergy listed out of fear of the consequences if the patient is wrong.

grog454 5y ago
Are you aware of any attempts to use machine learning in medical analysis and outcome prediction? I feel like this is one of the few applications where it could shine. I have no formal training in data science, but everything I've read so far seems to indicate that noisy and unreliable data is not an insurmountable problem. All of this talk about how "hard" the statistical analysis is, is strange to me. Maybe "advanced" would be a better term? If you get a patient with a contradictory medical history that somehow also contradicts what they are telling you, simply adjust your expected chance of success appropriately (to zero perhaps). In that extreme case, if you get a good outcome, congrats you got lucky. If you don't, it should have 0 impact on how you are evaluated as a doctor.
- rscho 5y ago
  Statistics are not the problem. The data is. 'Big healthcare data' does not exist. Building what you are thinking of would require huge data gathering capabilities, that are very clearly out of reach. Most big medical co do a lot of data science (Kaiser and others). Very efficient from a managerial pov. Totally useless, medically speaking.
- asmithmd1 5y ago
  > I feel like this is one of the few applications where it could shine. Read “The Alignment Problem”, a very good just above pop sci level book about machine learning. They have one example where ML determined seniors with COPD were at reduced risk from pneumonia, and obviously non-sensical result. Patients with COPD wound up in the hospital at a lower rate than average because doctors know they need careful attention right away.
- sxg 5y ago
  Before you can even do the statistical analysis you suggest, you need large amounts of high quality data—which we don't have. One place where the US (and the world?) gets data privacy right is in healthcare, but unfortunately that also means it's nearly impossible to create the data sets we need to do the statistical analysis you want. Institutions face severe penalties for wrongfully sharing patient data, so most opt to just not share any data. Any research that is performed is done internally on local populations with de-identified data sets. A few brave institutions go well out of their way to create and share de-identified data sets publically, but these data sets still undersample the general population. This is a critical problem because certain diseases are highly prevalent in certain regions (e.g., Lyme disease in New England) but unheard of in other regions (e.g., Lyme disease in Colorado). If your ML model is trained on data largely from New England, it's going to diagnose a patient with the classic "target-shaped" rash with Lyme disease even if the patient is from Colorado (high false positive rate). If the model is trained on data from Colorado, it will underdiagnose Lyme disease in patients from New England (high false negative rate). The only way I know to overcome this problem is to create even larger data sets, but this just isn't possible with data privacy laws.
  - grog454 5y ago
    > If the model is trained on data from Colorado, it will underdiagnose Lyme disease in patients from New England (high false negative rate). The only way I know to overcome this problem is to create even larger data sets, but this just isn't possible with data privacy laws. My understanding is that high dimensionality in the domain is exactly where ML excels (so add location as an input), and that is exactly what a medical diagnosis involves. Perhaps the legislation will get there one day.
  - clort 5y ago
    Re data privacy, as of last summer in the UK we had to opt out of data sharing of our NHS records with third parties. I am conflicted about this because as you suggested, large amounts of data could be beneficial for things like statistical analysis and machine learning to help medical research. On the other hand, what we know about human nature seems to indicate that there will already be sociopaths slavering over this data in order to enrich themselves at the expense of everybody else. https://inews.co.uk/opinion/nhs-data-shared-third-parties-weeks-opt-out-gps-worried-1028042 https://inews.co.uk/opinion/nhs-data-shared-third-parties-we... Anyway, its out there now.. so watch this space, I guess!
- chefandy 5y ago
  Yeah: https://www.nature.com/articles/d41586-019-03228-6 https://www.nature.com/articles/d41586-019-03228-6 I’m reluctantly pessimistic about humanity’s near-term capability to appropriately weight input from technology as imperfect but inscrutable as current ml.
- Fomite 5y ago
  I am. In that I have several CDC funded projects using machine learning in medical analysis and outcome prediction. Even on extremely well curated data sets this is a fairly hard problem.
- the_d3f4ult 5y ago
  Everyone and their mother is working on ML applications in medicine.
stuckonempty 5y ago
> patient histories are often garbage This is a little like programmers calling the programming language they invented and write in every day garbage. Separate from patient-reported histories, you medical doctors are the ones documenting these histories and hold the decision making power for how it’s done!
- tharkun__ 5y ago
  I think your analogy is a bit off here. Your first sentence would be the equivalent of the inventor of patient history data keeping calling patient history keeping a garbage tool. Which is actually something that would be totally OK to do and say if suffixed with "and unfortunately so far nobody has come up with a better tool and it's not for lack of trying". I'm assuming you didn't mean "you medical doctors" in the sense it's easy to read in. In any case, what you are doing here is telling one doctor that he is bad at the medical history writing and reading job when in fact he is the one telling you how he is able to spot other doctor's mistakes and trying to correct them. This is like telling one developer that he's bad at his job, that "you developers are the ones writing bad code and hold the decision making power for how it's done" when that developer is actually someone that tries to make things better both through his own maintainably written code (medical histories) and helping others in code reviews to make their code better and not let bad code get into Prod (finding errors in existing medical histories and trying to correct them). That can be very discouraging, being thrown in with the bad apples. And even good apples can have a bad day or misunderstand something. But I guess you are perfect and have never produced a bug in your life.
LorenPechtel 5y ago
Yup. As a patient I've found a bunch of bogus allergies in my records once that I was able to get fixed, but I know there's other stuff I've tried but the doc wasn't interested in listening. He felt he had "proved" I was mistaken/faking when he didn't listen in the first place. How am I supposed to identify the evil item in a challenge when it isn't even present? (I had said "ham". The challenge was with pork. I didn't say pork--whatever the actual evil item it was something that got added in the process of turning pork into ham.) How do you get a mistake like that out of your records?
- smeej 5y ago
  I've had doctors make up things as egregious as my height. I had one doc I had only met remotely who entered a height into my online, shared record that was several inches shorter than I was...back when I was an 11-year-old child. It's been 25 years since I was that short. I pressed them that they could at least have asked me how tall I am, or even consulted the previous entries by other doctors in their same system. There wasn't even an attempt at an excuse. It was plain, simple negligence. That doctor had also been insisting I needed to let him do an exploratory surgery, despite never even having had me come to the office in person, so I noped right out of there and started telling that story to everyone I thought might consider seeing him.

by sxg 5y ago

grog454 5y ago

Are you aware of any attempts to use machine learning in medical analysis and outcome prediction? I feel like this is one of the few applications where it could shine. I have no formal training in data science, but everything I've read so far seems to indicate that noisy and unreliable data is not an insurmountable problem. All of this talk about how "hard" the statistical analysis is, is strange to me. Maybe "advanced" would be a better term? If you get a patient with a contradictory medical history that somehow also contradicts what they are telling you, simply adjust your expected chance of success appropriately (to zero perhaps). In that extreme case, if you get a good outcome, congrats you got lucky. If you don't, it should have 0 impact on how you are evaluated as a doctor.

rscho 5y ago
Statistics are not the problem. The data is. 'Big healthcare data' does not exist. Building what you are thinking of would require huge data gathering capabilities, that are very clearly out of reach. Most big medical co do a lot of data science (Kaiser and others). Very efficient from a managerial pov. Totally useless, medically speaking.
asmithmd1 5y ago
> I feel like this is one of the few applications where it could shine. Read “The Alignment Problem”, a very good just above pop sci level book about machine learning. They have one example where ML determined seniors with COPD were at reduced risk from pneumonia, and obviously non-sensical result. Patients with COPD wound up in the hospital at a lower rate than average because doctors know they need careful attention right away.
sxg 5y ago
Before you can even do the statistical analysis you suggest, you need large amounts of high quality data—which we don't have. One place where the US (and the world?) gets data privacy right is in healthcare, but unfortunately that also means it's nearly impossible to create the data sets we need to do the statistical analysis you want. Institutions face severe penalties for wrongfully sharing patient data, so most opt to just not share any data. Any research that is performed is done internally on local populations with de-identified data sets. A few brave institutions go well out of their way to create and share de-identified data sets publically, but these data sets still undersample the general population. This is a critical problem because certain diseases are highly prevalent in certain regions (e.g., Lyme disease in New England) but unheard of in other regions (e.g., Lyme disease in Colorado). If your ML model is trained on data largely from New England, it's going to diagnose a patient with the classic "target-shaped" rash with Lyme disease even if the patient is from Colorado (high false positive rate). If the model is trained on data from Colorado, it will underdiagnose Lyme disease in patients from New England (high false negative rate). The only way I know to overcome this problem is to create even larger data sets, but this just isn't possible with data privacy laws.
- grog454 5y ago
  > If the model is trained on data from Colorado, it will underdiagnose Lyme disease in patients from New England (high false negative rate). The only way I know to overcome this problem is to create even larger data sets, but this just isn't possible with data privacy laws. My understanding is that high dimensionality in the domain is exactly where ML excels (so add location as an input), and that is exactly what a medical diagnosis involves. Perhaps the legislation will get there one day.
- clort 5y ago
  Re data privacy, as of last summer in the UK we had to opt out of data sharing of our NHS records with third parties. I am conflicted about this because as you suggested, large amounts of data could be beneficial for things like statistical analysis and machine learning to help medical research. On the other hand, what we know about human nature seems to indicate that there will already be sociopaths slavering over this data in order to enrich themselves at the expense of everybody else. https://inews.co.uk/opinion/nhs-data-shared-third-parties-weeks-opt-out-gps-worried-1028042 https://inews.co.uk/opinion/nhs-data-shared-third-parties-we... Anyway, its out there now.. so watch this space, I guess!
chefandy 5y ago
Yeah: https://www.nature.com/articles/d41586-019-03228-6 https://www.nature.com/articles/d41586-019-03228-6 I’m reluctantly pessimistic about humanity’s near-term capability to appropriately weight input from technology as imperfect but inscrutable as current ml.
Fomite 5y ago
I am. In that I have several CDC funded projects using machine learning in medical analysis and outcome prediction. Even on extremely well curated data sets this is a fairly hard problem.
the_d3f4ult 5y ago
Everyone and their mother is working on ML applications in medicine.

stuckonempty 5y ago

> patient histories are often garbage This is a little like programmers calling the programming language they invented and write in every day garbage. Separate from patient-reported histories, you medical doctors are the ones documenting these histories and hold the decision making power for how it’s done!

tharkun__ 5y ago
I think your analogy is a bit off here. Your first sentence would be the equivalent of the inventor of patient history data keeping calling patient history keeping a garbage tool. Which is actually something that would be totally OK to do and say if suffixed with "and unfortunately so far nobody has come up with a better tool and it's not for lack of trying". I'm assuming you didn't mean "you medical doctors" in the sense it's easy to read in. In any case, what you are doing here is telling one doctor that he is bad at the medical history writing and reading job when in fact he is the one telling you how he is able to spot other doctor's mistakes and trying to correct them. This is like telling one developer that he's bad at his job, that "you developers are the ones writing bad code and hold the decision making power for how it's done" when that developer is actually someone that tries to make things better both through his own maintainably written code (medical histories) and helping others in code reviews to make their code better and not let bad code get into Prod (finding errors in existing medical histories and trying to correct them). That can be very discouraging, being thrown in with the bad apples. And even good apples can have a bad day or misunderstand something. But I guess you are perfect and have never produced a bug in your life.

LorenPechtel 5y ago

Yup. As a patient I've found a bunch of bogus allergies in my records once that I was able to get fixed, but I know there's other stuff I've tried but the doc wasn't interested in listening. He felt he had "proved" I was mistaken/faking when he didn't listen in the first place. How am I supposed to identify the evil item in a challenge when it isn't even present? (I had said "ham". The challenge was with pork. I didn't say pork--whatever the actual evil item it was something that got added in the process of turning pork into ham.) How do you get a mistake like that out of your records?

smeej 5y ago
I've had doctors make up things as egregious as my height. I had one doc I had only met remotely who entered a height into my online, shared record that was several inches shorter than I was...back when I was an 11-year-old child. It's been 25 years since I was that short. I pressed them that they could at least have asked me how tall I am, or even consulted the previous entries by other doctors in their same system. There wasn't even an attempt at an excuse. It was plain, simple negligence. That doctor had also been insisting I needed to let him do an exploratory surgery, despite never even having had me come to the office in person, so I noped right out of there and started telling that story to everyone I thought might consider seeing him.