5 ms·

What actually is so hard about AI in health care? Why not just take a set of diagnostic indicators for inputs, map to conditions/treatments as outputs and train

What actually is so hard about AI in health care? Why not just take a set of diagnostic indicators for inputs, map to conditions/treatments as outputs and train a neural net?

blueadept111 8y ago
Either the AI can generalize or it can't. If it can't, you don't need neural nets at all. If it can, you need to worry about whether it's generalizing correctly, which is definitely "so hard".
deleted 8y ago
[deleted]
scottlocklin 8y ago
Sparse data. Neural nets (for example) do horribly at this sort of thing; they're data hungry. It's super difficult to beat something like linear regression in this sort of thing (ideally combined with domain knowledge -something neural approaches mostly fail at), and even linear regression gives awful results.
1e-9 8y ago
There are huge barriers to getting a healthcare AI product to market in the United States. First off, data is generally extremely difficult and expensive to gather due to the need to protect patient info according to the Health Insurance Portability and Accountability Act (HIPAA). To get representative data, you need to gather from many sites and the data needs to be anonymized and "truthed" by a medical expert who typically must examine multiple medical reports and case history for each patient. Each data collection site requires approval for data collection by an Institutional Review Board, which may take months or even years to get. Assuming you are able to get a reasonable amount of data for development and testing, you need to solve the actual AI problem (probably very difficult due to the extremely heterogeneous nature of disease you are dealing with) and then you need regulatory approval. Depending on the FDA classification of your product, this can be another many-month to many-year process of data collection and clinical studies.
Ensorceled 8y ago
The problem is that the numerous easy cases do not result in a useful network ... any 1st-year intern will get the easy results already. The hard cases, which would be useful to a doctor, occur very rarely. My father's unusual reaction to a post-bypass drug regimen was something like the 3rd time that happened in Canada. How do you "train" that into a neural network?
- draugadrotten 8y ago
  Sounds like anomaly detection would be quite useful for monitoring the expected reactions and comparing with actual outcomes. It may not be the answer to "what is the right drug?" but it may be enough to say "what he got ain't right for him"
- marmaduke 8y ago
  I work on an ML project on epilepsy, and this happened all the time. Additionally what the clinicians saw as easy/boring wasn’t always te case for the model.
reureu 8y ago
In addition to what other commenters have written, you also have the problem that your inputs aren't clean or straightforward. People don't always answer fully truthfully, and sometimes give extraneous information that's not germane to their particular complaint. So, even if you get past the regulatory hurdles of obtaining and using data, and you have enough data to train your neural network, and you've handled the huge class imbalance issue with rare diseases, you still have to deal with the fact that the inputs to your model are subject to all kinds of bias and confounding that's associated with human-generated data. To make this concrete: I used to do a lot of HIV testing and counseling. Whenever I'd ask someone "how often do you use condoms?" they'd 100% of the time say "every time!" When I'd then switch to asking "when was the last time you didn't use a condom?" they'd often reply along the lines of "last week." This kind of issue happens not just with awkward questions, but also with more "objective" data like labs and medications. - Was the antibiotic script written for a patient necessary for their condition, or was the physician tired and couldn't fend off a particularly assertive patient who was convinced antibiotics would solve their viral infection? - Are labs randomly ordered, or ordered in a targeted fashion based on the "hunch" of a physician? If the latter, then the presence of lab result would itself be correlated with having some disease, and could thus throw off your entire model. - How does the patient's insurance status influence the tests they have ordered? At one clinic I'm aware of, they typically order HIV/GC/Chlam screenings as a bundle, but if the patient has a PPO insurance they are also more likely to throw in a syphilis screening. - How does patient preference factor into what gets diagnosed, ordered, and treated? You might have two early stage, equal risk breast cancer patients, one whom opts for a double mastectomy because she saw what happened to a coworker who had breast cancer and doesn't want to risk recurrences, whereas the other opts for a lumpectomy because she thinks her risks are low enough to be more conservative in treatment. These issues are all solvable, but for AI to have a useful impact in healthcare, there's a lot more work than just throwing a bunch of data into a deep learning model.
mr_toad 8y ago
There are hundreds of thousands of diagnostic and treatment codes, and most patients will have several, or even dozens of codes for a presentation. Add in the patients age and other demographic variables, medical history, and family history, and you have more combinations of variables than patients. Except for some really common problems, like cases of influenza, every patient is unique. AI can be used to handle simpler problems - identifying patients at risk of repeat admissions, or flagging the likelihood of dehydration for example.

What actually is so hard about AI in health care? Why not just take a set of diagnostic indicators for inputs, map to conditions/treatments as outputs and train a neural net?

blueadept111 8y ago

Either the AI can generalize or it can't. If it can't, you don't need neural nets at all. If it can, you need to worry about whether it's generalizing correctly, which is definitely "so hard".

deleted 8y ago

[deleted]

scottlocklin 8y ago

Sparse data. Neural nets (for example) do horribly at this sort of thing; they're data hungry. It's super difficult to beat something like linear regression in this sort of thing (ideally combined with domain knowledge -something neural approaches mostly fail at), and even linear regression gives awful results.

1e-9 8y ago

There are huge barriers to getting a healthcare AI product to market in the United States. First off, data is generally extremely difficult and expensive to gather due to the need to protect patient info according to the Health Insurance Portability and Accountability Act (HIPAA). To get representative data, you need to gather from many sites and the data needs to be anonymized and "truthed" by a medical expert who typically must examine multiple medical reports and case history for each patient. Each data collection site requires approval for data collection by an Institutional Review Board, which may take months or even years to get. Assuming you are able to get a reasonable amount of data for development and testing, you need to solve the actual AI problem (probably very difficult due to the extremely heterogeneous nature of disease you are dealing with) and then you need regulatory approval. Depending on the FDA classification of your product, this can be another many-month to many-year process of data collection and clinical studies.

Ensorceled 8y ago

The problem is that the numerous easy cases do not result in a useful network ... any 1st-year intern will get the easy results already. The hard cases, which would be useful to a doctor, occur very rarely. My father's unusual reaction to a post-bypass drug regimen was something like the 3rd time that happened in Canada. How do you "train" that into a neural network?

draugadrotten 8y ago
Sounds like anomaly detection would be quite useful for monitoring the expected reactions and comparing with actual outcomes. It may not be the answer to "what is the right drug?" but it may be enough to say "what he got ain't right for him"
marmaduke 8y ago
I work on an ML project on epilepsy, and this happened all the time. Additionally what the clinicians saw as easy/boring wasn’t always te case for the model.

reureu 8y ago

In addition to what other commenters have written, you also have the problem that your inputs aren't clean or straightforward. People don't always answer fully truthfully, and sometimes give extraneous information that's not germane to their particular complaint. So, even if you get past the regulatory hurdles of obtaining and using data, and you have enough data to train your neural network, and you've handled the huge class imbalance issue with rare diseases, you still have to deal with the fact that the inputs to your model are subject to all kinds of bias and confounding that's associated with human-generated data. To make this concrete: I used to do a lot of HIV testing and counseling. Whenever I'd ask someone "how often do you use condoms?" they'd 100% of the time say "every time!" When I'd then switch to asking "when was the last time you didn't use a condom?" they'd often reply along the lines of "last week." This kind of issue happens not just with awkward questions, but also with more "objective" data like labs and medications. - Was the antibiotic script written for a patient necessary for their condition, or was the physician tired and couldn't fend off a particularly assertive patient who was convinced antibiotics would solve their viral infection? - Are labs randomly ordered, or ordered in a targeted fashion based on the "hunch" of a physician? If the latter, then the presence of lab result would itself be correlated with having some disease, and could thus throw off your entire model. - How does the patient's insurance status influence the tests they have ordered? At one clinic I'm aware of, they typically order HIV/GC/Chlam screenings as a bundle, but if the patient has a PPO insurance they are also more likely to throw in a syphilis screening. - How does patient preference factor into what gets diagnosed, ordered, and treated? You might have two early stage, equal risk breast cancer patients, one whom opts for a double mastectomy because she saw what happened to a coworker who had breast cancer and doesn't want to risk recurrences, whereas the other opts for a lumpectomy because she thinks her risks are low enough to be more conservative in treatment. These issues are all solvable, but for AI to have a useful impact in healthcare, there's a lot more work than just throwing a bunch of data into a deep learning model.

mr_toad 8y ago

There are hundreds of thousands of diagnostic and treatment codes, and most patients will have several, or even dozens of codes for a presentation. Add in the patients age and other demographic variables, medical history, and family history, and you have more combinations of variables than patients. Except for some really common problems, like cases of influenza, every patient is unique. AI can be used to handle simpler problems - identifying patients at risk of repeat admissions, or flagging the likelihood of dehydration for example.