13 ms·
AI models miss disease in Black and female patients
- hhh1111 2y ago[dead]
- mlinhares 2y ago[flagged]
- _bin_ 2y agoThis seems like a problem that should be worked on It also seems like we shouldn't let it prevent all AI deployment in the interim. It is better that we take the disease detection rate for part of the population up a few percent than we do not. Plus it's not like doctors or radiologists always diagnose at perfectly equal accuracy across all populations. Let's not let the perfect become the enemy of the good.
- Larrikin 2y ago[flagged]
- diordiderot 2y agoJust opt out blacks and females no?
- appleorchard46 2y agoThis is obviously a jab, but to answer the unspoken question: if it's still more effective than human doctors for black or female people, then yes it should be used. If it isn't, then don't use it for them. Simple as that. (fixing this problem should be a high priority either way, that goes without saying)
- megaman821 2y agoIf I'm a black female in a remote village with no doctor, I'm absolutely going to roll the dice with AI. In the mean time these problems should be surfaced and corrected.
- rvnx 2y agoEven if you are not a remote black woman: AI can be a serious helping hand when trying to figure out what diseases you have. Not everybody has access to doctors, sometimes, whole countries don't have good specialists in everything. When Google refuses to answer medical questions (when it could have answered and said "be careful, I'm not really sure"), it's actually putting lives in danger by hiding useful information. If you can, of course it's better to go to a triple Nobel Prize doctor that is 200 USD per session, but not everybody can, and these doctors may not even be available in your country. But otherwise in general, it's better to have the opinion of AI than nothing.
- nradov 2y agoFalse positive diagnoses cause a huge amount of patient harm. New technologies should only be deployed on a widespread basis when they are justified based on solid evidence-based medicine criteria.
- nonethewiser 2y agoNo one says you have to use the AI models stupidly. If it works poorly for black women and female women dont use it for them. Or simply dont use it for the initial diagnosis. Use it after the normal diagnosis process as more of a validation step. Anyways, this all points to the need to capture biological information as input or even having seperately models tuned to different factors.
- nradov 2y agoThe guidelines on how to use a particular AI model can only be written after extensive clinical research and data analysis. You can't skip that step without endangering patients, and it will take years to do properly for each one.
- Avshalom 2y agoEvery single AI company says you should use AI models stupidly. Replacing experts is the whole selling point.
- nonethewiser 2y agoOK so should we optimize for blindly listening to AI companies then?
- Avshalom 2y agoWe should assume people will use tools in the manner that they have been sold those tools yes.
- nonethewiser 2y ago
- bilbo0s 2y agoMmmm... You don't work in healthcare do you? I think it would be extremely bad if people found out that, um, "other already disliked/scapegoated people", get actual doctors and nurses working on them, but "people like me" only get the doctor or nurse checking an AI model. I'm saying that if you were going to do that, you'd better have an extremely high degree of secrecy about what you were doing in the background. Like, "we're doing this because it's medical research" kind of secrecy. Because there's a bajillion ways that could go sideways in today's world. Especially if that model performs worse than some rockstar doctor that's now freed up to take his/her time seeing the, uh, "other already disliked/scapegoated population". Your hospital or clinic's statistics start to look a bit off. Joint commission? Medical review boards? Next thing you know certain political types are out telling everyone how a certain population is getting preferential treatment at this or that facility. And that story always turns into, "All around the nation they're using AI to get <scapegoats> preferential treatment". It's just a big risk unless you're 100% certain that model can perform better than your best physician. Which is highly unlikely. This is the sort of thing you want to do the right way. Especially nowadays. Politics permeates everything in healthcare right now.
- tawayf1 2y ago[flagged]
- jcims 2y agoJust like doctors: https://kffhealthnews.org/news/article/medical-misdiagnosis-women-minorities-health-care-bias/ https://kffhealthnews.org/news/article/medical-misdiagnosis-... I wonder how well it does with folks that have chronic conditions like type 1 diabetes as a population. Maybe part of the problem is that we're treating these tools like humans that have to look at one fuzzy picture to figure things out. A 'multi-modal' model that can integrate inputs like raw ultrasound doppler, x-ray, ct scan, blood work, ekg, etc etc would likely be much more capable than a human counterpart.
- CharlesW 2y agoIt seems critical to have diverse, inclusive, and equitable data for model training. (I call this concept "DIET".)
- nonethewiser 2y agoOr take more inputs. If there are differences between race and gender and thats not captured as an input we should expect the accuracy to be lower. If an x-ray means different things based off the race or gender we should make sure the model knows the race and gender.
- 0cf8612b2e1e 2y agoFunny you should say that. There was a push to have more officially collected DIET data for exactly this reason. Unfortunately such efforts were recently terminated.
- appleorchard46 2y agoI'm calling it now. My prediction is that, 5-10 years from now(ish), once training efficiency has plateaued, and we have a better idea of how to do more with less, curated datasets will be the next big thing. Investors will throw money at startups claiming to make their own training data by consulting experts, finetuning as it is now will be obsolete, pre-ChatGPT internet scrapes will be worth their weight in gold. Once a block is hit on what we can do with data, the data itself is the next target.
- red75prime 2y agoAnd not applying fairness techniques to the resulting model.
- deleted 2y ago[deleted]
- AlecSchueler 2y agoHumans do the same. Everything from medical studies to doctor trainings treat the straight white man as the "default human" and this obviously leads to all sorts of issues. Caroline Criado-Perez has an entire chapter about this in her book about systemic bias Invisible Women, with a scary number of examples and real world consequences. It's no surprise that AI training sets reflect this also. People have been warning against it [0] specifically for at least 5 years. 0: https://www.pnas.org/doi/10.1073/pnas.1919012117 https://www.pnas.org/doi/10.1073/pnas.1919012117 Edit: I've never had a comment so heavily downvoted so quickly. I know it's not the done thing to complain but HN really feels more and more like a boys club sometimes. Could anyone explain what they find so contentieus about what I've said?
- unsupp0rted 2y agoEverybody knows that gay men have more livers and fewer kidneys than straight men
- AlecSchueler 2y agoWhy the snark? The OP, the study I linked and the book I referenced which contains many well researched examples of issues caused by defaultism surely represent a strong enough body of work that they should deserve a more engaged critique.
- consteval 2y agoNo, but they do have different risk profiles for various diseases and drug use. Surprise surprise, that affects diagnoses and treatment.
- Animats 2y agoWhat's so striking is how strongly race shows in X-rays. That's unexpected.
- banqjls 2y agoBut is it really?
- deleted 2y ago[deleted]
- danielmarkbruce 2y agoThe fact that the vast majority of physical differences don't matter in the modern world doesn't mean they don't actually exist..
- DickingAround 2y agoThis is a good point; a man or woman sitting behind a desk doing correlation analysis are going to look very similar in their function to a business. But they probably physically look pretty distinct to an x-ray picture.
- deleted 2y ago[deleted]
- sergiotapia 2y agoIt's odd how we can segment between different species in animals, but in humans it's taboo to talk about this. Threw the baby out with the baby water. I hope we can fix this soon so everybody can benefit from AI. The fact that I'm a male latino should be an input for an AI trained on male latinos! I want great care! I don't want pretend kumbaya that we are all humans in the end. That's not true. We are distinct! We all deserve love and respect and care, but we are distinct!
- schnable 2y agoThat's because humans are all the same species.
- yieldcrv 2y agojust giving globs of training sets and letting a process cook for a few months is just going to be seen as lazy in the near future more specialization of models is necessary, now that there is awareness
- computerthings 2y ago[dead]
- acobster 2y agoSpecialization in what though? Do you really think VCs are going to drive innovation on equitable outcomes? Where is the money in that? I have a hunch that oppression will continue to be profitable.
- yieldcrv 2y agothe model involved in this article was developed by Stanford, and tested by UCLA so yes I do believe that models will be created with more specific datasets, which is the specialization I was referring to
- deleted 2y ago[deleted]
- elietoubi 2y agoI came across a fascinating Microsoft research paper on MedFuzz (https://www.microsoft.com/en-us/research/blog/medfuzz-exploring-the-robustness-of-llms-on-medical-challenge-problems/ https://www.microsoft.com/en-us/research/blog/medfuzz-explor...) that explores how adding extra, misleading prompt details can cause large language models (LLMs) to arrive at incorrect answers. For example, a standard MedQA question describes a 6-year-old African American boy with sickle cell disease. Normally, the straightforward details (e.g., jaundice, bone pain, lab results) lead to “Sickle cell disease” as the correct diagnosis. However, under MedFuzz, an “attacker” LLM repeatedly modifies the question—adding information like low-income status, a sibling with alpha-thalassemia, or the use of herbal remedies—none of which should change the actual diagnosis. These additional, misleading hints can trick the “target” LLM into choosing the wrong answer. The paper highlights how real-world complexities and stereotypes can significantly reduce an LLM’s performance, even if it initially scores well on a standard benchmark. Disclaimer: I work in Medical AI and co-founded the AI Health Institute (https://aihealthinstitute.org/ https://aihealthinstitute.org/).
- AnimalMuppet 2y agoUnfortunately, humans talking to a doctor give lots of additional, misleading hints...
- onlyrealcuzzo 2y agoIt's almost as if you'd want to not feed what the patient says directly to an LLM. A non-trivial part of what doctors do is charting - where they strip out all the unimportant stuff you tell them unrelated to what they're currently trying to diagnose / treat, so that there's a clear and concise record. You'd want to have a charting stage before you send the patient input to the LLM. It's probably not important whether the patient is low income or high income or whether they live in the hood or the uppity part of town.
- nradov 2y agoI generally agree, however socioeconomic and environmental factors are highly correlated with certain medical conditions (social determinants of health). In some cases even causative. For example, patients who live near an oil refinery are more likely to have certain cancers or lung diseases. https://doi.org/10.1093/jncics/pkaa088 https://doi.org/10.1093/jncics/pkaa088
- orr94 2y ago"AIs want the future to be like the past, and AIs make the future like the past. If the training data is full of human bias, then the predictions will also be full of human bias, and then the outcomes will be full of human bias, and when those outcomes are copraphagically fed back into the training data, you get new, highly concentrated human/machine bias.” https://pluralistic.net/2025/03/18/asbestos-in-the-walls/#government-by-spicy-autocomplete https://pluralistic.net/2025/03/18/asbestos-in-the-walls/#go...
- mhuffman 2y ago"The model used in the new study, called CheXzero, was developed in 2022 by a team at Stanford University using a data set of almost 400,000 chest x-rays of people from Boston with conditions such as pulmonary edema, an accumulation of fluids in the lungs. Researchers fed their model the x-ray images without any of the associated radiologist reports, which contained information about diagnoses. " ... very interesting that the inputs to the model had nothing related to race or gender, but somehow it still was able to miss diagnose Black and female patients? I am curious of the mechanism for this. Can it just tell which x-rays belong to Black or female patients and then use some latent racism or misogyny to change the diagnosis? I do remember when it came out that AI could predict race from medical images with no other information[1], so that part seems possible. But where would it get the idea to do a worse diagnosis, even if it determines this? Surely there is no medical literature that recommends this! [1]https://news.mit.edu/2022/artificial-intelligence-predicts-patients-race-from-medical-images-0520 https://news.mit.edu/2022/artificial-intelligence-predicts-p...
- protonbob 2y agoI'm going to wager an uneducated guess. Black people are less likely to go to the doctor for both economic and historical reasons so images from them are going to be underrepresented. So in some way I guess you could say that yes, latent racism caused people to go to the doctor less which made them appear less in the data.
- apical_dendrite 2y ago
- blogabegonija 2y ago[dead]
- nonethewiser 2y agoRace and gender should be inputs then. The female part is actually a bit more surprising. Its easy to imagine a dataset not skewed towards black people. ~15% of the population in North America, probably less in Europe, and way less in Asia. But female? Thats ~52% globally.
- krapp 2y agoModern medicine has long operated under the assumption that whatever makes sense in a male body also makes sense in a female body, and womens' health concerns were often dismissed, misdiagnosed or misunderstood in patriarchal society. Women were rarely even included in medical trials prior to 1993. As a result, there is simply a dearth of medical research directly relevant to women for models to even train on.
- Avshalom 2y agohttps://www.npr.org/2022/11/01/1133375223/the-first-female-crash-test-dummy-has-only-now-arrived https://www.npr.org/2022/11/01/1133375223/the-first-female-c... Twenty Twenty Two!
- mrguyorama 2y agoRepublicans early in this admin actually bitched in congress that we were "wasting" money on woman crash test dummies. https://www.foxnews.com/video/6325465806112 https://www.foxnews.com/video/6325465806112
- chrisgarand 2y agoI'm going to lay this out how I understand it: The NIH Revitalization Act of 1993 was supposed to bring women back into medical research. The reality was that women were always included, HOWEVER in 1977,(1) because of the outcomes from thalidomide (causing birth defects), "women of childbearing potential" were excluded from the phase 1, and early phase 2 trials (the highest risk trials). They're still generally generally excluded, even after the passage of the act. This was/is to protect the women, and potential children. According to Edward E. Bartlett in his meta data analysis from 2001, men have been routinely under-represented in NIH data (even before adjusting for men's mortality rates) between 1966-1990. (2) There's also routinely twice as much spent every year on women's health studies vs men's by the NIH. (3) It makes sense to me, but I'm biased. Logically, since men lead in 9 of the top 10 causes for death, that shows there's something missing in the equation of research. (4 - It's not a straight forward table, you can view the total deaths, and causes and compare the two for men, and women) With that being said, it doesn't tell us about the quality of the funding or research topics, maybe the money is going towards pointless goals, or unproductive researchers. Are there gaps in research? Most definitely, like women who are pregnant. This is put in place to avoid harm but that doesn't help them when they fall into them. Are there more? Definitely. I'm not educated enough in the nuances to go into them. If you have information that counters what I've posted, please share it, I would love know where these folks are blind so I can take a look at my bias. (1) https://petrieflom.law.harvard.edu/2021/04/16/pregnant-clinical-trials-safety-autonomy/ https://petrieflom.law.harvard.edu/2021/04/16/pregnant-clini... (2) https://journals.lww.com/epidem/fulltext/2001/09000/did_medical_research_routinely_exclude_women__an.20.aspx https://journals.lww.com/epidem/fulltext/2001/09000/did_medi... (3) https://jameslnuzzo.substack.com/p/nih-funding-of-mens-and-womens-health https://jameslnuzzo.substack.com/p/nih-funding-of-mens-and-w... < I spot checked a couple of the figures, and those lined up. I'm assuming the rest is accurate (4) https://www.cdc.gov/womens-health/lcod/index.html#:~:text=Overall%2C%20the%20leading%20cause%20of,%25%20for%20females)%20in%202021. https://www.cdc.gov/womens-health/lcod/index.html#:~:text=Ov...
- jimnotgym 2y agoJust as good as a real doctor then?
- josefritzishere 2y agoAI does a terrible job huh? I wish there was a way we have done this for decades without that problem...
- LadyCailin 2y agoGood thing we got rid of DEI.
- kjkjadksj 2y agoThis isn’t an AI problem but a general medical field problem. It is a big issue with basically any population centric analysis where the people involved in the study don’t have a perfect subset of the worlds population to model human health; they have a couple hundred blood samples from patients at a Boise hospital over the past 10 years perhaps. And they validate this population against some other available cohort that is similarly constrained by what is practically possible to sample and catalog and might not even see the same markers shake out between disease and healthy. There are a couple populations that are really overrepresented as a result of these available datasets. Utah populations on one hand because they are genetically bottlenecked and therefore have better signal to noise in theory. And on the other the Yoruba tribe out of west africa as a model of the most diverse and ancestral population of humans for studies that concern themselves with how populations evolved perhaps. There are other projects too amassing population data. About 2/3rd of the population of iceland has been sequenced and this dataset is also frequently used.
- cratermoon 2y agoIt's a generative AI LLM hype issue because it follows the confidence game playbook. Feed someone correct ideas and answers that fit their biases until they trust you, then when the time is right, suggest things that fit their biases but give incorrect (and exploitative) results.
- NotYourLawyer 2y ago[flagged]
- nonethewiser 2y agoInteresting catch. TIL I learned about this split. >At the Columbia Journalism Review, we capitalize Black, and not white, when referring to groups in racial, ethnic, or cultural terms. For many people, Black reflects a shared sense of identity and community. White carries a different set of meanings; capitalizing the word in this context risks following the lead of white supremacists. https://www.cjr.org/analysis/capital-b-black-styleguide.php https://www.cjr.org/analysis/capital-b-black-styleguide.php
- NotYourLawyer 2y ago[flagged]
- antipaul 2y agoWhen was AI supposed to replace radiologists? Was it 7 years ago or something?
- chadd 2y agoA surprisingly high number of medical studies will not include women because the study doesn't want to account for "outliers" like pregnancy and menstrual cycles[0]. This is bound to have effects on LLM answers for women. [0] https://www.northwell.edu/katz-institute-for-womens-health/articles/women-overlooked-in-medical-research https://www.northwell.edu/katz-institute-for-womens-health/a...
- nottorp 2y ago> as well in those 40 years or younger Are we sure it's only about racial bias then? Looks to me like the training data set is too small overall. They had too few black people, too few women, but also too few younger people.
- xboxnolifes 2y agoIt's the same old story that's been occurring for years/decades. Bad data in, bad data out.
- olddog2 2y ago[flagged]
- sxp 2y agohttps://www.science.org/doi/10.1126/sciadv.adq0305 https://www.science.org/doi/10.1126/sciadv.adq0305 is the paper and https://www.science.org/cms/10.1126/sciadv.adq0305/asset/b6898f45-82c1-4486-9c3e-15c2e6c2df6a/assets/images/large/sciadv.adq0305-f3.jpg https://www.science.org/cms/10.1126/sciadv.adq0305/asset/b68... is the key graph.
- christkv 2y ago[flagged]
- zeagle 2y agoCool topic! This isn't surprising given the AI models would be trained such that existing medical practices, biases, and failures would propagate through them as others have said here. There is a published, recognized bias against women and blacks (borrowing the literature term) specifically in medicine when it comes to pain assessment and treatment. Racism is a part of it but too simplistic. Most of us don't go to work trying to be horrible people. I was in a fly in community earlier this week for work where 80% of housing is subsidized social housing... so spit balling a bit... things like assumptions about rate of metabolizing medications being equal, assess to medication, culture and stoicism, dismissing concerts, and the broad effects of poverty/trauma/inter-generational trauma all must play a role in this. For interest: https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2793179 https://jamanetwork.com/journals/jamanetworkopen/fullarticle... Overall, the authors found comparable ratings in Black and White participants’ perceptions of the patient-physician relationship across all three measures (...) Alternatively, the authors found significant racial differences in the pain-related outcomes, including higher pain intensity and greater back-related disability among Black participants compared with White participants (intensity mean: 7.1 vs 5.8; P < .001; disability mean: 15.8 vs 14.1; P < .001). The quality of the patient-physician relationship did not explain the association between participant race and the pain outcomes in the mediation analysis. https://www.aamc.org/news/how-we-fail-black-patients-pain https://www.aamc.org/news/how-we-fail-black-patients-pain (top line summary) Half of white medical trainees believe such myths as black people have thicker skin or less sensitive nerve endings than white people. An expert looks at how false notions and hidden biases fuel inadequate treatment of minorities’ pain. And https://www.washingtonpost.com/wellness/interactive/2022/women-pain-gender-bias-doctors/ https://www.washingtonpost.com/wellness/interactive/2022/wom...
- tennisflyi 2y agoYes. Almost certain there are dedicated books to IDing/how diseases present differently on skin other than white
- jdthedisciple 2y agoAnyone who thinks that the primary culprit for this is anything other than the input data distribution (and metadata inputs or the lack thereof) lacks even the most basic understanding of AI.
- b112 2y agoI remember a male and female specialist, whatever their discipline, holding a media scrum a decade ago. They pleaded for people to understand that men and women are physically different, including the brain, its neurological structure, and that this was in modern medicine being overlooked for political reasons. One of the results was that many clinical trials and studies were populated by males only. The theory being that they are less risk adverse, and as "there is no difference", then who cares? Well these two cared, and said that it was hurting medical outcomes for women. I wonder, if this AI issue is a result of this. Fewer examples of female bodies and brains, fewer studies and trials, means less data to match on... https://news.harvard.edu/gazette/story/2007/07/sex-differences-in-brains-reflect-disease-risks/ https://news.harvard.edu/gazette/story/2007/07/sex-differenc...
- mg794613 2y agoThats bad! Let's change that! Let's be better than our predecessors! Right? So, how do they suggest to tackle the problem? 1. Improve the science 2. Update the data or 3. Somehow focus on it being racist and then walking away like the hero of the day without actually solving the problem.
- saagarjha 2y agoYou'll note that they suggest both.
- sabareesh 2y agoRaw models may not be good enough. I wonder how thinking models do on these
- shermantanktop 2y agoThe article addresses much of what is incorrectly speculated on in the comments here. “Researchers fed their model the x-ray images without any of the associated radiologist reports, which contained information about diagnoses” (including demographics).
- csomar 2y ago> Compared with the patients’ doctors, the AI model more often failed to detect the presence of disease in Black patients or women, as well in those 40 years or younger. Garbage article. Garbage study. And garbage AI model that doesn't account for most of its audience.
- FalseNutrition 2y agoTo be honest, I'm not sure if I understand what is actually claimed here (it seems that they trained the model on their own, and claim that the problem is in the dataset?) but isn't the more sensible explanation that human doctors were overdiagnosing them?
- draw_down 2y ago[dead]