6 ms·

Adding this to my running list of algorithms encoding and amplifying systemic bias, like: * A hospital AI algorithm discriminating against Black people when pr

Adding this to my running list of algorithms encoding and amplifying systemic bias, like: * A hospital AI algorithm discriminating against Black people when providing additional healthcare outreach by amplifying racism already in the system. https://www.nature.com/articles/d41586-019-03228-6 https://www.nature.com/articles/d41586-019-03228-6 * Misdiagnosing people of African decent with genomic variants misclassified as pathogenic due to most of our reference data coming from European/white males. https://www.nejm.org/doi/full/10.1056/NEJMsa1507092 https://www.nejm.org/doi/full/10.1056/NEJMsa1507092 * When the dangers of ML in diagnosing Melanoma exacerbating healthcare disparities for darker skinned people. https://jamanetwork.com/journals/jamadermatology/article-abstract/2688587 https://jamanetwork.com/journals/jamadermatology/article-abs... * When Google's hate speech detecting AI inadvertantly censored anyone who used vernacular referred to in this article as being "African American English". https://fortune.com/2019/08/16/google-jigsaw-perspective-racial-bias/ https://fortune.com/2019/08/16/google-jigsaw-perspective-rac... * When Amazon's AI recruiting tool inadvertantly filtered out resumes from women. https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G https://www.reuters.com/article/us-amazon-com-jobs-automatio... * When AI criminal risk prediction software used by judges in deciding the severity of punishment for those convicted predicts a higher chance of future offence for a young, Black first time offender than for an older white repeat felon. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing https://www.propublica.org/article/machine-bias-risk-assessm... And here's some good news though: * When police wrongfully arrested a person based on faulty facial recognition match using grainy security camera footage, without any due diligence, asking for an alibi, or any other investigation. https://www.npr.org/2020/06/24/882683463/the-computer-got-it-wrong-how-facial-recognition-led-to-a-false-arrest-in-michig https://www.npr.org/2020/06/24/882683463/the-computer-got-it... * When the above is compounded for people of color according to studies which show that facial recognition systems misidentify dark-skinned women 40x more often than for light-skinned men. http://news.mit.edu/2018/study-finds-gender-skin-type-bias-artificial-intelligence-systems-0212 http://news.mit.edu/2018/study-finds-gender-skin-type-bias-a.... Another study showed false positives can be 10x to 100x more frequent for Asian and African American faces compared to Caucasian. https://www.nist.gov/news-events/news/2019/12/nist-study-evaluates-effects-race-age-sex-face-recognition-software https://www.nist.gov/news-events/news/2019/12/nist-study-eva... * When an algorithm blocked kidney transplants for Black patients. https://www.wired.com/story/how-algorithm-blocked-kidney-transplants-black-patients/ https://www.wired.com/story/how-algorithm-blocked-kidney-tra... * When clinical algorithms include “corrections” for race which directly raise the bar for the need for interventions in people of color, such that they then receive less clinical screening, less surveillance, less diagnoses, and less treatment for everything, including cancer, organ transplants, birth interventions, urinary and blood, bone, and heart disease. https://www.nejm.org/doi/10.1056/NEJMms2004740 https://www.nejm.org/doi/10.1056/NEJMms2004740

Rule35 6y ago
Maybe you should make multiple lists. There are some overt racist things, like redlining was, and gerrymandering can be, but a lot of those things appear to be natural accidents, or just looking at people without race before finding some physical (sickle-cell anemia) reason to. For instance: > * Misdiagnosing people of African decent with genomic variants misclassified as pathogenic due to most of our reference data coming from European/white males. It's obviously caused by studying the people who present, and then demographics changing. Nobody made a decision here with any ill intent. Both the hospital and the insurance company share the patient's interest in them getting better and are already looking for better data. > * When AI criminal risk prediction software used by judges in deciding the severity of punishment for those convicted predicts a higher chance of future offence for a young, Black first time offender than for an older white repeat felon. A higher chance of any offense, or of being a burden to society? Kids are more likely to commit smaller crimes, and those are a gateway to larger crimes that then put you away until you're a hardened repeat felon. If black kids were being enticed into crime we certainly want to know about it. If it's a subpopulation rather than an area the causes are likely different and good decisions only come from good data.
- JangoSteve 6y ago
  I'm not sure how you think my list would bifurcate into multiple lists. > It's obviously caused by studying the people who present, and then demographics changing. Nobody made a decision here with any ill intent. Both the hospital and the insurance company share the patient's interest in them getting better and are already looking for better data. Not a single example I gave was the result of ill intent. That's literally the point of my list. There's a difference between systemic bias and intentional bias. These are examples of systemic bias. > A higher chance of any offense, or of being a burden to society? I provided the link which answers your question. In this case, it predicted a higher likelihood of future crime for a kid who attempted to steal someone's bike and scooter, than for an adult who had shoplifted, been previously convicted of armed robbery, and served 5 years in prison already. > Kids are more likely to commit smaller crimes, and those are a gateway to larger crimes that then put you away until you're a hardened repeat felon. Your logic seems to acknowledge that one of the large drivers of crime is being "put away" in prison. In this case, the algorithm predicted a higher likelihood of committing future crimes for a kid than for someone who had already served 5 years in prison for armed robbery. Even worse is that the algorithm was being used to determine the severity of their respective punishments. So, the act of predicting a higher likelihood of future crime for the kid becomes a self-fulfilling prophecy, giving her a harsher sentence, which in turn is more likely to drive her toward future crime. This is systemic bias in action.

Rule35 6y ago

Maybe you should make multiple lists. There are some overt racist things, like redlining was, and gerrymandering can be, but a lot of those things appear to be natural accidents, or just looking at people without race before finding some physical (sickle-cell anemia) reason to. For instance: > * Misdiagnosing people of African decent with genomic variants misclassified as pathogenic due to most of our reference data coming from European/white males. It's obviously caused by studying the people who present, and then demographics changing. Nobody made a decision here with any ill intent. Both the hospital and the insurance company share the patient's interest in them getting better and are already looking for better data. > * When AI criminal risk prediction software used by judges in deciding the severity of punishment for those convicted predicts a higher chance of future offence for a young, Black first time offender than for an older white repeat felon. A higher chance of any offense, or of being a burden to society? Kids are more likely to commit smaller crimes, and those are a gateway to larger crimes that then put you away until you're a hardened repeat felon. If black kids were being enticed into crime we certainly want to know about it. If it's a subpopulation rather than an area the causes are likely different and good decisions only come from good data.

JangoSteve 6y ago
I'm not sure how you think my list would bifurcate into multiple lists. > It's obviously caused by studying the people who present, and then demographics changing. Nobody made a decision here with any ill intent. Both the hospital and the insurance company share the patient's interest in them getting better and are already looking for better data. Not a single example I gave was the result of ill intent. That's literally the point of my list. There's a difference between systemic bias and intentional bias. These are examples of systemic bias. > A higher chance of any offense, or of being a burden to society? I provided the link which answers your question. In this case, it predicted a higher likelihood of future crime for a kid who attempted to steal someone's bike and scooter, than for an adult who had shoplifted, been previously convicted of armed robbery, and served 5 years in prison already. > Kids are more likely to commit smaller crimes, and those are a gateway to larger crimes that then put you away until you're a hardened repeat felon. Your logic seems to acknowledge that one of the large drivers of crime is being "put away" in prison. In this case, the algorithm predicted a higher likelihood of committing future crimes for a kid than for someone who had already served 5 years in prison for armed robbery. Even worse is that the algorithm was being used to determine the severity of their respective punishments. So, the act of predicting a higher likelihood of future crime for the kid becomes a self-fulfilling prophecy, giving her a harsher sentence, which in turn is more likely to drive her toward future crime. This is systemic bias in action.