Yes, for one example, a system used to recommend sentencing based on how likely a criminal was to re-offend was found to have been heavily biased based on race,

Yes, for one example, a system used to recommend sentencing based on how likely a criminal was to re-offend was found to have been heavily biased based on race, despite not being given explicit information about race[1]. Edit: It’s unclear if this system is based on ML, but I think the point stands - if a system can be manually tuned to do this, the risk exists of this happening to a trained model as well. Given other cases of ML models learning an incorrect behavior (such as the system intended to detect skin cancer that ended up being a fancy ruler detector [2]). [1]: https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing https://www.propublica.org/article/machine-bias-risk-assessm... [2]: https://www.thedailybeast.com/why-doctors-arent-afraid-of-better-more-efficient-ai-diagnosing-cancer https://www.thedailybeast.com/why-doctors-arent-afraid-of-be...