From The flaws of policies requiring human oversight of government algorithms https://www.sciencedirect.com/science/article/pii/S0267364922000292 https://www.sc

From The flaws of policies requiring human oversight of government algorithms https://www.sciencedirect.com/science/article/pii/S0267364922000292 https://www.sciencedirect.com/science/article/pii/S026736492... : human oversight policies provide a false sense of security in adopting algorithms and enable vendors and agencies to shirk accountability for algorithmic harms. And the better The Algorithm is, the less people check, and the more of its faults pass through. Having worked on systems that required human oversight, I've seen first-hand, and analyzed data on, how people click the "LGTM" button without reflecting. We even noticed how when we improved our predictor with expert input, so that it performed better according to experts who worked with it, it would perform worse when we benchmarked on the previously "manually checked" data set. Analysis showed that the manual checks had let through so many of the algorithm's faults, that when the algorithm improved, it got a worse score. See also https://predictive-optimization.cs.princeton.edu/ https://predictive-optimization.cs.princeton.edu/ : We present seven flaws of predictive optimization. Our aim is to outline a set of objections inherent to predictive optimization that cannot be easily fixed using a design or engineering change. Taken together, these critical flaws undermine the legitimacy of applications of predictive optimization. 1. Good predictions may not lead to good decisions 2. It's hard to measure what we truly care about 3. The training data rarely matches the deployment setting 4. Social outcomes aren’t accurately predictable, with or without machine learning 5. Disparate performance between groups can’t be fixed by algorithmic interventions 6. Providing adequate contestability undercuts putative efficiency benefits 7. Predictive optimization doesn't account for strategic behavior