> If the software is accurate at picking choices that match the set goals > auditing of the results vs the desired outcomes The core problem is this assumptio

> If the software is accurate at picking choices that match the set goals > auditing of the results vs the desired outcomes The core problem is this assumption that the goal is to prioritize accuracy. When making "life-changing decisions ... in areas from job applications to immigration", many different goals need to be balanced. Who gets to define what "desired outcome" means? We already have a hard time simply defining those goals. How would you even begin to define a cost function for e.g. "immigration" decisions when simply bringing up the topic is one of the easier ways to start a heated political argument? > then it would seem that: the software is doing its job and is 'blameless' The software is doing exactly what it was told to do: find a particular signal in some set of data (without explicit race. The problem is we spent several hundred years systemically embedding race into most aspects of society, which becomes a minefields of problematic local minima/maxima traps for your ML algorithm. Let's say we have an good ML algorithm that provides some sort of evaluation about job applicants. Nothing related to race is included. The algorithm is used by a company in Chicago[1][3] or D.C.[2][3] hoping to hire the "best" people. Segregated populations means different schools/etc. Even if you didn't give the algorithm race data, most ML is practically guaranteed to find one of the many ways to infer race. Unintentionally, the ML algorithm ends up doing something similar to redlining[4], because that's what is in the data: the systemic effects of historical racism that still reverberate through society. [1] https://www.washingtonpost.com/graphics/2018/national/segregation-us-cities/img/Chicago_map_2016-600.jpg https://www.washingtonpost.com/graphics/2018/national/segreg... [2] https://www.washingtonpost.com/graphics/2018/national/segregation-us-cities/img/DC_map_2016-600.jpg https://www.washingtonpost.com/graphics/2018/national/segreg... [3] red-dots="White", blue-dots="Black", yellow-dots="Hispanic", etc. See the full article for many more examples: https://www.washingtonpost.com/graphics/2018/national/segregation-us-cities/ https://www.washingtonpost.com/graphics/2018/national/segreg... [4] https://en.wikipedia.org/wiki/Redlining https://en.wikipedia.org/wiki/Redlining