3 ms·

The author of this article is Virginia Eubanks [1], who has studied and written on this topic extensively, including a new book whose short title is 'Automating

The author of this article is Virginia Eubanks [1], who has studied and written on this topic extensively, including a new book whose short title is 'Automating Inequality'. Although I can't be sure, I think it's reasonable to assume that the content in this article is along the lines expressed in this CityLab interview [2], and other media coverage the author highlights [3]. Eubanks presents the impact that rigid, opaque systems and processes can have on individuals, and does so by focusing on troubling policy implications, but also by highlighting the suffering experienced by named individuals whose unique personal circumstances are examined. This is not only an effective writing device, but also demonstrates the empathy gap of software, and how an impersonal process -- whether intentionally or not -- serves as a tool for bureaucrats and low-level decision-makers to offload difficult negative decisions to an amorphous entity like 'the process'. I feel that this is a key, if not always overt motivation of most bureaucracies, and has universal applicability. There's also the notion that when ML is fed with raw data, it picks up correlations present in the data regardless of whether their presence is due to a lurking variable. Later, making use of such a model for decision-making perpetuates the bias, and a haphazardly chained sequence of such processes increases the likelihood that a cohort's observed outcomes will self-fulfill in the future. Conversation on this matter in recent years has tended towards the social justice implications more so than the lack of intellectual rigor that allows this to be possible -- the former is a compelling point, but it's unfortunate that there's not enough coverage of about the harm of giving complex tools to those who don't understand them. [1] https://virginia-eubanks.com/ https://virginia-eubanks.com/ [2] https://www.citylab.com/equity/2018/02/the-rise-of-digital-poorhouses/552161/ https://www.citylab.com/equity/2018/02/the-rise-of-digital-p... [3] https://virginia-eubanks.com/media/ https://virginia-eubanks.com/media/

Bartweiss 8y ago
This is a great summary, and that Citylab piece is well worth checking out. I'm sometimes skeptical of human-narrative analyses, because they make it easy to launch equally-vivid attacks on a plan that fails 10 people and a plan that fails 10,000. But with rigor behind them, there's a lot to be said for them as existence proofs - especially when we're talking about the margins of a plan instead of its core. Knowing what the edge cases look like matters, and for that it's hard to beat talking to individuals with problems that don't fit on a form. (As an aside, I think software often creates not only an empathy gap but a mechanical one. An analog system can often be forced to accommodate oddities like self-employment and unconventional address formats, but "the computer won't let me enter that address" is a brick wall. I've never found an investigation of this, but I suspect it's a major drawback of digitization.) Your final point about ML is a vital one, and one I'm endlessly frustrated by. We're increasingly aware of issues like racial bias in parole-predictions systems, but there's alarmingly little focus on the fact that a system which does that is flawed in general, and might be making the same assumptions about e.g. low-income neighborhoods without anyone noticing. Unlike human bias, a flawed algorithm is equally happy to abuse any strange correlation it can find.

Bartweiss 8y ago

This is a great summary, and that Citylab piece is well worth checking out. I'm sometimes skeptical of human-narrative analyses, because they make it easy to launch equally-vivid attacks on a plan that fails 10 people and a plan that fails 10,000. But with rigor behind them, there's a lot to be said for them as existence proofs - especially when we're talking about the margins of a plan instead of its core. Knowing what the edge cases look like matters, and for that it's hard to beat talking to individuals with problems that don't fit on a form. (As an aside, I think software often creates not only an empathy gap but a mechanical one. An analog system can often be forced to accommodate oddities like self-employment and unconventional address formats, but "the computer won't let me enter that address" is a brick wall. I've never found an investigation of this, but I suspect it's a major drawback of digitization.) Your final point about ML is a vital one, and one I'm endlessly frustrated by. We're increasingly aware of issues like racial bias in parole-predictions systems, but there's alarmingly little focus on the fact that a system which does that is flawed in general, and might be making the same assumptions about e.g. low-income neighborhoods without anyone noticing. Unlike human bias, a flawed algorithm is equally happy to abuse any strange correlation it can find.