3 ms·
This reminds of of a 2019 paper on "moral crumple zones"[1] which talks about how the human component of automated systems are increasingly there to act as the
by aeturnum 6y ago
This reminds of of a 2019 paper on "moral crumple zones"[1] which talks about how the human component of automated systems are increasingly there to act as the focus for moral failures. Did your giant automated system do something bad? Blame the one human who was assigned to somehow stop that from happening, no matter how impossible that might be.
[1] https://estsjournal.org/index.php/ests/article/view/260 https://estsjournal.org/index.php/ests/article/view/260
- aftbit 6y agoAlso see Normal Accidents[1] which discusses "human error" as a PR cover for systems that are simply too complicated for unaided humans to monitor and understand. 1: https://www.goodreads.com/book/show/192408.Normal_Accidents https://www.goodreads.com/book/show/192408.Normal_Accidents
- aeturnum 6y agoNormal Accidents is a real classic of the genre of disaster studies and points out some very useful realities for tightly coupled systems. Engineers building highly complex systems would do well to read the book and take its lessons to heart.
- tt433 6y agoThe film "Brazil" was mentioned on another comment on another story a few days ago that touched on this theme, very good movie.
- SilasX 6y agoInteresting, that’s a great concept. But I’m not sure of the applicability to this case. I’d expect most people to feel icky contacting a lead on this basis, and so that feels like it’s well within the kind of thing a low level employee should throw a red flag at.
- aeturnum 6y agoThe aspect that reminded me of the paper (and the concept) was how the low level employee can really only screw up. If they do well, then it's a credit to their boss, but if they do something wrong it's on them and they'll be fired.