As the real-world consequences of various types of software grow, society's expectations for them to be carefully built and well understood will grow. [0] Bowi

As the real-world consequences of various types of software grow, society's expectations for them to be carefully built and well understood will grow. [0] Bowing down to the techno-gods and expecting them to make better decisions than we do has serious ramifications when the decision making is literally inscrutable. Right now the conversation seems to be hovering at the level of "shut up you luddites, and accept it." That doesn't mean the system should be trusted. In fact, it shows the opposite by illustrating those who would use the systems have nothing but contempt for those they would use the system on. A simple example of how an inscrutable system could be considered benign is making it well-constrained and doing a failure modes analysis. But the question becomes: is the analysis preformed? thorough? and are the results acceptable? For things like self-driving cars: of course. It'd have to be to get on the road. When the failure modes aren't both blindingly obvious and unquestionably unacceptable to the consumer? "lulz, move fast and brake things!" I fear the only way to improve the systems thinking around machine learning at fly-by-the-seat-of-your-pants companies are things like the EU regulation forcing the issue. That's a bit depressing, isn't it? Hopefully the level of discourse can be improved, even though the providers of such technology are incentivized to paint resistance as simpletons that just don't get it. [0] Bruce Schneier explains better than I can, but with focus on security https://www.youtube.com/watch?v=KWD2v0SJAM8&feature=youtu.be&t=1727 https://www.youtube.com/watch?v=KWD2v0SJAM8&feature=youtu.be... through 30:30ish