Computers "predict recidivism" and will tell you you're not allowed to be paroled because "the algorithm says" you're going to reoffend. Computers will tell yo

Computers "predict recidivism" and will tell you you're not allowed to be paroled because "the algorithm says" you're going to reoffend. Computers will tell you you can't get a loan because the algorithm says you won't repay. Or you can't get this job, or that opportunity, or should be spied on, or whatever. These may be due to biases in the data (imagine a neural net confusing being black with being poor based on historical data, etc). Now, I am pretty convinced that this stuff is going to screw us all over -- if you thought human bureaucracy was bad, wait until it's all in the computer and no one has permission or ability to change it. It seems to me that we may have a rare opportunity here: I'm not sure if this wave of tolerance and caring about minorities is permanent or not, but while people do care, and while computers are provably screwing all kinds of people over, we may be able to get enough people to care so that we can curb some of this stuff. i.e. perhaps don't allow neural nets to predict who is going to commit a crime or who shouldn't have a job, because they're too fallible and too easy to rig. For example, you can keep feeding it new datasets until it produces the biased result you want and now that's your model, and then you can blame "the algorithm" and "bad data" and not yourself. e.g. maybe you feed it old census data instead of recent data, or old crime data, etc. Of course, knee-jerk "ban the algorithms" isn't a solution, but starting to talk about and think about a computer bureaucracy where no one accessible to you has permission or ability to change the output of the computer, no matter how exceptional the case may be.