I was especially interested to read this after reading yesterday's NYTimes story about an opaque, for-profit algorithm that is being used to help judges sentenc

I was especially interested to read this after reading yesterday's NYTimes story about an opaque, for-profit algorithm that is being used to help judges sentence people: https://www.nytimes.com/2017/05/01/us/politics/sent-to-prison-by-a-software-programs-secret-algorithms.html https://www.nytimes.com/2017/05/01/us/politics/sent-to-priso... HN discussion: https://news.ycombinator.com/item?id=14238786 https://news.ycombinator.com/item?id=14238786 The algorithm discussed in the NYTimes story purports to be able to predict a criminal defendant's risk of violent reoffending, recidivism, and pretrial risk. It does so, apparently, by examining historical data, looking at the outcomes of other people who match the defendant's profile, and on the basis of that information providing a probability score. As I mentioned in my comment on that post, it is entirely inappropriate to use an algorithm that, at best, predicts only aggregate risk to impute individual risk, especially when that probability is being used to determine something as serious as a prison sentence. And as this TechCrunch piece points out, it's not just this specialized software that has potentially undesirable consequences, but the everyday algorithms used by Google, etc. One thing that would help is transparency. But we are very quickly getting to a point where lack of transparency is not merely a business decision, but a problem inherent in the technology we're using. A machine-learning model might not be easily interpretable. Its own creators might not be able to fully understand why it's making the decisions it's making. We may soon get to an uncomfortable place where certain technologies are effective, but nevertheless must be avoided because the cost of making them sufficiently tunable to address ethical and regulatory requirements becomes prohibitive.