The discussion in the paper describes COMPAS as "no more accurate or fair than the predictions of people with little to no criminal justice expertise who respon

The discussion in the paper describes COMPAS as "no more accurate or fair than the predictions of people with little to no criminal justice expertise who responded to an online survey", an assessment that doesn't seem unreasonable given the results presented. So, COMPAS is uninspiring because it provides no demonstrable benefits over even a crude analog of existing methods. Furthermore, it is not competitive against a two-feature linear regression model which only considers age and the number of previous convictions. As it happens, however, I agree with you - people do have unrealistic expectations of what can be achieved with AI. Replacing humans is, in many domains, a more difficult prospect than might be imagined, because so much of what we do is implicit. Suggesting that a particular problem is "solved" because a system achieves non-significant parity on a handful of measures is short-sighted to say the least.