Seeing it applied to the chance of violating probation or reoffending after a minor crime alarms me for exactly this reason - lots of people commit minor probat

Seeing it applied to the chance of violating probation or reoffending after a minor crime alarms me for exactly this reason - lots of people commit minor probation violations (e.g. associating with criminals) or crimes (e.g. marijuana possession) and are never caught or charged. Even if this is a case of "the data is biased, not the algorithm", it's important to remember that the data is "who was caught doing this" rather than "who did this". Concluding that people who live in stop-and-frisk areas are more likely to get drug convictions isn't actually wrong, but it's still unhelpful.