One issue I haven't heard much about is what misconceptions people have about AI/ML in general. This kind of thing only makes it worse. There's a lot of ways a

One issue I haven't heard much about is what misconceptions people have about AI/ML in general. This kind of thing only makes it worse. There's a lot of ways an ML system can go wrong, and it won't be obvious to everyone who uses it. Dataset badly sampled? Cop ain't gonna know that. In fact, from his POV he's using fancy new equipment that the department has paid some highly qualified consultants for. What I worry about is we have actual data scientists like the guys at top unis who teach and know how the sausage is made. And then we have some consultant people who kinda know how to shove some data into a model and get a result out. I'm not saying they're totally incompetent, but their incentives are misaligned. They want the data to mean some things that it can't mean, in terms of epistemology, or in terms of level of confidence. You could very easily bake a false correlation <> causation into this kind of thing, and use it to arrest whoever you like. This is especially true if the guy you're selling it to also has wrong incentives. And the guy at the bottom using this model ends up thinking that he has a magic box that tells him who is gonna be a criminal in the future. When he arrests someone, it's a very deep explanation why it isn't the way he thought, something I could see the judge not getting. That is not to say such a system is never going to work anywhere, but if you're operating a predictive system you need to know a bunch of stuff about it to do so responsibly.