I'd imagine the problems are far more fundamental -- do they work at all given the typically poorly constructed training sets. And also what is their FP/FN rate

I'd imagine the problems are far more fundamental -- do they work at all given the typically poorly constructed training sets. And also what is their FP/FN rates for subsets of the population -- do they work as well/poorly for everyone? The problem I see with many of these systems are -- they are trained on one population and used for inference on a very different distribution (minorities.) OR their accuracy is only high on the overall population, not universally accurate across all subsets. When those in charge, those with political power, etc (usually non-minorities) "see" the performance of the system, they experience it in their own context where it works. They don't feel the pain from the experience of the sub-populations where false positives/negatives are larger.