This would work to give some evidence for the accuracy levels of the automated parts of the system, at least for true positives vs false negatives, but doesnt t

This would work to give some evidence for the accuracy levels of the automated parts of the system, at least for true positives vs false negatives, but doesnt test for how the system generates false positives. A bigger problem is that this doesnt approach any of the process problems with the human moderated parts of the system. For that you would need cooperation from the law enforcement customer. The customer would need to make a request like "are you sure you didnt find any gun shots around this time and place?" and see how the company responds.