Say there's a car manufacturer that makes a car. It's a reasonably safe car, but sometimes people speed in it. Without really telling anyone, they install a we

Say there's a car manufacturer that makes a car. It's a reasonably safe car, but sometimes people speed in it. Without really telling anyone, they install a webcam and point it at the driver to see whether they might speed on a certain day, based on facial expression and skin color. Why those? Just a weird hunch! They find that the person actually did speed. (Turns out, they didn't really tell anyone that all your driving data was uploaded to their servers, too!) The problem is two fold. First, they shouldn't have been collecting this data on their users in the first place. The manufactures job is to build a car, not ensure that the car is being used responsibly. This is simply an impossible task to charge anyone with, and the implications are frightening. Second, I'm not a believer in using algorithms to predict behavior. Too much black box "mangle the numbers until it works". You could make an argument that there's a correlation between skin color and speeding. You might even be able to make pretty charts, but I think this is simply bad science. You make it sound like this model would actually continue to work at 75% perpetually. I think that it's far more likely it worked this one test, and performance would trend sharply downwards. However it makes this decision, I think it's about as absurd and wrong as the crazy example above. On the other side of the coin, people are putting too much trust in magic algorithms they don't properly understand, and are far too complacent about egregious invasion of privacy.