4 ms·

A person is not a statistics and that's exactly the crux of this issue. It is a human who ultimately decides what features classifier has access to. If you deci

A person is not a statistics and that's exactly the crux of this issue. It is a human who ultimately decides what features classifier has access to. If you decide to include race, nationality and sex then classifier would use that and may predict that person of race X has Y probability of committing a crime. So now when you go in the field and run your classifier on all people of race X, classifier will say ALL of them are very likely criminal even if a person is a professor at Harvard or has won a nobel peace price or charismatic leader or a philanthropist or just small restaurant owner. The first issue is that humans are about the most complex objects that we know in universe and trying to make predictions on their behaviour and long term fate would require complex and very detailed data on each of them - if at all its possible. Trying to predict behavior just based on age, race, sex is mind bogglingly dumb, unjust and irresponsible. Trying to predict criminal behaviour itself is unethical because it may pre-disposes large number of people as future culprit without any evidence even if they are perfectly normal people trying to care for their families and peacefully live their lives. In my view, any classifier built for predicting human behaviour should not use attributes of a person that they have no control over whatsoever, such as race, age and sex. Doing this usually is very tempting because your classifier might start giving better accuracy on test set but one always need to think about the cost of false positives. Each false positive is ruining an innocent hopeful life without any wrongdoing.

yorwba 9y ago
If the classifier is being used for medical diagnosis, race, age and sex may be important factors. Ignoring them would amount to systematically ruining innocent hopeful life without any wrongdoing. The problem usually isn't injustice caused by model errors, because humans make unjust errors all the time; using a model that's better than humans is still a win even if the model isn't perfect. Likewise for medical diagnosis, if a model is better than the average doctor, it should replace them. The actual problem is that humans mistake the model prediction for something it is not. E.g. if you train a model on the data of people accused of a crime to predict whether they will be convicted, you're modeling P(convicted | accused). Even if you train this model to perfection (zero test errors), you would at best reproduce the existing legal system; which is a win if the model is cheaper. But people tend to mistake the model output for P(should be convicted | accused), which assumes that the legal system is perfect. Or worse, they think of it as P(should be convicted), without accusation. The solution to your "go in the field and run your classifier on everyone" thought experiment is not to hide race, sex and age from the model. If the model had no access to any features at all, it would just predict the baseline conviction rate, much higher than the baseline crime rate. Using the model output as evidence of a crime would be just as silly, because that's not what the model was trained to do. The solution is to not even try to predict something for which you don't have any data. Modeling can never make a process more accurate, it can only make an accurate process faster and cheaper (and slightly less accurate). Sometimes the accurate process is to wait for a few years and see what happens, e.g. for cancer mortality or recidivism rates. Sometimes the accurate process is to ask a lot of experts. Once you have a model for the accurate process that is better than the cheaper process you'd otherwise use, then you can replace it by the model. Instead of a single human judge or a jury, have a model trained on the decisions of multiple judges and juries deliberating much longer than usual. Instead of a single doctor, have a model that has seen the medical histories of thousands of cancer patients and knows exactly which treatments worked.
PeterStuer 9y ago
False positives in classification or prediction are in practice unavoidable with any probabilistic technique for any non-trivial task. Including certain features where they are forbidden by regulation is obviously not done. The problem arises that your techniques can potentially derive those features anyway from correlated proxies. Meta-physically you are using a probablisitc inference technique to derive from unknown-knowns in your data a given known-unknown predictor. You can try to normalize for known-knowns, blurring the biases you don't want to account, but you can't control for unknowns. In some cases it is easy to detect bias in the model when there was discrete data for the feature you try to control. E.g. if your data would have an explicit 'gender' field, then you can test how much the outcome of the prediction is not correlated to the specific value of that field. If the data does not have such a feature, but derives it implicitly though e.g. food purchase preferences, that is much harder to detect. You can try to control for that if for whatever reason you deem the model not in line with bias preferences, but due to the implicitness it will be very hard to isolate the impacts. I feel you will never get a model satisfactory to all, as in the end what is and what isn't accepted as 'neutral-data' is not a scientific but a political statement. There is nothing wrong with that. All our ethical choices are political. Just don't expect universal acceptance.

yorwba 9y ago

If the classifier is being used for medical diagnosis, race, age and sex may be important factors. Ignoring them would amount to systematically ruining innocent hopeful life without any wrongdoing. The problem usually isn't injustice caused by model errors, because humans make unjust errors all the time; using a model that's better than humans is still a win even if the model isn't perfect. Likewise for medical diagnosis, if a model is better than the average doctor, it should replace them. The actual problem is that humans mistake the model prediction for something it is not. E.g. if you train a model on the data of people accused of a crime to predict whether they will be convicted, you're modeling P(convicted | accused). Even if you train this model to perfection (zero test errors), you would at best reproduce the existing legal system; which is a win if the model is cheaper. But people tend to mistake the model output for P(should be convicted | accused), which assumes that the legal system is perfect. Or worse, they think of it as P(should be convicted), without accusation. The solution to your "go in the field and run your classifier on everyone" thought experiment is not to hide race, sex and age from the model. If the model had no access to any features at all, it would just predict the baseline conviction rate, much higher than the baseline crime rate. Using the model output as evidence of a crime would be just as silly, because that's not what the model was trained to do. The solution is to not even try to predict something for which you don't have any data. Modeling can never make a process more accurate, it can only make an accurate process faster and cheaper (and slightly less accurate). Sometimes the accurate process is to wait for a few years and see what happens, e.g. for cancer mortality or recidivism rates. Sometimes the accurate process is to ask a lot of experts. Once you have a model for the accurate process that is better than the cheaper process you'd otherwise use, then you can replace it by the model. Instead of a single human judge or a jury, have a model trained on the decisions of multiple judges and juries deliberating much longer than usual. Instead of a single doctor, have a model that has seen the medical histories of thousands of cancer patients and knows exactly which treatments worked.

PeterStuer 9y ago

False positives in classification or prediction are in practice unavoidable with any probabilistic technique for any non-trivial task. Including certain features where they are forbidden by regulation is obviously not done. The problem arises that your techniques can potentially derive those features anyway from correlated proxies. Meta-physically you are using a probablisitc inference technique to derive from unknown-knowns in your data a given known-unknown predictor. You can try to normalize for known-knowns, blurring the biases you don't want to account, but you can't control for unknowns. In some cases it is easy to detect bias in the model when there was discrete data for the feature you try to control. E.g. if your data would have an explicit 'gender' field, then you can test how much the outcome of the prediction is not correlated to the specific value of that field. If the data does not have such a feature, but derives it implicitly though e.g. food purchase preferences, that is much harder to detect. You can try to control for that if for whatever reason you deem the model not in line with bias preferences, but due to the implicitness it will be very hard to isolate the impacts. I feel you will never get a model satisfactory to all, as in the end what is and what isn't accepted as 'neutral-data' is not a scientific but a political statement. There is nothing wrong with that. All our ethical choices are political. Just don't expect universal acceptance.