7 ms·

From the article: "What this means is that, we calibrate classifiers parameters such that it has the same acceptance ratio for all subgroups of sensitive featur

From the article: "What this means is that, we calibrate classifiers parameters such that it has the same acceptance ratio for all subgroups of sensitive features, e.g. race, sex, etc." But statistics such as crime rates and default rates are NOT the same across race and sex. Is she saying that models should be fiddled with to make the same average prediction by race, sex, and other demographic variables?

xyzzyz 9y ago
Indeed. What's the point of creating predictive models, if we know a priori what these models are supposed to predict?
wyager 9y ago
This seems to be the (extremely unreasonable) standard held by parties who are either statistically ignorant or statistically malicious. A vastly more reasonable standard, but easy to confuse without stats background, is e.g. http://ttic.uchicago.edu/~nati/Publications/HardtPriceSrebro2016.pdf http://ttic.uchicago.edu/~nati/Publications/HardtPriceSrebro... Attempting to censor nodes in a classifier’s Bayesian network will strictly result in some combination of A) the classifier routing around the damage and B) the classifier producing worse results. Either way, we’ve accomplished nothing. The linked paper, on the other hand, still allows the classifier to be accurate without giving incorrect results across protected categories.
- eindiran 9y ago
  The paper was beyond my level of stats-comprehension, so I couldn't figure out the answer to this question while looking it over. How does their methodology get around the 'redundant encodings' problem?
  - mathperson 9y ago
    I am going to assume by 'redundant encoding' you mean a model that takes some non-racial feature like- living in an urban area- and uses that to predict something that is very different across races- say whether or not your loan is approved. "Definition 2.1 (Equalized odds). We say that a predictor Yhat satisfies equalized odds with respect to protected attribute A and outcome Y , if Yhat and A are independent conditional on Y ." This is from page 3. Yhat is the model trained on A (protected class) T (training outcome). Do you see how if this definition holds there can be absolutely no redundant encoding?
    - eindiran 9y ago
      Yes that helped quite a bit. Looking over that section, I thought this summarized it quite well: "For the outcome y = 1, the constraint requires that Yhat has equal true positive rates across the two demographics A = 0 and A = 1. For y = 0, the constraint equalizes false positive rates."
      - mathperson 9y ago
        Fantastic! I thought more about your question (at least what I thought it was) and it wouldn't necessarily prevent redundant encoding but it would sort of restrict how 'damaging' such an encoding could be (if that makes sense). This whole field is very new but very exciting and very troubling. Like- what is fairness really? Its an intersection of philosophy/ethics and very UN-intuitive mathematics..there are many open questions
zengid 9y ago
I feel like the implications of what she was saying are a little vague, so I'm not sure what she meant here. I think the main point is that this is still early days for defining a mathematical measure for 'fairness'. Wouldn't it be better to have a model that could tell whether an individual will be more susceptible to defaulting or crime based on their context, and not strictly because of their gender or race? I think that is the point here, because otherwise it would be a biased prediction.
- xyzzyz 9y ago
  I don't understand what you are proposing here. The way things are done right now is we define some specific features which can include things like age, height, eye color, skin color, default history, average time a person wakes up, whether they prefer pancakes or waffles, income, and pretty much anything else you come up with. Then, we obtain a training set containing instances of the specific values of the features paired with the output variable we want to predict. Then, we pick a model and train it using this data. After training, we can test the model on some new data and measure its accuracy, that is, how often it is wrong, what is the direction of the error (i.e. bias), if it's wrong, by how much (variance) etc. Some of the variables might turn out to be more predictive, and some variables probably quite a bit less. Some variables might turn out to be predictive even though one would guess they shouldn't. For example, people with brown eyes are more likely to be victims of crime than people with blue eyes. It might seem like a mystery, until you realize that black people overwhelmingly have brown eyes, and they are more likely to be a victim of crime than white people. Here, removing race as a variable will only make you and your model more confused. It is important to realize that removing predictive variable will not necessarily make your model less "biased" -- in fact, it is likely to make your model more often wrong than before, and therefore less useful. The difficulty here stems from the fact that often laws require you to make "non-discriminatory" decisions, and if you attempt to do that, for example by introducing counteracting bias to the bias you get from reality, your model will be less predictive and so less useful. Fairness in general is very complicated issue. Suppose our model tells us that blue agents are much more likely to default on loan, and this is in fact the underlying reality. The creditor, to maximize his profit, should therefore charge blue agents higher interest than red agents. If you are blue agents, this is obviously unfair for you -- it's not your fault some other agents who share similar characteristic default more often. However, if we have laws forbidding discrimination based on agent's color, the creditor must then charge both colors of agents equally, and since one is more likely to default, this interest rate would be higher than what red agents would pay if the anti-discriminatory laws weren't put in place. Isn't it unfair towards red agents then? After all, it's not your fault that some other agents who share some characteristic default more often...
Tarq0n 9y ago
Isn't that the logical consequence of having laws that forbid disparate impact on protected classes? One could also argue that a person's attributes that are not the result of choice should not be used to assess them, because they reflect historical and environmental influences rather than personal ones.
- grandmczeb 9y ago
  It may not be “fair” in some cosmic sense, but traits that you don’t have control over are important for lots of things you want to assess. For example, should modeling agencies not take into account physical appearance?
- maldeh 9y ago
  Additionally, this extends to the usage of latent or inferred variables that behave as a proxy to a protected class - so neural networks and other discriminative approaches outside of a distinctly diagnosable decision tree, could be in a rather grey area legally if used for credit scores, etc. It's possible that the upshot of a legal challenge could be a crippling cap on the accuracy of models as they explicitly avoid any information derived from any protected factors.
ABCLAW 9y ago
No, she's saying that if the input data is itself skewed, then one method of fixing the model you generate is to apply corrections to the calibrating parameters. Imagine we have a 50/50 population of purple and orange people. We know that purple offenders are caught twice as often for a certain crime, and 10 purple offenders are caught and 5 orange offenders are caught. If we run our data and our model predicts that we're getting substantially more purple offenders, then we know our model is wrong - it isn't taking into account the skew due to differential enforcement. How do you fix this? She gives 3 methods. 1) Change the params to normalize, 2) resample and get better data, 3) use causal reasoning and other data to plug in the holes.
- PeterStuer 9y ago
  But in reality not all feature dimensions are orthogonal. You can't make the same 'adjustment' for any potential trait bias. Correcting for the purple/orange thing might skew the data on square and round heads, and correcting for that might skew it on pointy and rounded ears and so on and on. The intention is laudable, but many a road to hell is paved with good intentions.
  - visarga 9y ago
    So they got to balance data on all sub-combinations of features "orange round head with pointy ears" vs "purple square head rounded ears", etc - which are exponential in the number of features. The curse of dimensionality.
- pacala 9y ago
  How do you quantify "skew due to differential enforcement"? The proposal is to posit some value for "skew due to differential enforcement", then massage the results to match the baseless bias.
- bobcostas55 9y ago
  The data[0] shows that the race distribution of the assailant in crime reports is about the same as the race distribution of arrestees. There's no support for the differential enforcement hypothesis. [0] https://www.ncjrs.gov/App/Publications/abstract.aspx?ID=256035 https://www.ncjrs.gov/App/Publications/abstract.aspx?ID=2560...
sytelus 9y ago
A person is not a statistics and that's exactly the crux of this issue. It is a human who ultimately decides what features classifier has access to. If you decide to include race, nationality and sex then classifier would use that and may predict that person of race X has Y probability of committing a crime. So now when you go in the field and run your classifier on all people of race X, classifier will say ALL of them are very likely criminal even if a person is a professor at Harvard or has won a nobel peace price or charismatic leader or a philanthropist or just small restaurant owner. The first issue is that humans are about the most complex objects that we know in universe and trying to make predictions on their behaviour and long term fate would require complex and very detailed data on each of them - if at all its possible. Trying to predict behavior just based on age, race, sex is mind bogglingly dumb, unjust and irresponsible. Trying to predict criminal behaviour itself is unethical because it may pre-disposes large number of people as future culprit without any evidence even if they are perfectly normal people trying to care for their families and peacefully live their lives. In my view, any classifier built for predicting human behaviour should not use attributes of a person that they have no control over whatsoever, such as race, age and sex. Doing this usually is very tempting because your classifier might start giving better accuracy on test set but one always need to think about the cost of false positives. Each false positive is ruining an innocent hopeful life without any wrongdoing.
- yorwba 9y ago
  If the classifier is being used for medical diagnosis, race, age and sex may be important factors. Ignoring them would amount to systematically ruining innocent hopeful life without any wrongdoing. The problem usually isn't injustice caused by model errors, because humans make unjust errors all the time; using a model that's better than humans is still a win even if the model isn't perfect. Likewise for medical diagnosis, if a model is better than the average doctor, it should replace them. The actual problem is that humans mistake the model prediction for something it is not. E.g. if you train a model on the data of people accused of a crime to predict whether they will be convicted, you're modeling P(convicted | accused). Even if you train this model to perfection (zero test errors), you would at best reproduce the existing legal system; which is a win if the model is cheaper. But people tend to mistake the model output for P(should be convicted | accused), which assumes that the legal system is perfect. Or worse, they think of it as P(should be convicted), without accusation. The solution to your "go in the field and run your classifier on everyone" thought experiment is not to hide race, sex and age from the model. If the model had no access to any features at all, it would just predict the baseline conviction rate, much higher than the baseline crime rate. Using the model output as evidence of a crime would be just as silly, because that's not what the model was trained to do. The solution is to not even try to predict something for which you don't have any data. Modeling can never make a process more accurate, it can only make an accurate process faster and cheaper (and slightly less accurate). Sometimes the accurate process is to wait for a few years and see what happens, e.g. for cancer mortality or recidivism rates. Sometimes the accurate process is to ask a lot of experts. Once you have a model for the accurate process that is better than the cheaper process you'd otherwise use, then you can replace it by the model. Instead of a single human judge or a jury, have a model trained on the decisions of multiple judges and juries deliberating much longer than usual. Instead of a single doctor, have a model that has seen the medical histories of thousands of cancer patients and knows exactly which treatments worked.

xyzzyz 9y ago

Indeed. What's the point of creating predictive models, if we know a priori what these models are supposed to predict?

wyager 9y ago

This seems to be the (extremely unreasonable) standard held by parties who are either statistically ignorant or statistically malicious. A vastly more reasonable standard, but easy to confuse without stats background, is e.g. http://ttic.uchicago.edu/~nati/Publications/HardtPriceSrebro2016.pdf http://ttic.uchicago.edu/~nati/Publications/HardtPriceSrebro... Attempting to censor nodes in a classifier’s Bayesian network will strictly result in some combination of A) the classifier routing around the damage and B) the classifier producing worse results. Either way, we’ve accomplished nothing. The linked paper, on the other hand, still allows the classifier to be accurate without giving incorrect results across protected categories.

eindiran 9y ago
The paper was beyond my level of stats-comprehension, so I couldn't figure out the answer to this question while looking it over. How does their methodology get around the 'redundant encodings' problem?
- mathperson 9y ago
  I am going to assume by 'redundant encoding' you mean a model that takes some non-racial feature like- living in an urban area- and uses that to predict something that is very different across races- say whether or not your loan is approved. "Definition 2.1 (Equalized odds). We say that a predictor Yhat satisfies equalized odds with respect to protected attribute A and outcome Y , if Yhat and A are independent conditional on Y ." This is from page 3. Yhat is the model trained on A (protected class) T (training outcome). Do you see how if this definition holds there can be absolutely no redundant encoding?
  - eindiran 9y ago
    Yes that helped quite a bit. Looking over that section, I thought this summarized it quite well: "For the outcome y = 1, the constraint requires that Yhat has equal true positive rates across the two demographics A = 0 and A = 1. For y = 0, the constraint equalizes false positive rates."
    - mathperson 9y ago
      Fantastic! I thought more about your question (at least what I thought it was) and it wouldn't necessarily prevent redundant encoding but it would sort of restrict how 'damaging' such an encoding could be (if that makes sense). This whole field is very new but very exciting and very troubling. Like- what is fairness really? Its an intersection of philosophy/ethics and very UN-intuitive mathematics..there are many open questions

zengid 9y ago

I feel like the implications of what she was saying are a little vague, so I'm not sure what she meant here. I think the main point is that this is still early days for defining a mathematical measure for 'fairness'. Wouldn't it be better to have a model that could tell whether an individual will be more susceptible to defaulting or crime based on their context, and not strictly because of their gender or race? I think that is the point here, because otherwise it would be a biased prediction.

xyzzyz 9y ago
I don't understand what you are proposing here. The way things are done right now is we define some specific features which can include things like age, height, eye color, skin color, default history, average time a person wakes up, whether they prefer pancakes or waffles, income, and pretty much anything else you come up with. Then, we obtain a training set containing instances of the specific values of the features paired with the output variable we want to predict. Then, we pick a model and train it using this data. After training, we can test the model on some new data and measure its accuracy, that is, how often it is wrong, what is the direction of the error (i.e. bias), if it's wrong, by how much (variance) etc. Some of the variables might turn out to be more predictive, and some variables probably quite a bit less. Some variables might turn out to be predictive even though one would guess they shouldn't. For example, people with brown eyes are more likely to be victims of crime than people with blue eyes. It might seem like a mystery, until you realize that black people overwhelmingly have brown eyes, and they are more likely to be a victim of crime than white people. Here, removing race as a variable will only make you and your model more confused. It is important to realize that removing predictive variable will not necessarily make your model less "biased" -- in fact, it is likely to make your model more often wrong than before, and therefore less useful. The difficulty here stems from the fact that often laws require you to make "non-discriminatory" decisions, and if you attempt to do that, for example by introducing counteracting bias to the bias you get from reality, your model will be less predictive and so less useful. Fairness in general is very complicated issue. Suppose our model tells us that blue agents are much more likely to default on loan, and this is in fact the underlying reality. The creditor, to maximize his profit, should therefore charge blue agents higher interest than red agents. If you are blue agents, this is obviously unfair for you -- it's not your fault some other agents who share similar characteristic default more often. However, if we have laws forbidding discrimination based on agent's color, the creditor must then charge both colors of agents equally, and since one is more likely to default, this interest rate would be higher than what red agents would pay if the anti-discriminatory laws weren't put in place. Isn't it unfair towards red agents then? After all, it's not your fault that some other agents who share some characteristic default more often...

Tarq0n 9y ago

Isn't that the logical consequence of having laws that forbid disparate impact on protected classes? One could also argue that a person's attributes that are not the result of choice should not be used to assess them, because they reflect historical and environmental influences rather than personal ones.

grandmczeb 9y ago
It may not be “fair” in some cosmic sense, but traits that you don’t have control over are important for lots of things you want to assess. For example, should modeling agencies not take into account physical appearance?
maldeh 9y ago
Additionally, this extends to the usage of latent or inferred variables that behave as a proxy to a protected class - so neural networks and other discriminative approaches outside of a distinctly diagnosable decision tree, could be in a rather grey area legally if used for credit scores, etc. It's possible that the upshot of a legal challenge could be a crippling cap on the accuracy of models as they explicitly avoid any information derived from any protected factors.

ABCLAW 9y ago

No, she's saying that if the input data is itself skewed, then one method of fixing the model you generate is to apply corrections to the calibrating parameters. Imagine we have a 50/50 population of purple and orange people. We know that purple offenders are caught twice as often for a certain crime, and 10 purple offenders are caught and 5 orange offenders are caught. If we run our data and our model predicts that we're getting substantially more purple offenders, then we know our model is wrong - it isn't taking into account the skew due to differential enforcement. How do you fix this? She gives 3 methods. 1) Change the params to normalize, 2) resample and get better data, 3) use causal reasoning and other data to plug in the holes.

PeterStuer 9y ago
But in reality not all feature dimensions are orthogonal. You can't make the same 'adjustment' for any potential trait bias. Correcting for the purple/orange thing might skew the data on square and round heads, and correcting for that might skew it on pointy and rounded ears and so on and on. The intention is laudable, but many a road to hell is paved with good intentions.
- visarga 9y ago
  So they got to balance data on all sub-combinations of features "orange round head with pointy ears" vs "purple square head rounded ears", etc - which are exponential in the number of features. The curse of dimensionality.
pacala 9y ago
How do you quantify "skew due to differential enforcement"? The proposal is to posit some value for "skew due to differential enforcement", then massage the results to match the baseless bias.
bobcostas55 9y ago
The data[0] shows that the race distribution of the assailant in crime reports is about the same as the race distribution of arrestees. There's no support for the differential enforcement hypothesis. [0] https://www.ncjrs.gov/App/Publications/abstract.aspx?ID=256035 https://www.ncjrs.gov/App/Publications/abstract.aspx?ID=2560...

sytelus 9y ago

A person is not a statistics and that's exactly the crux of this issue. It is a human who ultimately decides what features classifier has access to. If you decide to include race, nationality and sex then classifier would use that and may predict that person of race X has Y probability of committing a crime. So now when you go in the field and run your classifier on all people of race X, classifier will say ALL of them are very likely criminal even if a person is a professor at Harvard or has won a nobel peace price or charismatic leader or a philanthropist or just small restaurant owner. The first issue is that humans are about the most complex objects that we know in universe and trying to make predictions on their behaviour and long term fate would require complex and very detailed data on each of them - if at all its possible. Trying to predict behavior just based on age, race, sex is mind bogglingly dumb, unjust and irresponsible. Trying to predict criminal behaviour itself is unethical because it may pre-disposes large number of people as future culprit without any evidence even if they are perfectly normal people trying to care for their families and peacefully live their lives. In my view, any classifier built for predicting human behaviour should not use attributes of a person that they have no control over whatsoever, such as race, age and sex. Doing this usually is very tempting because your classifier might start giving better accuracy on test set but one always need to think about the cost of false positives. Each false positive is ruining an innocent hopeful life without any wrongdoing.

yorwba 9y ago
If the classifier is being used for medical diagnosis, race, age and sex may be important factors. Ignoring them would amount to systematically ruining innocent hopeful life without any wrongdoing. The problem usually isn't injustice caused by model errors, because humans make unjust errors all the time; using a model that's better than humans is still a win even if the model isn't perfect. Likewise for medical diagnosis, if a model is better than the average doctor, it should replace them. The actual problem is that humans mistake the model prediction for something it is not. E.g. if you train a model on the data of people accused of a crime to predict whether they will be convicted, you're modeling P(convicted | accused). Even if you train this model to perfection (zero test errors), you would at best reproduce the existing legal system; which is a win if the model is cheaper. But people tend to mistake the model output for P(should be convicted | accused), which assumes that the legal system is perfect. Or worse, they think of it as P(should be convicted), without accusation. The solution to your "go in the field and run your classifier on everyone" thought experiment is not to hide race, sex and age from the model. If the model had no access to any features at all, it would just predict the baseline conviction rate, much higher than the baseline crime rate. Using the model output as evidence of a crime would be just as silly, because that's not what the model was trained to do. The solution is to not even try to predict something for which you don't have any data. Modeling can never make a process more accurate, it can only make an accurate process faster and cheaper (and slightly less accurate). Sometimes the accurate process is to wait for a few years and see what happens, e.g. for cancer mortality or recidivism rates. Sometimes the accurate process is to ask a lot of experts. Once you have a model for the accurate process that is better than the cheaper process you'd otherwise use, then you can replace it by the model. Instead of a single human judge or a jury, have a model trained on the decisions of multiple judges and juries deliberating much longer than usual. Instead of a single doctor, have a model that has seen the medical histories of thousands of cancer patients and knows exactly which treatments worked.