6 ms·

Have we even figured out what bias we care about? Race, gender, age, etc. are some potential problems, but is that it? These sorts of problems are often formul

by mirker 5y ago

Have we even figured out what bias we care about? Race, gender, age, etc. are some potential problems, but is that it? These sorts of problems are often formulated theoretically (“suppose we want fairness with respect to variable Z”). It seems that half the battle is to figure out what to be fair with respect to. Often, the fairness variable in question isn’t even a feature, but is implicitly in the data (e.g., race in human photos). Therefore, the fairness space is potentially infinite. For example, maybe life insurance models are biased toward those predisposed to developing cancer. Maybe ads target those suffering depression. You can continue partitioning the space in this way forever, and, therefore, it seems that the algorithms are relatively straightforward if you could formalize the bias requirements. This is before you even consider the fairness variables interacting (e.g., age and gender and race), which requires potentially normalizing across exponentially growing feature combinations.

mulmen 5y ago
This is a philosophical gap that society in general needs to bridge. Data is truth. If we discover bias in data that tells us something about the world. I can't fix systemic racism with a SQL statement. But I can tell you where it happened. Society can decide what classes are protected and data can tell us when that happens. It is then up to the offending entity to change their behavior. In other words bias in data isn't unethical but bias in action can be.
- whatshisface 5y ago
  I feel that talking about SQL in a way misses this issue - when data analysis is done by human beings, human virtues can restrain the worst excesses of self-interest. In contrast, a model trained blindly to maximize some kind of metric can commit essentially any sin in pursuit of that goal.
  - mulmen 5y ago
    A model is just another form of analysis. A SQL statement is already built on levels of abstraction. An “AI” model is only one more level. At the bottom it’s all ones and zeros. But no matter how high you make the tower there’s always a human at the top.
  - mirker 5y ago
    That’s not really true. Say you want to know if you should approve someone for insurance. SELECT AVG(profit) WHERE feature == $user_feature; Say “feature” is education level and then suppose education level is correlated with race or gender. Then you implicitly are writing a query to filter by race or gender. Humans often don’t consider these implicit correlations. An AI would do the same. When it’s human face detection, then sure, humans would find the mistakes quickly. But SQL data is very easy to implicitly bias, and like I said, you can slice the pie millions of ways for various reasons. On the other hand, if we had a simple definition, then the database could raise an exception when the condition for the query failed.
    - mattmcknight 5y ago
      > Say “feature” is education level and then suppose education level is correlated with race or gender. Then you implicitly are writing a query to filter by race or gender. You aren't though. Because the predictive attribute is correlated with the irrelevant attributes, not determined by them. So you will find people with multiple values of your irrelevant attributes (race, gender) for the positive value of your predictive attribute (education). Thus, it is not implicitly filtering for the irrelevant attributes. It if is truly education that is predictive, using only that would be avoiding biasing your decision on the irrelevant attributes. Perhaps the risk comes when it is not really "education" that is causative, but some hidden personality trait, say conscientiousness, for which "education" is a possible proxy. In this case, trying to correct for some measure of race/gender access to education is still not going to work. For example, it is generally reported that females score higher on both measures of conscientiousness and educational achievement. Would you lower the value of your "education" score for females and raise them for males on this basis to correct for the bias in educational achievement against those with low conscientiousness? It would only seem to make the predictive value of the feature worse.
      - mirker 5y ago
        It depends on the definition of fairness, which is what I’m trying to argue for. Practically speaking, heavy correlation and casual relationships are similarly problematic. If you correlate enough of the variables, you can limit your pool to a very particular group of people. And if your group of people is 99% of race/gender/age, then is that practically any different from doing a direct filter? The usual definitions of fairness actually have the strong constraint that the response (Y=f(X)) is independent of the fairness variables (f(X|Z) is the same for all fairness values of Z). So, concretely, if Z is gender, then you have the same insurance outcome regardless of gender. In this definition, the setting I described is unfair, since the correlation on education will bias the model to predict in a way that is correlated with race. The only practical way to satisfy such a strong definition is to do a correction on the data, which is a pretty straightforward optimization problem if you formalize the constraints. And yeah, usually you lose predictive power in doing so. For the record, the strong definition makes sense often times (to me). You wouldn’t want health insurance using your DNA as a predictor, likely, even if genetics are very predictive in that domain.
- Spivak 5y ago
  I really don’t think anyone is really going to disagree with you that data is truth and actions are what matters but you run into two problems. * If you draw any conclusion from the data it will reflect the biases in the data. * If you take any action from the data it will reflect the biases in the data. So you’re going to be “correcting” at some point if you want to avoid that bias which is what really matters.
  - mirker 5y ago
    Right. And once you start doing corrections you are basically making another ML model to turn “what happened” into “what should have happened”.
  - cratermoon 5y ago
    I disagree. Data is data. Before it's even collected and organized, bias and error creep in. Is the Library of Congress card catalog a collection of data that is "truth" about books? Let's take a look at Library of Congress Classification Outline: Class D: https://www.loc.gov/catdir/cpso/lcco/lcco_d.pdf https://www.loc.gov/catdir/cpso/lcco/lcco_d.pdf D – History (General) DA – History of Great Britain – History of Ireland – History of the British Isles DAW – History of Central Europe OK so far, I guess? DB – History of Austria – Liechtenstein – Hungary – Czechoslovakia DC – History of France – Andorra – Monaco Wait, what? I left out a bunch here, but various countries get their own class, or share a class with one or two other countries. But I guess not "central Europe? 10 DH – History of Low Countries – Benelux Countries Huh? 17 DR – History of the Balkan Peninsula Really drilling down here 18 DS – History of Asia Uhm, Asia? ALL of Asia? 19 DT – History of Africa 20 DU – History of Oceania (South Seas) Seems fair. Not. 21 DX – History of Gypsies OK now they're just trolling us.
    - mulmen 5y ago
      I don't follow. This is how the Library of Congress catalogs books. That is the truth in the data. I don't see the problem with having a category for books on Central Europe in general and another section for individual countries that happen to be in Central Europe. Some books cover general topics and others specialize. Shouldn't we expect datasets that describe the content of books to reflect this?
- pjmorris 5y ago
  IMO, 'Data is truth' is not true. The data the machine sees was chosen in a way biased, at least, by the intent of the data collectors and by the capability of the sensors used to collect the data, both of which are shadows of reality. The Y2K problem is an illustration.

by mirker 5y ago

mulmen 5y ago

This is a philosophical gap that society in general needs to bridge. Data is truth. If we discover bias in data that tells us something about the world. I can't fix systemic racism with a SQL statement. But I can tell you where it happened. Society can decide what classes are protected and data can tell us when that happens. It is then up to the offending entity to change their behavior. In other words bias in data isn't unethical but bias in action can be.

whatshisface 5y ago
I feel that talking about SQL in a way misses this issue - when data analysis is done by human beings, human virtues can restrain the worst excesses of self-interest. In contrast, a model trained blindly to maximize some kind of metric can commit essentially any sin in pursuit of that goal.
- mulmen 5y ago
  A model is just another form of analysis. A SQL statement is already built on levels of abstraction. An “AI” model is only one more level. At the bottom it’s all ones and zeros. But no matter how high you make the tower there’s always a human at the top.
- mirker 5y ago
  That’s not really true. Say you want to know if you should approve someone for insurance. SELECT AVG(profit) WHERE feature == $user_feature; Say “feature” is education level and then suppose education level is correlated with race or gender. Then you implicitly are writing a query to filter by race or gender. Humans often don’t consider these implicit correlations. An AI would do the same. When it’s human face detection, then sure, humans would find the mistakes quickly. But SQL data is very easy to implicitly bias, and like I said, you can slice the pie millions of ways for various reasons. On the other hand, if we had a simple definition, then the database could raise an exception when the condition for the query failed.
  - mattmcknight 5y ago
    > Say “feature” is education level and then suppose education level is correlated with race or gender. Then you implicitly are writing a query to filter by race or gender. You aren't though. Because the predictive attribute is correlated with the irrelevant attributes, not determined by them. So you will find people with multiple values of your irrelevant attributes (race, gender) for the positive value of your predictive attribute (education). Thus, it is not implicitly filtering for the irrelevant attributes. It if is truly education that is predictive, using only that would be avoiding biasing your decision on the irrelevant attributes. Perhaps the risk comes when it is not really "education" that is causative, but some hidden personality trait, say conscientiousness, for which "education" is a possible proxy. In this case, trying to correct for some measure of race/gender access to education is still not going to work. For example, it is generally reported that females score higher on both measures of conscientiousness and educational achievement. Would you lower the value of your "education" score for females and raise them for males on this basis to correct for the bias in educational achievement against those with low conscientiousness? It would only seem to make the predictive value of the feature worse.
    - mirker 5y ago
      It depends on the definition of fairness, which is what I’m trying to argue for. Practically speaking, heavy correlation and casual relationships are similarly problematic. If you correlate enough of the variables, you can limit your pool to a very particular group of people. And if your group of people is 99% of race/gender/age, then is that practically any different from doing a direct filter? The usual definitions of fairness actually have the strong constraint that the response (Y=f(X)) is independent of the fairness variables (f(X|Z) is the same for all fairness values of Z). So, concretely, if Z is gender, then you have the same insurance outcome regardless of gender. In this definition, the setting I described is unfair, since the correlation on education will bias the model to predict in a way that is correlated with race. The only practical way to satisfy such a strong definition is to do a correction on the data, which is a pretty straightforward optimization problem if you formalize the constraints. And yeah, usually you lose predictive power in doing so. For the record, the strong definition makes sense often times (to me). You wouldn’t want health insurance using your DNA as a predictor, likely, even if genetics are very predictive in that domain.
Spivak 5y ago
I really don’t think anyone is really going to disagree with you that data is truth and actions are what matters but you run into two problems. * If you draw any conclusion from the data it will reflect the biases in the data. * If you take any action from the data it will reflect the biases in the data. So you’re going to be “correcting” at some point if you want to avoid that bias which is what really matters.
- mirker 5y ago
  Right. And once you start doing corrections you are basically making another ML model to turn “what happened” into “what should have happened”.
- cratermoon 5y ago
  I disagree. Data is data. Before it's even collected and organized, bias and error creep in. Is the Library of Congress card catalog a collection of data that is "truth" about books? Let's take a look at Library of Congress Classification Outline: Class D: https://www.loc.gov/catdir/cpso/lcco/lcco_d.pdf https://www.loc.gov/catdir/cpso/lcco/lcco_d.pdf D – History (General) DA – History of Great Britain – History of Ireland – History of the British Isles DAW – History of Central Europe OK so far, I guess? DB – History of Austria – Liechtenstein – Hungary – Czechoslovakia DC – History of France – Andorra – Monaco Wait, what? I left out a bunch here, but various countries get their own class, or share a class with one or two other countries. But I guess not "central Europe? 10 DH – History of Low Countries – Benelux Countries Huh? 17 DR – History of the Balkan Peninsula Really drilling down here 18 DS – History of Asia Uhm, Asia? ALL of Asia? 19 DT – History of Africa 20 DU – History of Oceania (South Seas) Seems fair. Not. 21 DX – History of Gypsies OK now they're just trolling us.
  - mulmen 5y ago
    I don't follow. This is how the Library of Congress catalogs books. That is the truth in the data. I don't see the problem with having a category for books on Central Europe in general and another section for individual countries that happen to be in Central Europe. Some books cover general topics and others specialize. Shouldn't we expect datasets that describe the content of books to reflect this?
pjmorris 5y ago
IMO, 'Data is truth' is not true. The data the machine sees was chosen in a way biased, at least, by the intent of the data collectors and by the capability of the sensors used to collect the data, both of which are shadows of reality. The Y2K problem is an illustration.