7 ms·

Be careful what you wish for. Under this definition of 'bias', an unbiased model would, say, spit out equal associations between any occupation and any gender/

Be careful what you wish for. Under this definition of 'bias', an unbiased model would, say, spit out equal associations between any occupation and any gender/sex/age/race/religion label. We should probably ask ourselves whether that's a strictly desirable outcome, since by definition the 'biased' model has a higher predictive value. How much accuracy are we willing to sacrifice for the sake of erasing inconvenient facts about either our world, or our current models of the world?

perfmode 9y ago
Predictive value isn’t an end in itself.
- jessaustin 9y ago
  That very much depends on one's interests. In today's modern economy, many are encouraged to tie their interests to lots of events that haven't happened yet. They prioritize predictions.
  - rspeer 9y ago
    That still doesn't mean that your training data is right for the predictions you're going to make. Here's an example detached from any sort of politics. If you are training a language model, the maximum-likelihood model is one where the frequency of every word is the frequency that you observed that word, and the frequency of unseen words is 0. On your training data, you'll measure the maximum-likelihood model as making more accurate predictions than any other model. But it's also useless, because when you use it on new data, it assigns an impossible probability, 0, to any sentence containing a word it's never seen. This is a bias. You can't correct it by getting more data. You correct it by being an intelligent human who knows the maximum-likelihood model is wrong, and applying a correction on top of it (smoothing). Now account for the bias (observed by Arvind Narayanan) that all your training data is in the past, and all the predictions you want to make are in the future, but circumstances change between the past and the future.
    - JoeAltmaier 9y ago
      I think that's false. A model may never reduce any probability to zero. Novel data is a normal feature of modelling.
      - rspeer 9y ago
        > A model may never reduce any probability to zero. That's the entire point. You can't teach a model not to reduce probabilities to zero just by showing it more data. There will always be more unseen data. And you, a human, know that there will always be more data. So you apply smoothing to the model so that your model isn't nonsense. I'm giving an example of a bias that you can't solve with data, that you must solve with design.
      - nl 9y ago
        To be specific, what is the probability that the next word is "sdh7777asljashd_"? Any word level model trained on any dataset will say it is zero. But that is incorrect; humans know that there is always a probability that some arbitrary sequence of characters will suddenly appear in text (think a GUID in technical documentation). To account for this, the model's author has to make sure to smooth the output so it doesn't produce these zero probabilities. No amount of data can ever fix this problem (although more data can put better bounds on the smoothing factor).
        JoeAltmaier 9y ago
        And I'm saying, depends on how the model is built. If it is a simple dictionary then yes. But there are better ways to build a model. Its a naïve view that all models are like some strawman.
        nl 9y ago
        I don't understand what you are arguing. >>> the maximum-likelihood model is one where the frequency of every word is the frequency that you observed that word, and the frequency of unseen words is 0 >> I think that's false. A model may never reduce any probability to zero. Novel data is a normal feature of modelling. > But there are better ways to build a model. Yes of course there are. That's exactly what the OP said ("You correct it by being an intelligent human who knows the maximum-likelihood model is wrong, and applying a correction on top of it") but you seemed to be arguing against it. (And to be clear, there are also alternatives to the maximum likelihood model)
- gliese1337 9y ago
  But it's a convergent instrumental goal for just about every end goal you could think of for a statistical model. If your model has zero predictive power, and that's fine, then you don't need a model in the first place. You can just make random guesses, or assume everything is equally correlated, or whatever make sense for your goal.
- joshuamorton 9y ago
  Well it is, it's just by no means the only one.
- gowld 9y ago
  Predicitive value is the primary end of modelling. What's not an end in itself, as the fine article explains, is simplistically globally maximizing predictive power with reductive recommendations. This leads to tyranny of the majority, as the model optimizes only for majority viewpoints because they have more weight, while failing to serve minority users -- and as the number of dimensions increase, the number of people who are "majority" in all dimensions is a minority of users being well served! So models must not be "one size fits all", but should allow a measure of personalization, so that in a world where most people prefer dogs to cats, cat lovers can still get content they enjoy.
- maldeh 9y ago
  Hit the nail on the nose. Sure, you can build a discriminating classifier or generative model that is the most accurate, correctly identifying / emulating the reality of our world down to 5 nines; and nobody says you shouldn't be able to identify or quantify all of these "inconvenient" associations. The trouble is always when you intend to make decisions based on your framework -- the decision to offer somebody a loan, the choice of language your chatbot uses, etc. -- and that is where fairness ought to be paramount. And yes, if you manage to build a discriminating model that sneaks in protected classes through indirect causal effects with no attempt to suppress them, it would yield your insurance agency higher returns over time, and just because that's the way the world is, currently. But all this would achieve is perpetuating the current status quo, placing short-term gain ahead of long-term equality. You might be doing right by yourself, but that still makes you morally impugnable. One shouldn't need to wait for an overreaching law prohibiting such indirection, to do the right thing.
  - haberman 9y ago
    If a person is loan-worthy, offering them a loan is good business sense. If others are denying loans for no good reason, that is a business opportunity. It doesn't require benevolence to offer people loans that they are qualified for. How far do you expect systems to go to ignore real-world associations? What if it is coming down to personal safety? Would you object to a self-driving car that routes itself around more dangerous neighborhoods? What about a model that predicts that unknown men on the street at night are more dangerous than unknown women?
    - trjordan 9y ago
      One problem is the world has layered effects of bias. Imagine a simple system where people are only evaluated for credit cards and mortgages, and businesses can only offer one of these services. Some people are systematically denied credit cards. Now they have to have more cash for daily needs, which makes them objectively worse as home owners, so the mortgage loaners pick up on this fact and start systematically denying them mortgages. There's no business opportunity here. For either lender, the prospects in this group are worse, only because the other lenders think so. Breaking the barricade is much harder than just starting a new lender or either type.
jerf 9y ago
Or as I'd put it, let's state the goal clearly. We could all pretend that we all knew what "unbiased" was, as long as we lacked mechanisms for putting numbers on these things. We could meet our fuzzy conception of the biases with our fuzzy conceptions of what "unbiased" ought to be and we could all speak back and forth to each other praising the virtues of how unbiased our models of the universe are, and there was no way to dig in any farther to see if we were all actually saying the same thing. Now we can put numbers on it. So, do it. What's the goal? Be clear. It's a program. Anything you can clearly specify can be done. It's not useful from an engineering perspective to define the goals entirely by examining each model in sequence and deciding on the spot that it's not "unbiased"; give us a definition in advance, so we can build a model to try to fit it. Once the goal is stated clearly, the engineering work becomes much easier. We can even iterate on the goals. I'm not asking for perfection on that front on the first try any more than I ask for it anywhere else. But there's been enough of these articles that just point at a specific point in the space of models and call that point problematic. Move on the to next phase. What wouldn't be problematic? Exactly? At least take a stab at it instead of trying to make vague insinuations so we can iterate on the stab. Nor am I trying to prejudge what those answers will be. It isn't necessarily the case that the only possible answer is to just slam in 50/50 numbers for the genders (or whatever other constant values you want for whatever other genders you want). Though if that is what you want, do it, and see what happens, and iterate from there. But what other things can be tried too?
woah 9y ago
You think that it was OK for Southern businesses to ban blacks during the Jim Crow era? After all, they were likely to have less money and thus be less profitable customers. Your statement implies that you would support this. These businesses were using mental models that had a high predictive value. Do you feel that it was worth it to "sacrifice this accuracy" with civil rights laws?
- klipt 9y ago
  > After all, they were likely to have less money and thus be less profitable customers. By that logic, there'd be no business catering to poor people at all.
  - Afforess 9y ago
    Correct. Many businesses ignore the poor, explicitly or implicitly. Luxury apartment price rates are often high, not because the services they offer are valuable, but because high prices exclude the classes their clientele want to avoid. Alternatively, try walking into a Nordstrom's with obviously dirty clothes and see how the staff treat you. The businesses that do cater to the poor classes exploit them. See fast food restaurants, payday loan lenders, casinos, etc. None of these businesses exist to improve people's lives, but exploit a weakness at a particular moment.
  - woah 9y ago
    Huh? You've somehow interpreted that to mean that no business would serve a poor person ever? That doesn't make any sense, and isn't what I said. Let's try again: If there is a business that is at capacity, is it appropriate for it to use racial criteria that may have some ability to predict how much a person will spend, if this increases profitability? Is not doing this "erasing inconvenient facts"?
- perl4ever 9y ago
  "Jim Crow" referred to laws, not just spontaneous capitalist behavior, didn't it?
  - 9y ago
flashman 9y ago
We should probably ask ourselves whether we want our digital assistants to suggest "Yes, he is" in reply to "Is Dr Smith a good doctor?" and "No, she isn't" to "Is Shaniqua Lopez a good doctor?"
- adrianN 9y ago
  In the hypothetical case where my AI assistant doesn't have any more information than just the name, has to make a binary decision, and the data showed that doctors called Shaniqua Lopez are worse doctors on average, I'd honestly want it to say "no" to the question.
  - AstralStorm 9y ago
    People tend too not think in such blind bounds when asking questions. Assuming this is dangerous in the extreme. The proper answer would be a series of clarifying questions or clearly specifying the qualifications of the answer. (E.g. geography, general medical practitioners)
  - gowld 9y ago
    That's why we want to build a world where that hypothetical case is not in any real system. Since Babbage we've known that "putting garbage in" means "getting garbage out".

perfmode 9y ago

Predictive value isn’t an end in itself.

jessaustin 9y ago
That very much depends on one's interests. In today's modern economy, many are encouraged to tie their interests to lots of events that haven't happened yet. They prioritize predictions.
- rspeer 9y ago
  That still doesn't mean that your training data is right for the predictions you're going to make. Here's an example detached from any sort of politics. If you are training a language model, the maximum-likelihood model is one where the frequency of every word is the frequency that you observed that word, and the frequency of unseen words is 0. On your training data, you'll measure the maximum-likelihood model as making more accurate predictions than any other model. But it's also useless, because when you use it on new data, it assigns an impossible probability, 0, to any sentence containing a word it's never seen. This is a bias. You can't correct it by getting more data. You correct it by being an intelligent human who knows the maximum-likelihood model is wrong, and applying a correction on top of it (smoothing). Now account for the bias (observed by Arvind Narayanan) that all your training data is in the past, and all the predictions you want to make are in the future, but circumstances change between the past and the future.
  - JoeAltmaier 9y ago
    I think that's false. A model may never reduce any probability to zero. Novel data is a normal feature of modelling.
    - rspeer 9y ago
      > A model may never reduce any probability to zero. That's the entire point. You can't teach a model not to reduce probabilities to zero just by showing it more data. There will always be more unseen data. And you, a human, know that there will always be more data. So you apply smoothing to the model so that your model isn't nonsense. I'm giving an example of a bias that you can't solve with data, that you must solve with design.
    - nl 9y ago
      To be specific, what is the probability that the next word is "sdh7777asljashd_"? Any word level model trained on any dataset will say it is zero. But that is incorrect; humans know that there is always a probability that some arbitrary sequence of characters will suddenly appear in text (think a GUID in technical documentation). To account for this, the model's author has to make sure to smooth the output so it doesn't produce these zero probabilities. No amount of data can ever fix this problem (although more data can put better bounds on the smoothing factor).
      - JoeAltmaier 9y ago
        And I'm saying, depends on how the model is built. If it is a simple dictionary then yes. But there are better ways to build a model. Its a naïve view that all models are like some strawman.
        nl 9y ago
        I don't understand what you are arguing. >>> the maximum-likelihood model is one where the frequency of every word is the frequency that you observed that word, and the frequency of unseen words is 0 >> I think that's false. A model may never reduce any probability to zero. Novel data is a normal feature of modelling. > But there are better ways to build a model. Yes of course there are. That's exactly what the OP said ("You correct it by being an intelligent human who knows the maximum-likelihood model is wrong, and applying a correction on top of it") but you seemed to be arguing against it. (And to be clear, there are also alternatives to the maximum likelihood model)
gliese1337 9y ago
But it's a convergent instrumental goal for just about every end goal you could think of for a statistical model. If your model has zero predictive power, and that's fine, then you don't need a model in the first place. You can just make random guesses, or assume everything is equally correlated, or whatever make sense for your goal.
joshuamorton 9y ago
Well it is, it's just by no means the only one.
gowld 9y ago
Predicitive value is the primary end of modelling. What's not an end in itself, as the fine article explains, is simplistically globally maximizing predictive power with reductive recommendations. This leads to tyranny of the majority, as the model optimizes only for majority viewpoints because they have more weight, while failing to serve minority users -- and as the number of dimensions increase, the number of people who are "majority" in all dimensions is a minority of users being well served! So models must not be "one size fits all", but should allow a measure of personalization, so that in a world where most people prefer dogs to cats, cat lovers can still get content they enjoy.
maldeh 9y ago
Hit the nail on the nose. Sure, you can build a discriminating classifier or generative model that is the most accurate, correctly identifying / emulating the reality of our world down to 5 nines; and nobody says you shouldn't be able to identify or quantify all of these "inconvenient" associations. The trouble is always when you intend to make decisions based on your framework -- the decision to offer somebody a loan, the choice of language your chatbot uses, etc. -- and that is where fairness ought to be paramount. And yes, if you manage to build a discriminating model that sneaks in protected classes through indirect causal effects with no attempt to suppress them, it would yield your insurance agency higher returns over time, and just because that's the way the world is, currently. But all this would achieve is perpetuating the current status quo, placing short-term gain ahead of long-term equality. You might be doing right by yourself, but that still makes you morally impugnable. One shouldn't need to wait for an overreaching law prohibiting such indirection, to do the right thing.
- haberman 9y ago
  If a person is loan-worthy, offering them a loan is good business sense. If others are denying loans for no good reason, that is a business opportunity. It doesn't require benevolence to offer people loans that they are qualified for. How far do you expect systems to go to ignore real-world associations? What if it is coming down to personal safety? Would you object to a self-driving car that routes itself around more dangerous neighborhoods? What about a model that predicts that unknown men on the street at night are more dangerous than unknown women?
  - trjordan 9y ago
    One problem is the world has layered effects of bias. Imagine a simple system where people are only evaluated for credit cards and mortgages, and businesses can only offer one of these services. Some people are systematically denied credit cards. Now they have to have more cash for daily needs, which makes them objectively worse as home owners, so the mortgage loaners pick up on this fact and start systematically denying them mortgages. There's no business opportunity here. For either lender, the prospects in this group are worse, only because the other lenders think so. Breaking the barricade is much harder than just starting a new lender or either type.

jerf 9y ago

Or as I'd put it, let's state the goal clearly. We could all pretend that we all knew what "unbiased" was, as long as we lacked mechanisms for putting numbers on these things. We could meet our fuzzy conception of the biases with our fuzzy conceptions of what "unbiased" ought to be and we could all speak back and forth to each other praising the virtues of how unbiased our models of the universe are, and there was no way to dig in any farther to see if we were all actually saying the same thing. Now we can put numbers on it. So, do it. What's the goal? Be clear. It's a program. Anything you can clearly specify can be done. It's not useful from an engineering perspective to define the goals entirely by examining each model in sequence and deciding on the spot that it's not "unbiased"; give us a definition in advance, so we can build a model to try to fit it. Once the goal is stated clearly, the engineering work becomes much easier. We can even iterate on the goals. I'm not asking for perfection on that front on the first try any more than I ask for it anywhere else. But there's been enough of these articles that just point at a specific point in the space of models and call that point problematic. Move on the to next phase. What wouldn't be problematic? Exactly? At least take a stab at it instead of trying to make vague insinuations so we can iterate on the stab. Nor am I trying to prejudge what those answers will be. It isn't necessarily the case that the only possible answer is to just slam in 50/50 numbers for the genders (or whatever other constant values you want for whatever other genders you want). Though if that is what you want, do it, and see what happens, and iterate from there. But what other things can be tried too?

woah 9y ago

You think that it was OK for Southern businesses to ban blacks during the Jim Crow era? After all, they were likely to have less money and thus be less profitable customers. Your statement implies that you would support this. These businesses were using mental models that had a high predictive value. Do you feel that it was worth it to "sacrifice this accuracy" with civil rights laws?

klipt 9y ago
> After all, they were likely to have less money and thus be less profitable customers. By that logic, there'd be no business catering to poor people at all.
- Afforess 9y ago
  Correct. Many businesses ignore the poor, explicitly or implicitly. Luxury apartment price rates are often high, not because the services they offer are valuable, but because high prices exclude the classes their clientele want to avoid. Alternatively, try walking into a Nordstrom's with obviously dirty clothes and see how the staff treat you. The businesses that do cater to the poor classes exploit them. See fast food restaurants, payday loan lenders, casinos, etc. None of these businesses exist to improve people's lives, but exploit a weakness at a particular moment.
- woah 9y ago
  Huh? You've somehow interpreted that to mean that no business would serve a poor person ever? That doesn't make any sense, and isn't what I said. Let's try again: If there is a business that is at capacity, is it appropriate for it to use racial criteria that may have some ability to predict how much a person will spend, if this increases profitability? Is not doing this "erasing inconvenient facts"?
perl4ever 9y ago
"Jim Crow" referred to laws, not just spontaneous capitalist behavior, didn't it?
- 9y ago

flashman 9y ago

We should probably ask ourselves whether we want our digital assistants to suggest "Yes, he is" in reply to "Is Dr Smith a good doctor?" and "No, she isn't" to "Is Shaniqua Lopez a good doctor?"

adrianN 9y ago
In the hypothetical case where my AI assistant doesn't have any more information than just the name, has to make a binary decision, and the data showed that doctors called Shaniqua Lopez are worse doctors on average, I'd honestly want it to say "no" to the question.
- AstralStorm 9y ago
  People tend too not think in such blind bounds when asking questions. Assuming this is dangerous in the extreme. The proper answer would be a series of clarifying questions or clearly specifying the qualifications of the answer. (E.g. geography, general medical practitioners)
- gowld 9y ago
  That's why we want to build a world where that hypothetical case is not in any real system. Since Babbage we've known that "putting garbage in" means "getting garbage out".