4 ms·

Coauthor here. The blog post is written in relatively non-technical language for a general audience, but our paper has tons of technical details that HN readers

Coauthor here. The blog post is written in relatively non-technical language for a general audience, but our paper has tons of technical details that HN readers might enjoy. Give it a read! http://randomwalker.info/publications/language-bias.pdf http://randomwalker.info/publications/language-bias.pdf

cs702 10y ago
Doing this was a great idea. Great paper: easy-to-follow and to-the-point. The results are not too surprising, as the models for learning word embeddings like GloVe, word2vec, etc. learn to map to vectors existing relationships between words in training corpora. If a corpus is biased, the embeddings learned from it will necessarily be biased too. However, the implications of this finding are wide-ranging. For starters, any machine learning system that relies on word embeddings learned from biased corpora to make predictions (or to make decisions!) will necessarily be biased in favor of certain groups of people and against others. Moreover, it's not obvious to me how one would go about obtaining "unbiased" corpora without somehow relying on subjective societal values that are different everywhere and continually evolving. You have raised an important, non-trivial problem.
- wolfgke 10y ago
  > Moreover, it's not obvious to me how one would go about obtaining "unbiased" corpora without somehow relying on subjective societal values that are different everywhere and continually evolving. I don't believe that problem will ever be completely solvable. But I think the road to go is to make these assumptions always explicit. I.e. when the machine learning system derives a result, program it to additionally return a proof of how it came to this result. And also give a way to let the ML system return a list of all axioms and derivation rules that it has currently learned, so that they can independently be checked how much they are biased and can thus be corrected.
  - nl 10y ago
    It's pretty hard to return "rules" for a ML system, especially a non-linear system. Google is currently working systems that use a trillion features - I can't imagine returning some kind of rule list for that. LIME[1] is a nice start, though. [1] https://www.oreilly.com/learning/introduction-to-local-interpretable-model-agnostic-explanations-lime https://www.oreilly.com/learning/introduction-to-local-inter...
    - wolfgke 10y ago
      Another text about interpreting convolutional neural networks: http://cs231n.github.io/understanding-cnn/ http://cs231n.github.io/understanding-cnn/ > Google is currently working systems that use a trillion features - I can't imagine returning some kind of rule list for that. As I wrote: It would already help if the ML system as a first step returned the derivation with only the rules that were concretely used for a concrete derivation - this list is much shorter and can thus much easier be checked.
- yummyfajitas 10y ago
  For starters, any machine learning system that relies on word embeddings learned from biased corpora to make predictions (or to make decisions!) will necessarily be biased in favor of certain groups of people and against others. This is not true. Here's an oversimplified example. Suppose your machine learning system wants to predict something, e.g. loan repayment probabilities. One input might be a written evaluation by a loan officer. When trained on a corpora of group X, the predicted probability might be: pred = a*written_evaluation + other_factors (Using linear regression to make example simple.) However, now lets suppose the written evaluation is biased to the tune of 25% against group Y. I.e., group Y has written scores that are 25% less than group X. Then a new predictor which includes pairwise terms, trained on a corpora of group X and Y, will work out to be: pred = a*written_evaluation + 0.33*written_evaluation*isY + other_factors This predictor would be unbiased. In general, if you have a biased input and the biasing factor is also present in your input, your model should correct the bias. (Obvious caveats: your model needs to be sufficiently expressive, etc.) Interestingly, everyone's favorite bogeyman, namely redundant encoding ( http://deliprao.com/archives/129 http://deliprao.com/archives/129 ) will actually help fix this problem *even if you don't include the biasing factors in the model.
  - cs702 10y ago
    ...now lets suppose the written evaluation is biased to the tune of 25% against group Y... How do you find out that the written evaluation is biased "to the tune of 25% against group Y?" THAT is the problem. It's not obvious to me how you would go about determining written evaluations are biased (and to what extent!) against group Y without somehow relying on subjective societal values that are different everywhere and continually evolving.
    - yummyfajitas 10y ago
      Finding out is the easy part. I don't mean to trivialize it, because doing stats right is actually a very technical matter, but this is just ordinary statistics. You build a sufficiently expressive statistical model and include the potentially biasing factors as features in the model. Then the model will correct the bias all by itself because correcting for bias maximizes accuracy. In the example above, you find the bias by doing linear regression and including (written_evaluation x isY) as a term. Least squares will handle the rest. If you using something fancier than least squares (e.g. deep neural networks, SVMs with interesting kernels), you probably don't even need to explicitly include potentially debiasing terms - the model will do it for you. I give toy examples (designed to illustrate the point and also be easy to understand) here: https://www.chrisstucchio.com/blog/2016/alien_intelligences_and_discriminatory_algorithms.html https://www.chrisstucchio.com/blog/2016/alien_intelligences_... This paper does the same thing - it discovers that standard predictors of college performance (grades, GPA) are biased in favor of blacks and men, against Asians and women, and the model itself fixes these biases: http://ftp.iza.org/dp8733.pdf http://ftp.iza.org/dp8733.pdf Statistics turns fixing racism into a math problem. If the topic were anything less emotionally charged, you wouldn't even think twice about it. If I suggested including `isMobile`, `isDesktop` and `isTablet` as features in an ad-targeting algorithm to deal with the fact that users on mobile and desktop browse differently, you'd yawn.
joe_the_user 10y ago
Gave it a quick read. Are biases distinct from "preferences" - humans view flowers as more pleasurable than insects - human language associates flowers with pleasurable terms, states and so-forth. "Bias" is term associated with "irrational beliefs" whereas "preferences" more often imply "arbitrary preferences". Especially, biases are held to prevent rational deduction whereas preferences have no such stumbling block. Now, one supposes that question would come down to whether a computer would "know it's a computer, not a person". If the AI was asked "do you like cockroaches or daisies better", would it say "why daises are prettier and smell better" or would it say "most people like daisies but I'm a machine, can't smell or taste, and only care about the preferences entered into my control panel" (or something). And you'd expect that a thing that merely "parroted" human speech without understanding would give the former answer. Which is to say I don't think you are really fully grappling with word-association and word-logic coming together, ie, "meaning".
jdp23 10y ago
Very interesting results. I really like the approach of paralleling the classic bias experiments. And I think your recommendations in the last paragraph of the "Awareness is better than blindness" section are excellent - although I'd go farther and suggest that the long-term interdisciplinary research program should have a highly diverse team, and include experts on diversity. I thought the section on "Challenges" could have been stronger. You talk about the bias in "the basic representation of knowledge" used in these systems today -- but it's not like there isn't aren't other possible representations of knowledge. How much effort has gone into exploring knowledge representation (and approaches to derive semantics) that are designed to highlight and reduce biases look like?
sideshowb 10y ago
Hi. I'd be interested to know what you think of my attempts to deduce bias from a text corpus. Very early stage compared to yours mind https://linkingideasblog.wordpress.com/2015/08/19/data-mining-for-the-taboo-searching-for-what-isnt-there/ https://linkingideasblog.wordpress.com/2015/08/19/data-minin...

cs702 10y ago

Doing this was a great idea. Great paper: easy-to-follow and to-the-point. The results are not too surprising, as the models for learning word embeddings like GloVe, word2vec, etc. learn to map to vectors existing relationships between words in training corpora. If a corpus is biased, the embeddings learned from it will necessarily be biased too. However, the implications of this finding are wide-ranging. For starters, any machine learning system that relies on word embeddings learned from biased corpora to make predictions (or to make decisions!) will necessarily be biased in favor of certain groups of people and against others. Moreover, it's not obvious to me how one would go about obtaining "unbiased" corpora without somehow relying on subjective societal values that are different everywhere and continually evolving. You have raised an important, non-trivial problem.

wolfgke 10y ago
> Moreover, it's not obvious to me how one would go about obtaining "unbiased" corpora without somehow relying on subjective societal values that are different everywhere and continually evolving. I don't believe that problem will ever be completely solvable. But I think the road to go is to make these assumptions always explicit. I.e. when the machine learning system derives a result, program it to additionally return a proof of how it came to this result. And also give a way to let the ML system return a list of all axioms and derivation rules that it has currently learned, so that they can independently be checked how much they are biased and can thus be corrected.
- nl 10y ago
  It's pretty hard to return "rules" for a ML system, especially a non-linear system. Google is currently working systems that use a trillion features - I can't imagine returning some kind of rule list for that. LIME[1] is a nice start, though. [1] https://www.oreilly.com/learning/introduction-to-local-interpretable-model-agnostic-explanations-lime https://www.oreilly.com/learning/introduction-to-local-inter...
  - wolfgke 10y ago
    Another text about interpreting convolutional neural networks: http://cs231n.github.io/understanding-cnn/ http://cs231n.github.io/understanding-cnn/ > Google is currently working systems that use a trillion features - I can't imagine returning some kind of rule list for that. As I wrote: It would already help if the ML system as a first step returned the derivation with only the rules that were concretely used for a concrete derivation - this list is much shorter and can thus much easier be checked.
yummyfajitas 10y ago
For starters, any machine learning system that relies on word embeddings learned from biased corpora to make predictions (or to make decisions!) will necessarily be biased in favor of certain groups of people and against others. This is not true. Here's an oversimplified example. Suppose your machine learning system wants to predict something, e.g. loan repayment probabilities. One input might be a written evaluation by a loan officer. When trained on a corpora of group X, the predicted probability might be: pred = a*written_evaluation + other_factors (Using linear regression to make example simple.) However, now lets suppose the written evaluation is biased to the tune of 25% against group Y. I.e., group Y has written scores that are 25% less than group X. Then a new predictor which includes pairwise terms, trained on a corpora of group X and Y, will work out to be: pred = a*written_evaluation + 0.33*written_evaluation*isY + other_factors This predictor would be unbiased. In general, if you have a biased input and the biasing factor is also present in your input, your model should correct the bias. (Obvious caveats: your model needs to be sufficiently expressive, etc.) Interestingly, everyone's favorite bogeyman, namely redundant encoding ( http://deliprao.com/archives/129 http://deliprao.com/archives/129 ) will actually help fix this problem *even if you don't include the biasing factors in the model.
- cs702 10y ago
  ...now lets suppose the written evaluation is biased to the tune of 25% against group Y... How do you find out that the written evaluation is biased "to the tune of 25% against group Y?" THAT is the problem. It's not obvious to me how you would go about determining written evaluations are biased (and to what extent!) against group Y without somehow relying on subjective societal values that are different everywhere and continually evolving.
  - yummyfajitas 10y ago
    Finding out is the easy part. I don't mean to trivialize it, because doing stats right is actually a very technical matter, but this is just ordinary statistics. You build a sufficiently expressive statistical model and include the potentially biasing factors as features in the model. Then the model will correct the bias all by itself because correcting for bias maximizes accuracy. In the example above, you find the bias by doing linear regression and including (written_evaluation x isY) as a term. Least squares will handle the rest. If you using something fancier than least squares (e.g. deep neural networks, SVMs with interesting kernels), you probably don't even need to explicitly include potentially debiasing terms - the model will do it for you. I give toy examples (designed to illustrate the point and also be easy to understand) here: https://www.chrisstucchio.com/blog/2016/alien_intelligences_and_discriminatory_algorithms.html https://www.chrisstucchio.com/blog/2016/alien_intelligences_... This paper does the same thing - it discovers that standard predictors of college performance (grades, GPA) are biased in favor of blacks and men, against Asians and women, and the model itself fixes these biases: http://ftp.iza.org/dp8733.pdf http://ftp.iza.org/dp8733.pdf Statistics turns fixing racism into a math problem. If the topic were anything less emotionally charged, you wouldn't even think twice about it. If I suggested including `isMobile`, `isDesktop` and `isTablet` as features in an ad-targeting algorithm to deal with the fact that users on mobile and desktop browse differently, you'd yawn.

joe_the_user 10y ago

Gave it a quick read. Are biases distinct from "preferences" - humans view flowers as more pleasurable than insects - human language associates flowers with pleasurable terms, states and so-forth. "Bias" is term associated with "irrational beliefs" whereas "preferences" more often imply "arbitrary preferences". Especially, biases are held to prevent rational deduction whereas preferences have no such stumbling block. Now, one supposes that question would come down to whether a computer would "know it's a computer, not a person". If the AI was asked "do you like cockroaches or daisies better", would it say "why daises are prettier and smell better" or would it say "most people like daisies but I'm a machine, can't smell or taste, and only care about the preferences entered into my control panel" (or something). And you'd expect that a thing that merely "parroted" human speech without understanding would give the former answer. Which is to say I don't think you are really fully grappling with word-association and word-logic coming together, ie, "meaning".

jdp23 10y ago

Very interesting results. I really like the approach of paralleling the classic bias experiments. And I think your recommendations in the last paragraph of the "Awareness is better than blindness" section are excellent - although I'd go farther and suggest that the long-term interdisciplinary research program should have a highly diverse team, and include experts on diversity. I thought the section on "Challenges" could have been stronger. You talk about the bias in "the basic representation of knowledge" used in these systems today -- but it's not like there isn't aren't other possible representations of knowledge. How much effort has gone into exploring knowledge representation (and approaches to derive semantics) that are designed to highlight and reduce biases look like?

sideshowb 10y ago

Hi. I'd be interested to know what you think of my attempts to deduce bias from a text corpus. Very early stage compared to yours mind https://linkingideasblog.wordpress.com/2015/08/19/data-mining-for-the-taboo-searching-for-what-isnt-there/ https://linkingideasblog.wordpress.com/2015/08/19/data-minin...