14 ms·

Is the problem that AI is biased? Or is the problem that AI lacks the biases that we expect people to have? For instance, people have detected that AI models a

Is the problem that AI is biased? Or is the problem that AI lacks the biases that we expect people to have? For instance, people have detected that AI models associate technical words with men more often than women and point to this as evidence of bias. I would argue that this is the opposite situation. There's social stigma attached to acknowledging differences between groups, so people have developed biased against acknowledging those differences. The AI on the other hand sees that 70-80% of technologists are men, and associates accordingly. So the problem isn't that the AI is biased it's that it lacks the biases we expect. Now, there are good arguments why this bias may be a good thing. Unbiased does not automatically mean good.

dmix 7y ago
That's true, a lot social activists are very much in favour of putting biases in systems. That's also pretty much what most gov policy is, various competing special interests fighting to intervene in various systems in a favourable direction.
bilbo0s 7y ago
Wait? Isn't "biased in a way we don't expect", itself, a bias? Unless I'm misunderstanding you, you're saying the AI is biased, and people are biased. Unbiased is an AI that will not miss an outlier. My field is medical imaging. The most difficult thing to get the AI vendors that come calling on us to understand is that being able to recognize well known patterns in data is of limited use to us. That's what the radiologists at remote sites do all day. It's being able to recognize pathologies that 99.999% of rads would definitely have missed that has value. And the AI's always fail this test. I've come to believe that "bias" is the primary reason these AI's perform so poorly. They are basically "biased towards the known stuff" in layman's terms. At least in my field, we need AI's that are able to make connections that humans cannot. I've been going to RSNA for decades now, and I would say we haven't seen a single AI that could be said to pass this kind of a test yet.
- pigscantfly 7y ago
  I've been working in data science or in academia for most of the last decade and would argue that the primary benefits of using statistical models for automation are that they are fast and cheap compared to a trained professional, which frees those trained professionals up to spend more time on the difficult judgment calls. I've seen very few examples of superhuman performance on complex, practical tasks (although it is very satisfying to train a model to that level).
- ska 7y ago
  > And the AI's always fail this test. This isn't quite true as it has been successful in limited scopes since the 90's (i.e. well before the existence and rise of deep learning). See for example microcalc TP rates on CAD breast screening that was beating average radiologists back then, but not on general screening. This is somewhat specialized though, and there are important constraints to screening also, as opposed to diagnostic radiology. But you are right in an important way - it isn't a feature for the current throw-a-ton-of-data-at-a-deep-model approach that a lot of "AI companies" are built around, and I agree they are mostly knocking on the wrong doors in medical imaging. Importantly, if you are trying to replace people at a relatively simple task and make it cheaper your approach will look much different than the case you describe, where you are trying to learn unusual configurations and long-tail sort of things. ML in medical imaging is complicated by a lot of factors, but people that think a little transfer learning and a few hundred case reports is going to get somewhere useful are being incredibly naive. Without exception, all the promising work I have seen has involved painful and expensive labeling and a lot of modeling and pre-processing effort.
- manfredo 7y ago
  > I've come to believe that "bias" is the primary reason these AI's perform so poorly. They are basically "biased towards the known stuff" in layman's terms. At least in my field, we need AI's that are able to make connections that humans cannot. AI, at least current AI, cannot do this. I'm really confused at the way you say AI perform poorly because they're biased towards "known stuff". We train AI on known data. AI being biased towards "known stuff" is fundamental to the way our current AI technology functions.
  - lazyasciiart 7y ago
    You really sound like this topic of bias in AI is new to you. Here’s a real world example: a medical school used to be male-only. Recently they began admitting women. More recently, they began running “AI” trained on past student applications to figure out which applications were likely to be successful students (where successful == graduating normally). Not surprisingly at all, it turned out that being female wasn’t associated with being a good student there, because it was an uncommon attribute of previous students. This is a poor result. Would you argue that the model should not be adjusted?
    - manfredo 7y ago
      The field of AI is not new to me. Quite the opposite, I find that many people in this thread are making fundamental misunderstandings about what AI and machine learning are about. Hence the commenters like the one above which say AI is limited because it is biased towards past data, which sounds ridiculous to anyone that works with AI because using past data to train the model is fundamental to how AI works. Yes, if people feed in a bad data set the results are going to be bad. Like how a pentagon project tried to use AI to classify Russian vs NATO vehicles. The photos of the former were taken on a sunny day and the latter on an overcast day. The result was that their model was just detecting the average brightness of the input. But what happens when the data is valid, but people don't like the results because it doesn't match their worldview? Altering the model to match one's world view is not elimination of bias, it's deliberately introducing bias.
      - 7y ago
seph-reed 7y ago
Humans really don't like to feel like their infinite potential is less than infinite. Pair that with the inherent feedback loop of bias in human controlled systems and viola! Never ending cycles of soul crushing but realistic statistical thinking, and glorious but idealistic "we can be free" thinking.
mtgx 7y ago
> Is the problem that AI is biased? Or is the problem that AI lacks the biases that we expect people to have? But can't you see - that's exactly the problem! Let's take the paper clip theory as an example. Humans tell AI: "Build as many paper clips as possible, given all available resources!" AI then proceeds to destroy all humans and the planet to make the paper clips. So yeah, in this case we do have a "bias" of "don't kill us in the process". Yes, this is a very extreme example, but the gist is that this is the problem with AI - you can't really tell it to do something, without also having all sorts of unexpected consequences (from the humans' point of view, of course) that would be bad for humans.
gbrown 7y ago
I think you've very severely missed the point. Differences like those you describe generally reflect biases in our socal structure, which are then (correctly, but problematically) reflected in statistical models and algorithms of all stripes. This becomes more problematic when we overinterpret the resulting findings, or uncritically incorporate the resulting predictions into real systems. A good example is the criminal justice system in the US. Minority communities are much more intensively policed, prosecuted, and jailed. A model will, without care, use race or poxies thereof to predict "criminality" without understanding or accounting for the bias inherent in the label. If that is used in policy, it runs the risk of amplifying existing social problems and injustices. We alway have to ask: "bias for what?", or the conversation will be hopelessly confused.
- nostromo 7y ago
  It's unpopular to talk about, but there is quantifiably more crime in some minority communities in the US. There are some crimes, like drug use, that are indeed over-reported in some communities due to over-policing. But there are other crimes, like homicide, that are nearly universally reported, regardless of where they are committed. And those crimes are much more frequent in minority-majority parts of the country. Turning a blind eye to this problem is a disservice to those communities, because they are the same communities that are most commonly the victims of crimes.
  - crooked-v 7y ago
    > there is quantifiably more crime in some minority communities Sure... and what about the chronic lead exposure issues in many of these areas, the effects of long-term overpolicing of minor crimes on family stability, and the impact of (lack of) generational wealth and education from being used as de facto or literal slaves for generations? The 'minority' part here is a correlative factor, not a causative one.
    - nostromo 7y ago
      An AI classifier doesn't know, or care, why a particular population has higher crime and lower income, it only recognizes and reports on the pattern.
majormajor 7y ago
Your definition of "AI" seems to be "summary statistics" if all it's doing is saying "here's the current distribution." Men vs women seems like a pretty unsophisticated and crude feature. If AI's going to be useful it would be able to find out deeper underlying things, faster than we can ourselves. If bias exists in people today, it's going to skew outcomes, so looking at stats on current outcomes alone isn't going to tell you much. An AI that's blind to conditioning and upbringing and everything else that's part of society is not a particularly useful AI, you could replace them with any average person off the street's naive "common sense."
- nostromo 7y ago
  > you could replace them with any average person off the street's naive "common sense." If only... An AI with common sense is an AI researcher's dream come true.
hinkley 7y ago
More broadly, if AI enforces the status quo, then it becomes an enemy of progress. Or rather, it risks becoming the last progress we make. So far we aren't doing a very good job of coming up with ways to live in harmony with AI, instead of in an antagonistic relationship. We can do better than this.
crooked-v 7y ago
You seem to be trying to very politely slip in the claim here that some inherent difference between genders is why there are more men 'technologists', but that percentage varies wildly between different countries and companies.
- remarkEon 7y ago
  I’m not sure how this happened, but somewhere along the line the claim that there are no inherent differences between genders became less extreme than the claim that there might be some. You’re not making the former claim, but I see it all the time and it’s just as nuts as saying that 100% of observed difference are due to biology.
  - crooked-v 7y ago
    The difference in treatment is probably because these days the people who talk loudest about the "some" are often trying to use it as a cover to support gender-essentialist defenses of an unequal status quo, while people who talk about "none" are obviously inaccurate in some ways but are for the most part harmless.
    - remarkEon 7y ago
      Well, I don’t know about you but I like my claims to reflect reality and not some ideal that is touted as “probably wrong but mostly harmless!”
- manfredo 7y ago
  It does not vary wildly between countries. The overwhelming majority of countries fall into the range of 10-30% of tech roles filled by women and the remaining 70-90% by men. I don't think that a wide variety of countries falling into a band of ~20% is what most people would characterize as "varies wildly". Regardless, the cause of the disparity is not relevant to the discussion at hand. The real world population of people involved in technology is disproportionately of one gender, so an unbiased machine learning model meant to associate terms with genders will associate technology terms with men at higher rates. Sure, some may claim that the reason why disproportionately more men go into technology is because society is biased. But the AI's role was not to model a hypothetical world with no biases. Its role was to model the real world. As I wrote in my root comment, some may see value in biasing their models to achieve results that fit what they see as an ideal worldview. But the point remains: to do this is to deliberately introduce biases, not remove them.
Buttons840 7y ago
AI is biased. It is biased towards the past. We feed data from the past into our statistical models and "AIs" and then present the results as though it's what will happen in the future. But the future will not be like the past, not exactly, and thus we have bias. (As a philosophical question, why do we expect the future to be like the past? Well, because that is how things have worked in the past. And thus we arrive at a nice circular argument. :)
ska 7y ago
> Is the problem that AI is biased? Or is the problem that AI lacks the biases that we expect people to have? Neither. The problem referred to is that if the data sets contain biases then the machine learning will learn them.
- manfredo 7y ago
  Sure. But what it means to "contain biases" is a very ambiguous statement. If, say, a dataset was fed to the model that had an underrepresentation of women in tech that would be a biases input. But if the AI was fed a an input that had 20-25% women in tech, which is a representative sample, and the AI associated men with tech because men make up 75-80% of people in tech thats not a biased sample. Sure, some may point out that biases in other parts of society may be part of the reason why tech is 70-80% male, but the AI itself is not biased. If we were to engineer the AI to produce equal results despite unequal input we are engineering biases into the AI, not removing them.
  - ska 7y ago
    It really isn't ambiguous. What is less clear is what to do about it, and where that even makes sense. There are few meanings of bias that are important to keep clear about when discussing things. (1) There is the statistical sense of a biased estimator - one with a consistent trend in it's error. (2) There is the notion of bias introduced by data sampling. No matter how perfect your algorithm is, if the training data is a poor sampling of the general population you are targeting, you are likely introducing systematic bias (for example, early face detection approaches had best performance on Caucasian, male, college aged faces - people were using the data it was easiest for them to collect[1]. Finally (3) if the data you have access to has encoded a systematic bias, even if the first 2 have been avoided you are at best able to reinforce that bias. Here we are mostly talking about latter one, and problems being encountered it (and a bit of 2). This is exacerbated by a combination of machine-learning and AI people being fairly unsophisticated about data on average (as opposed to data handling), and popular techniques these days (specifically, deep learning) de-emphasizing feature design making it harder sometimes to see what is happening. Nobody serious I have seen is advocating engineering outcomes into AI to adjust outcomes. For the sake of argument let's assume there is in fact an over representation of men (compared to women) in technology relative to capability and desire. And that we are designing an AI to make or aid hiring decisions for entry level jobs. The answer then is not to engineer in a quota for women applicants, but to remove gender entirely from the training and evaluation inputs. This would give you exactly the desired outcome, no? You refer to the type (2) bias issue if we systematically under represent women in tech in this case, which would be a problem. However the article is focused on the type (3), which is not merely an issue of what is in the data set, but what you are trying to do with it. The deep issue here is that deep learning approach of throw everything at the inputs and let the network sort it out will capture both (2) and (3) types of biases whether or not we are aware of them. At least in areas where there is very objective proof of potential problems in the historical data (e.g. redlining impact on mortgage decisions) we could get ahead of it and normalize the inputs. But what about areas where it is less clear cut or more contentious? There is the fact that humans fail in exactly the same way. If I'm making the decision to hire, it's very difficult for me to avoid my own biases. I think one of the things people are concerned about with applying ML techniques to things like this is it gives the appearance of being more likely to be unbiased, and potentially provide cover from those who would like to benefit from it. I suspect the real answer is to start thinking about these things the same way we have learned to thing about security and cryptography. In other words, for serious work a system is not considered ready for prime time until real professionals have tried to find the weak points and break it. It is also worth noting that this is not necessarily a problem so much as an operational feature of the approaches one should be aware of when designing and using them. And in some applications it probably a very significant problem. [1] one of the interesting thing about this and similar problems is how completely predictable it was, and how surprising it was for practitioners to discover it. In this way the ML community needlessly recapitulated lessons learned by other disciplines decades earlier.
thrower123 7y ago
People love the idea of being data-driven, right up until the point where they feed the data in, and what comes out the other side doesn't match their preconceived notions of what it ought to show. Then they start poking and prodding the model, and slicing and dicing the data until it rounds into what they wanted to see.
landryraccoon 7y ago
A danger of this view is that it can turn a descriptive view of the world into a normative view of the world. Suppose an AI does detect that men are more technical, on average, than women. It would be a colossal mistake to then assume that our society should be a society where men are on average more technical than women based merely on the fact that historically that has been the case. Would this sort of AI have entrenched past social evils, such as slavery, if in the 1860s it classified people according to the data available at the time? The fact that an AI classifies things accurately given the statistical data at the time should not be confused with giving a normative claim that that is the desired state of affairs of the world.
YeGoblynQueenne 7y ago
Well, a language model is supposed to model the way that language works, not the way that language is used. It might be a subtle difference, but "My mom is a doctor" is as correct grammatically as "My dad is a doctor". If a language model is assigning a much lower probability to the former, then it's modelling language _use_ but not language _structure_. So it's not doing the job it's supposed to be doing.
loopz 7y ago
The problem is that the problem of bias is undefined, so there's no correct solution or even authority to turn to. We do see problems with systems taking data as gospel, but when the data itself have undesirable patterns, we're at loss at what to do. Of course, simply saying this triggers some creativity: You could generate artificial data and train on that. However, the problem is then to generate "unbiased" artificial data, and somehow make it useful! ;-) So who's vision should be realized, then becomes the problem, or solution for some subset of people.

dmix 7y ago

That's true, a lot social activists are very much in favour of putting biases in systems. That's also pretty much what most gov policy is, various competing special interests fighting to intervene in various systems in a favourable direction.

bilbo0s 7y ago

Wait? Isn't "biased in a way we don't expect", itself, a bias? Unless I'm misunderstanding you, you're saying the AI is biased, and people are biased. Unbiased is an AI that will not miss an outlier. My field is medical imaging. The most difficult thing to get the AI vendors that come calling on us to understand is that being able to recognize well known patterns in data is of limited use to us. That's what the radiologists at remote sites do all day. It's being able to recognize pathologies that 99.999% of rads would definitely have missed that has value. And the AI's always fail this test. I've come to believe that "bias" is the primary reason these AI's perform so poorly. They are basically "biased towards the known stuff" in layman's terms. At least in my field, we need AI's that are able to make connections that humans cannot. I've been going to RSNA for decades now, and I would say we haven't seen a single AI that could be said to pass this kind of a test yet.

pigscantfly 7y ago
I've been working in data science or in academia for most of the last decade and would argue that the primary benefits of using statistical models for automation are that they are fast and cheap compared to a trained professional, which frees those trained professionals up to spend more time on the difficult judgment calls. I've seen very few examples of superhuman performance on complex, practical tasks (although it is very satisfying to train a model to that level).
ska 7y ago
> And the AI's always fail this test. This isn't quite true as it has been successful in limited scopes since the 90's (i.e. well before the existence and rise of deep learning). See for example microcalc TP rates on CAD breast screening that was beating average radiologists back then, but not on general screening. This is somewhat specialized though, and there are important constraints to screening also, as opposed to diagnostic radiology. But you are right in an important way - it isn't a feature for the current throw-a-ton-of-data-at-a-deep-model approach that a lot of "AI companies" are built around, and I agree they are mostly knocking on the wrong doors in medical imaging. Importantly, if you are trying to replace people at a relatively simple task and make it cheaper your approach will look much different than the case you describe, where you are trying to learn unusual configurations and long-tail sort of things. ML in medical imaging is complicated by a lot of factors, but people that think a little transfer learning and a few hundred case reports is going to get somewhere useful are being incredibly naive. Without exception, all the promising work I have seen has involved painful and expensive labeling and a lot of modeling and pre-processing effort.
manfredo 7y ago
> I've come to believe that "bias" is the primary reason these AI's perform so poorly. They are basically "biased towards the known stuff" in layman's terms. At least in my field, we need AI's that are able to make connections that humans cannot. AI, at least current AI, cannot do this. I'm really confused at the way you say AI perform poorly because they're biased towards "known stuff". We train AI on known data. AI being biased towards "known stuff" is fundamental to the way our current AI technology functions.
- lazyasciiart 7y ago
  You really sound like this topic of bias in AI is new to you. Here’s a real world example: a medical school used to be male-only. Recently they began admitting women. More recently, they began running “AI” trained on past student applications to figure out which applications were likely to be successful students (where successful == graduating normally). Not surprisingly at all, it turned out that being female wasn’t associated with being a good student there, because it was an uncommon attribute of previous students. This is a poor result. Would you argue that the model should not be adjusted?
  - manfredo 7y ago
    The field of AI is not new to me. Quite the opposite, I find that many people in this thread are making fundamental misunderstandings about what AI and machine learning are about. Hence the commenters like the one above which say AI is limited because it is biased towards past data, which sounds ridiculous to anyone that works with AI because using past data to train the model is fundamental to how AI works. Yes, if people feed in a bad data set the results are going to be bad. Like how a pentagon project tried to use AI to classify Russian vs NATO vehicles. The photos of the former were taken on a sunny day and the latter on an overcast day. The result was that their model was just detecting the average brightness of the input. But what happens when the data is valid, but people don't like the results because it doesn't match their worldview? Altering the model to match one's world view is not elimination of bias, it's deliberately introducing bias.
    - 7y ago

seph-reed 7y ago

Humans really don't like to feel like their infinite potential is less than infinite. Pair that with the inherent feedback loop of bias in human controlled systems and viola! Never ending cycles of soul crushing but realistic statistical thinking, and glorious but idealistic "we can be free" thinking.

mtgx 7y ago

> Is the problem that AI is biased? Or is the problem that AI lacks the biases that we expect people to have? But can't you see - that's exactly the problem! Let's take the paper clip theory as an example. Humans tell AI: "Build as many paper clips as possible, given all available resources!" AI then proceeds to destroy all humans and the planet to make the paper clips. So yeah, in this case we do have a "bias" of "don't kill us in the process". Yes, this is a very extreme example, but the gist is that this is the problem with AI - you can't really tell it to do something, without also having all sorts of unexpected consequences (from the humans' point of view, of course) that would be bad for humans.

gbrown 7y ago

I think you've very severely missed the point. Differences like those you describe generally reflect biases in our socal structure, which are then (correctly, but problematically) reflected in statistical models and algorithms of all stripes. This becomes more problematic when we overinterpret the resulting findings, or uncritically incorporate the resulting predictions into real systems. A good example is the criminal justice system in the US. Minority communities are much more intensively policed, prosecuted, and jailed. A model will, without care, use race or poxies thereof to predict "criminality" without understanding or accounting for the bias inherent in the label. If that is used in policy, it runs the risk of amplifying existing social problems and injustices. We alway have to ask: "bias for what?", or the conversation will be hopelessly confused.

nostromo 7y ago
It's unpopular to talk about, but there is quantifiably more crime in some minority communities in the US. There are some crimes, like drug use, that are indeed over-reported in some communities due to over-policing. But there are other crimes, like homicide, that are nearly universally reported, regardless of where they are committed. And those crimes are much more frequent in minority-majority parts of the country. Turning a blind eye to this problem is a disservice to those communities, because they are the same communities that are most commonly the victims of crimes.
- crooked-v 7y ago
  > there is quantifiably more crime in some minority communities Sure... and what about the chronic lead exposure issues in many of these areas, the effects of long-term overpolicing of minor crimes on family stability, and the impact of (lack of) generational wealth and education from being used as de facto or literal slaves for generations? The 'minority' part here is a correlative factor, not a causative one.
  - nostromo 7y ago
    An AI classifier doesn't know, or care, why a particular population has higher crime and lower income, it only recognizes and reports on the pattern.

majormajor 7y ago

Your definition of "AI" seems to be "summary statistics" if all it's doing is saying "here's the current distribution." Men vs women seems like a pretty unsophisticated and crude feature. If AI's going to be useful it would be able to find out deeper underlying things, faster than we can ourselves. If bias exists in people today, it's going to skew outcomes, so looking at stats on current outcomes alone isn't going to tell you much. An AI that's blind to conditioning and upbringing and everything else that's part of society is not a particularly useful AI, you could replace them with any average person off the street's naive "common sense."

nostromo 7y ago
> you could replace them with any average person off the street's naive "common sense." If only... An AI with common sense is an AI researcher's dream come true.

hinkley 7y ago

More broadly, if AI enforces the status quo, then it becomes an enemy of progress. Or rather, it risks becoming the last progress we make. So far we aren't doing a very good job of coming up with ways to live in harmony with AI, instead of in an antagonistic relationship. We can do better than this.

crooked-v 7y ago

You seem to be trying to very politely slip in the claim here that some inherent difference between genders is why there are more men 'technologists', but that percentage varies wildly between different countries and companies.

remarkEon 7y ago
I’m not sure how this happened, but somewhere along the line the claim that there are no inherent differences between genders became less extreme than the claim that there might be some. You’re not making the former claim, but I see it all the time and it’s just as nuts as saying that 100% of observed difference are due to biology.
- crooked-v 7y ago
  The difference in treatment is probably because these days the people who talk loudest about the "some" are often trying to use it as a cover to support gender-essentialist defenses of an unequal status quo, while people who talk about "none" are obviously inaccurate in some ways but are for the most part harmless.
  - remarkEon 7y ago
    Well, I don’t know about you but I like my claims to reflect reality and not some ideal that is touted as “probably wrong but mostly harmless!”
manfredo 7y ago
It does not vary wildly between countries. The overwhelming majority of countries fall into the range of 10-30% of tech roles filled by women and the remaining 70-90% by men. I don't think that a wide variety of countries falling into a band of ~20% is what most people would characterize as "varies wildly". Regardless, the cause of the disparity is not relevant to the discussion at hand. The real world population of people involved in technology is disproportionately of one gender, so an unbiased machine learning model meant to associate terms with genders will associate technology terms with men at higher rates. Sure, some may claim that the reason why disproportionately more men go into technology is because society is biased. But the AI's role was not to model a hypothetical world with no biases. Its role was to model the real world. As I wrote in my root comment, some may see value in biasing their models to achieve results that fit what they see as an ideal worldview. But the point remains: to do this is to deliberately introduce biases, not remove them.

Buttons840 7y ago

AI is biased. It is biased towards the past. We feed data from the past into our statistical models and "AIs" and then present the results as though it's what will happen in the future. But the future will not be like the past, not exactly, and thus we have bias. (As a philosophical question, why do we expect the future to be like the past? Well, because that is how things have worked in the past. And thus we arrive at a nice circular argument. :)

ska 7y ago

> Is the problem that AI is biased? Or is the problem that AI lacks the biases that we expect people to have? Neither. The problem referred to is that if the data sets contain biases then the machine learning will learn them.

manfredo 7y ago
Sure. But what it means to "contain biases" is a very ambiguous statement. If, say, a dataset was fed to the model that had an underrepresentation of women in tech that would be a biases input. But if the AI was fed a an input that had 20-25% women in tech, which is a representative sample, and the AI associated men with tech because men make up 75-80% of people in tech thats not a biased sample. Sure, some may point out that biases in other parts of society may be part of the reason why tech is 70-80% male, but the AI itself is not biased. If we were to engineer the AI to produce equal results despite unequal input we are engineering biases into the AI, not removing them.
- ska 7y ago
  It really isn't ambiguous. What is less clear is what to do about it, and where that even makes sense. There are few meanings of bias that are important to keep clear about when discussing things. (1) There is the statistical sense of a biased estimator - one with a consistent trend in it's error. (2) There is the notion of bias introduced by data sampling. No matter how perfect your algorithm is, if the training data is a poor sampling of the general population you are targeting, you are likely introducing systematic bias (for example, early face detection approaches had best performance on Caucasian, male, college aged faces - people were using the data it was easiest for them to collect[1]. Finally (3) if the data you have access to has encoded a systematic bias, even if the first 2 have been avoided you are at best able to reinforce that bias. Here we are mostly talking about latter one, and problems being encountered it (and a bit of 2). This is exacerbated by a combination of machine-learning and AI people being fairly unsophisticated about data on average (as opposed to data handling), and popular techniques these days (specifically, deep learning) de-emphasizing feature design making it harder sometimes to see what is happening. Nobody serious I have seen is advocating engineering outcomes into AI to adjust outcomes. For the sake of argument let's assume there is in fact an over representation of men (compared to women) in technology relative to capability and desire. And that we are designing an AI to make or aid hiring decisions for entry level jobs. The answer then is not to engineer in a quota for women applicants, but to remove gender entirely from the training and evaluation inputs. This would give you exactly the desired outcome, no? You refer to the type (2) bias issue if we systematically under represent women in tech in this case, which would be a problem. However the article is focused on the type (3), which is not merely an issue of what is in the data set, but what you are trying to do with it. The deep issue here is that deep learning approach of throw everything at the inputs and let the network sort it out will capture both (2) and (3) types of biases whether or not we are aware of them. At least in areas where there is very objective proof of potential problems in the historical data (e.g. redlining impact on mortgage decisions) we could get ahead of it and normalize the inputs. But what about areas where it is less clear cut or more contentious? There is the fact that humans fail in exactly the same way. If I'm making the decision to hire, it's very difficult for me to avoid my own biases. I think one of the things people are concerned about with applying ML techniques to things like this is it gives the appearance of being more likely to be unbiased, and potentially provide cover from those who would like to benefit from it. I suspect the real answer is to start thinking about these things the same way we have learned to thing about security and cryptography. In other words, for serious work a system is not considered ready for prime time until real professionals have tried to find the weak points and break it. It is also worth noting that this is not necessarily a problem so much as an operational feature of the approaches one should be aware of when designing and using them. And in some applications it probably a very significant problem. [1] one of the interesting thing about this and similar problems is how completely predictable it was, and how surprising it was for practitioners to discover it. In this way the ML community needlessly recapitulated lessons learned by other disciplines decades earlier.

thrower123 7y ago

People love the idea of being data-driven, right up until the point where they feed the data in, and what comes out the other side doesn't match their preconceived notions of what it ought to show. Then they start poking and prodding the model, and slicing and dicing the data until it rounds into what they wanted to see.

landryraccoon 7y ago

A danger of this view is that it can turn a descriptive view of the world into a normative view of the world. Suppose an AI does detect that men are more technical, on average, than women. It would be a colossal mistake to then assume that our society should be a society where men are on average more technical than women based merely on the fact that historically that has been the case. Would this sort of AI have entrenched past social evils, such as slavery, if in the 1860s it classified people according to the data available at the time? The fact that an AI classifies things accurately given the statistical data at the time should not be confused with giving a normative claim that that is the desired state of affairs of the world.

YeGoblynQueenne 7y ago

Well, a language model is supposed to model the way that language works, not the way that language is used. It might be a subtle difference, but "My mom is a doctor" is as correct grammatically as "My dad is a doctor". If a language model is assigning a much lower probability to the former, then it's modelling language _use_ but not language _structure_. So it's not doing the job it's supposed to be doing.

loopz 7y ago

The problem is that the problem of bias is undefined, so there's no correct solution or even authority to turn to. We do see problems with systems taking data as gospel, but when the data itself have undesirable patterns, we're at loss at what to do. Of course, simply saying this triggers some creativity: You could generate artificial data and train on that. However, the problem is then to generate "unbiased" artificial data, and somehow make it useful! ;-) So who's vision should be realized, then becomes the problem, or solution for some subset of people.