5 ms·

> Why on earth would we expect AI descision making to display a lack of prejudice when human decision making is suffused with it. Particularly when we consider

> Why on earth would we expect AI descision making to display a lack of prejudice when human decision making is suffused with it. Particularly when we consider what we mean by prejudice, which is presumably something like Making a decision on grounds which we deem it important to ignore. This is a very complex concept. It's a function of society, and changes with society. It's not something with a rigorous definition. Obvious example: reasonable modern people know it's indefensible to make an engineering hiring decision on the grounds of ethnicity, regardless of whether there are any correlations associated with ethnicity. This is even enshrined in law in many countries. To make a decision on the grounds of someone's qualifications and job experience, however, does not count as prejudice. We should expect a machine learning system to act as a correlation-seeker (that is after all what it is designed to do), without a nuanced understanding of what prejudice means. We've seen this issue crop up in the context of an AI having a say in parole decisions. [0] also relevant discussion at [1] [0] https://www.forbes.com/sites/bernardmarr/2019/01/29/3-steps-to-tackle-the-problem-of-bias-in-artificial-intelligence/ https://www.forbes.com/sites/bernardmarr/2019/01/29/3-steps-... [1] https://news.ycombinator.com/item?id=13156153 https://news.ycombinator.com/item?id=13156153

dasil003 7y ago
> We should expect a machine learning system to act as a correlation-seeker [...], without a nuanced understanding of what prejudice means. One likely outcome is that increased usage of ML will lay bare more gray-area correlations that force us to confront broader systematic prejudice in an uncomfortable way. Now whether we make material changes based on these discoveries or whether we leverage our significant ability for post-hoc rationalization to paper over the cognitive dissonance is another question entirely.
- moduspol 7y ago
  > more gray-area correlations that force us to confront broader systematic prejudice in an uncomfortable way Or, perhaps we discover that correlations don't necessarily point to systematic prejudice, and continue to argue over which correlations are more indicative of prejudice than others. Some point at the proportion of women in top software engineering job and claim prejudice in hiring. Others point at the hiring pool for top software engineering job and disagree. And around and around we go. Machine learning won't solve that.
jonnypotty 7y ago
The boundry between informed and uninformed decisions becomes blurred doesn't it. Very interesting.
marcosdumay 7y ago
> We should expect a machine learning system to act as a correlation-seeker We should expect machine learning systems to dismiss accidental indirect correlation for the benefit of the variables that are directly correlated. There are plenty of algorithms that achieve that, it's only that gradient descent doesn't. The fact that our AIs are becoming biased is a bug. It should be fixed.
- MaxBarraclough 7y ago
  No, that doesn't sound right. It doesn't matter which particular algorithm you're thinking of. The issue here is at a different level. Let's consider an AI that weighs in on hiring decisions. Let's consider what it might make of maternity. Decades ago, it wasn't unusual for a prospective employer to ask a female applicant whether she was planning on getting pregnant. Society decided that wasn't acceptable, and made the practice unlawful (as well as forcing employers to offer paid maternity leave), despite that, from a profit-seeking perspective, that is valuable information for the prospective employer to have. Let's suppose an AI has been trained on a dataset of previous hiring decisions, and some measure of how much value those employees went on to deliver. Let's suppose it's also told the age and sex of the applicants. It might notice that female applicants within a certain age range, have a tendency to have a long period of zero productivity, some time after being hired. Having noticed this correlation, it would then discriminate against applicants matching that profile. But wait!, I hear you think, Why did we provide it data on sex and age? Remove those data-points and the problem goes away. Not so. There could be non-obvious correlations acting as proxies for that information. The system might detect these correlations. In a parole-board context, suspects' names and addresses could be proxies for ethnicity, which could correlate with recidivism rates. The 'correct' behaviour for a correlation-seeking AI is to detect those correlations, and then start to systematically discriminate against individuals of certain ethnicities, even when their crime and personal conduct is identical to another individual of a different ethnicity. Back to our maternity leave example, then. It's an interesting exercise to come up with possible correlations. The ones that occur to me: * Given names (if provided to the AI) generally tell you a person's sex * Given names also fall in and out of fashion over the decades, so there's a correlation with age * The topics a person chose to study at university, can correlate with the person's sex, as well as their age, as some fields didn't exist 40 years ago * The culture a person is from, may impact the statistically expected number of children for them to have, and at what age. Various things could act as proxies for this.
Retric 7y ago
Prejudice is using a proxy to approximate something else. It’s inherently inaccurate like trying to find the fastest vehicle and then listening them by price tag. The problem for AI is these proxies can seem to be very useful. Want to know how fast a vehicle is, well price seems like a really good approximation when you start. Except, the more information you have the more likely price is going to fool you.

dasil003 7y ago

> We should expect a machine learning system to act as a correlation-seeker [...], without a nuanced understanding of what prejudice means. One likely outcome is that increased usage of ML will lay bare more gray-area correlations that force us to confront broader systematic prejudice in an uncomfortable way. Now whether we make material changes based on these discoveries or whether we leverage our significant ability for post-hoc rationalization to paper over the cognitive dissonance is another question entirely.

moduspol 7y ago
> more gray-area correlations that force us to confront broader systematic prejudice in an uncomfortable way Or, perhaps we discover that correlations don't necessarily point to systematic prejudice, and continue to argue over which correlations are more indicative of prejudice than others. Some point at the proportion of women in top software engineering job and claim prejudice in hiring. Others point at the hiring pool for top software engineering job and disagree. And around and around we go. Machine learning won't solve that.

jonnypotty 7y ago

The boundry between informed and uninformed decisions becomes blurred doesn't it. Very interesting.

marcosdumay 7y ago

> We should expect a machine learning system to act as a correlation-seeker We should expect machine learning systems to dismiss accidental indirect correlation for the benefit of the variables that are directly correlated. There are plenty of algorithms that achieve that, it's only that gradient descent doesn't. The fact that our AIs are becoming biased is a bug. It should be fixed.

MaxBarraclough 7y ago
No, that doesn't sound right. It doesn't matter which particular algorithm you're thinking of. The issue here is at a different level. Let's consider an AI that weighs in on hiring decisions. Let's consider what it might make of maternity. Decades ago, it wasn't unusual for a prospective employer to ask a female applicant whether she was planning on getting pregnant. Society decided that wasn't acceptable, and made the practice unlawful (as well as forcing employers to offer paid maternity leave), despite that, from a profit-seeking perspective, that is valuable information for the prospective employer to have. Let's suppose an AI has been trained on a dataset of previous hiring decisions, and some measure of how much value those employees went on to deliver. Let's suppose it's also told the age and sex of the applicants. It might notice that female applicants within a certain age range, have a tendency to have a long period of zero productivity, some time after being hired. Having noticed this correlation, it would then discriminate against applicants matching that profile. But wait!, I hear you think, Why did we provide it data on sex and age? Remove those data-points and the problem goes away. Not so. There could be non-obvious correlations acting as proxies for that information. The system might detect these correlations. In a parole-board context, suspects' names and addresses could be proxies for ethnicity, which could correlate with recidivism rates. The 'correct' behaviour for a correlation-seeking AI is to detect those correlations, and then start to systematically discriminate against individuals of certain ethnicities, even when their crime and personal conduct is identical to another individual of a different ethnicity. Back to our maternity leave example, then. It's an interesting exercise to come up with possible correlations. The ones that occur to me: * Given names (if provided to the AI) generally tell you a person's sex * Given names also fall in and out of fashion over the decades, so there's a correlation with age * The topics a person chose to study at university, can correlate with the person's sex, as well as their age, as some fields didn't exist 40 years ago * The culture a person is from, may impact the statistically expected number of children for them to have, and at what age. Various things could act as proxies for this.

Retric 7y ago

Prejudice is using a proxy to approximate something else. It’s inherently inaccurate like trying to find the fastest vehicle and then listening them by price tag. The problem for AI is these proxies can seem to be very useful. Want to know how fast a vehicle is, well price seems like a really good approximation when you start. Except, the more information you have the more likely price is going to fool you.