7 ms·

Kind of. AI/ML systems are supposed to learn things from the data they are fed to accomplish some sort of more generalizable goal or task. Echoing the biases th

by wpasc 7y ago

Kind of. AI/ML systems are supposed to learn things from the data they are fed to accomplish some sort of more generalizable goal or task. Echoing the biases they're fed is probably more akin to overfitting to their training data.

shantly 7y ago
A glance over some dictionaries reveals I'm leaning on an essentially archaic and poetic sense of "bias" here to evade its meaning exclusively an unreasonable judgement, it seems.
Doingmything123 7y ago
I guess the question is how well to we expect AI/ML systems to be able to generalize on a biased subset data. If you are trying to use one of these systems to make hiring decisions but you train it on data from the hires you made, should we really be surprised if biases in the original hiring(education level, gender, race, income, personality) are reflected in the AI's decisions?
bsanr2 7y ago
The line between fit and overfit is arbitrary, though. What is happening is that the systems are revealing unconscious biases present in the more traditional systems the AI/ML are supposed to replicate, build upon, or speed up. If the results are odious to us, it should be impetus to critically analyze not only the AI/ML systems, but also the underlying assumptions that they're built on. Instead, developers become defensive and cage-y about their processes. If you don't want systems to have disparate impact, you have to be adamant about it in your design. If you think society is better off with systems that reflect preexisting biases, then fine, but be ready for the backlash. At the end of the day, it really is up to what humans want to do with themselves. It's an opportunity to be truer to our intent, not a bug to be covered up.
- im3w1l 7y ago
  > The line between fit and overfit is arbitrary, though. What is happening is that the systems are revealing unconscious biases present in the more traditional systems the AI/ML are supposed to replicate, build upon, or speed up. When you say unconscious bias, you are kind of implying that the model learns something that is false. But more often it's the case that the model learns something true that we don't want it to learn. That's what makes the problem so hard, you are trying to hide the truth from a system you only half-understand processing data you only half-understand. There is a big risk the truth slips through the cracks if you aren't careful.
  - wan23 7y ago
    > When you say unconscious bias, you are kind of implying that the model learns something that is false. It's more that the model learns something that is undesirable. It could be the case, for example, that the true thing that the AI learns is that your resume screening process tends to exclude women. This is true, sure, but it could lead to the undesirable outcome where the presence of a female name on a resume might be weighted heavily against the candidate.
  - bsanr2 7y ago
    Well, in many cases, they are indeed learning something false (or, to be more charitable, something that is not sufficiently true). For example, a facial recognition system that has trouble detecting dark-skinned faces. Black and brown people clearly have faces, but because the systems were not trained on their facial data, and the capability to adequately adjust exposure or other aspects of the sensor stack was not built into the system, it provides an objectively wrong solution given perfectly reasonable inputs. I agree that this process is one of resolving blind spots, but I disagree that the blind spots are simply areas devoid of light. AI/ML systems are frequently employed to augment or stand in for human perception, which is known to be necessarily incomplete with respect to reality. In other words, they can learn things that seem true to us but that are false from another perspective, or undesirable once exposed. What's exciting about them is that they provide an opportunity to interrogate the flaws in our individual perception with a systematized observation and analysis, in a much more sophisticated manner than in the past. But fulfilling that potential requires humility.
gambler 7y ago
>AI/ML systems are supposed to learn things from the data they are fed to accomplish some sort of more generalizable goal or task. ...which means that whether a model is "biased" depends on where and how it's applied. This is an important point that is missing from most discussions, articles and even research papers on the so-called "AI ethics".
- int_19h 7y ago
  Such a model is objectively biased regardless of where and how it's applied - it prefers X over Y, and that's bias by definition. The hard question here is deciding whether any given bias is ethical or not.
rcthompson 7y ago
No, it's not the same as overfitting. It's nothing to do with the algorithm at all. The problem is with the training data, which contains patterns induced by the way the data was collected rather than the reality of what you want to classify. For example, imagine that you wanted to train an algorithm to distinguish photos of dogs from photos humans. So you collect a bunch of photos of both dogs and humans and use them to train a classifier. You do all the proper cross-validation, bootstrapping, etc. to ensure that you are not overfitting, and you get really good results. Then, looking at the mis-classifications, you notice something: all the photos that are taken looking at an angle down toward the ground are classified as dog photos, and all the photos taken looking straight ahead are classified as human photos. It turns out that in your training set, most of the dog photos are taken at a downward angle while must of the human photos are taken facing straight ahead, because humans are taller than dogs, and your machine learning algorithm identified this feature as the most reliable way to distinguish the two groups of photos in your training set. In this hypothetical example, no overfitting occurred. The difference in photo angles is a real difference in the training sets that you provided to the algorithm, and the algorithm did its job and correctly identified this difference between the two groups of photos as a reliable predictor. The problem is that your training set has a variable (photo angle) that is highly correlated with what you want to classify (species). This is considered an unwanted bias (and not a reliable indicator) because the correlation is caused by the means of data collection (most photos are taken from human head height) and has nothing to do with the subject of the photos.
- CardenB 7y ago
  I think you’re arguing semantics a bit. What you’re saying checks out, but one could say that overfitting was occurring but the test dataset distribution was not wide enough to catch it.
  - etbebl 7y ago
    As I understand it, if it's overfitting, testing on a second random sample gathered in the same way as the training set should degrade performance. That's not the case here. (Though maybe the term as used in industry is less strict.)
haberman 7y ago
The test for overfitting is whether the model holds up for data outside the training set. If the biases are consistent with other real-world data, then it's not overfitting.
deleted 7y ago
[deleted]

by wpasc 7y ago

shantly 7y ago

A glance over some dictionaries reveals I'm leaning on an essentially archaic and poetic sense of "bias" here to evade its meaning exclusively an unreasonable judgement, it seems.

Doingmything123 7y ago

I guess the question is how well to we expect AI/ML systems to be able to generalize on a biased subset data. If you are trying to use one of these systems to make hiring decisions but you train it on data from the hires you made, should we really be surprised if biases in the original hiring(education level, gender, race, income, personality) are reflected in the AI's decisions?

bsanr2 7y ago

The line between fit and overfit is arbitrary, though. What is happening is that the systems are revealing unconscious biases present in the more traditional systems the AI/ML are supposed to replicate, build upon, or speed up. If the results are odious to us, it should be impetus to critically analyze not only the AI/ML systems, but also the underlying assumptions that they're built on. Instead, developers become defensive and cage-y about their processes. If you don't want systems to have disparate impact, you have to be adamant about it in your design. If you think society is better off with systems that reflect preexisting biases, then fine, but be ready for the backlash. At the end of the day, it really is up to what humans want to do with themselves. It's an opportunity to be truer to our intent, not a bug to be covered up.

im3w1l 7y ago
> The line between fit and overfit is arbitrary, though. What is happening is that the systems are revealing unconscious biases present in the more traditional systems the AI/ML are supposed to replicate, build upon, or speed up. When you say unconscious bias, you are kind of implying that the model learns something that is false. But more often it's the case that the model learns something true that we don't want it to learn. That's what makes the problem so hard, you are trying to hide the truth from a system you only half-understand processing data you only half-understand. There is a big risk the truth slips through the cracks if you aren't careful.
- wan23 7y ago
  > When you say unconscious bias, you are kind of implying that the model learns something that is false. It's more that the model learns something that is undesirable. It could be the case, for example, that the true thing that the AI learns is that your resume screening process tends to exclude women. This is true, sure, but it could lead to the undesirable outcome where the presence of a female name on a resume might be weighted heavily against the candidate.
- bsanr2 7y ago
  Well, in many cases, they are indeed learning something false (or, to be more charitable, something that is not sufficiently true). For example, a facial recognition system that has trouble detecting dark-skinned faces. Black and brown people clearly have faces, but because the systems were not trained on their facial data, and the capability to adequately adjust exposure or other aspects of the sensor stack was not built into the system, it provides an objectively wrong solution given perfectly reasonable inputs. I agree that this process is one of resolving blind spots, but I disagree that the blind spots are simply areas devoid of light. AI/ML systems are frequently employed to augment or stand in for human perception, which is known to be necessarily incomplete with respect to reality. In other words, they can learn things that seem true to us but that are false from another perspective, or undesirable once exposed. What's exciting about them is that they provide an opportunity to interrogate the flaws in our individual perception with a systematized observation and analysis, in a much more sophisticated manner than in the past. But fulfilling that potential requires humility.

gambler 7y ago

>AI/ML systems are supposed to learn things from the data they are fed to accomplish some sort of more generalizable goal or task. ...which means that whether a model is "biased" depends on where and how it's applied. This is an important point that is missing from most discussions, articles and even research papers on the so-called "AI ethics".

int_19h 7y ago
Such a model is objectively biased regardless of where and how it's applied - it prefers X over Y, and that's bias by definition. The hard question here is deciding whether any given bias is ethical or not.

rcthompson 7y ago

No, it's not the same as overfitting. It's nothing to do with the algorithm at all. The problem is with the training data, which contains patterns induced by the way the data was collected rather than the reality of what you want to classify. For example, imagine that you wanted to train an algorithm to distinguish photos of dogs from photos humans. So you collect a bunch of photos of both dogs and humans and use them to train a classifier. You do all the proper cross-validation, bootstrapping, etc. to ensure that you are not overfitting, and you get really good results. Then, looking at the mis-classifications, you notice something: all the photos that are taken looking at an angle down toward the ground are classified as dog photos, and all the photos taken looking straight ahead are classified as human photos. It turns out that in your training set, most of the dog photos are taken at a downward angle while must of the human photos are taken facing straight ahead, because humans are taller than dogs, and your machine learning algorithm identified this feature as the most reliable way to distinguish the two groups of photos in your training set. In this hypothetical example, no overfitting occurred. The difference in photo angles is a real difference in the training sets that you provided to the algorithm, and the algorithm did its job and correctly identified this difference between the two groups of photos as a reliable predictor. The problem is that your training set has a variable (photo angle) that is highly correlated with what you want to classify (species). This is considered an unwanted bias (and not a reliable indicator) because the correlation is caused by the means of data collection (most photos are taken from human head height) and has nothing to do with the subject of the photos.

CardenB 7y ago
I think you’re arguing semantics a bit. What you’re saying checks out, but one could say that overfitting was occurring but the test dataset distribution was not wide enough to catch it.
- etbebl 7y ago
  As I understand it, if it's overfitting, testing on a second random sample gathered in the same way as the training set should degrade performance. That's not the case here. (Though maybe the term as used in industry is less strict.)

haberman 7y ago

The test for overfitting is whether the model holds up for data outside the training set. If the biases are consistent with other real-world data, then it's not overfitting.

deleted 7y ago

[deleted]