4 ms·

The line between fit and overfit is arbitrary, though. What is happening is that the systems are revealing unconscious biases present in the more traditional sy

by bsanr2 7y ago

The line between fit and overfit is arbitrary, though. What is happening is that the systems are revealing unconscious biases present in the more traditional systems the AI/ML are supposed to replicate, build upon, or speed up. If the results are odious to us, it should be impetus to critically analyze not only the AI/ML systems, but also the underlying assumptions that they're built on. Instead, developers become defensive and cage-y about their processes. If you don't want systems to have disparate impact, you have to be adamant about it in your design. If you think society is better off with systems that reflect preexisting biases, then fine, but be ready for the backlash. At the end of the day, it really is up to what humans want to do with themselves. It's an opportunity to be truer to our intent, not a bug to be covered up.

im3w1l 7y ago
> The line between fit and overfit is arbitrary, though. What is happening is that the systems are revealing unconscious biases present in the more traditional systems the AI/ML are supposed to replicate, build upon, or speed up. When you say unconscious bias, you are kind of implying that the model learns something that is false. But more often it's the case that the model learns something true that we don't want it to learn. That's what makes the problem so hard, you are trying to hide the truth from a system you only half-understand processing data you only half-understand. There is a big risk the truth slips through the cracks if you aren't careful.
- wan23 7y ago
  > When you say unconscious bias, you are kind of implying that the model learns something that is false. It's more that the model learns something that is undesirable. It could be the case, for example, that the true thing that the AI learns is that your resume screening process tends to exclude women. This is true, sure, but it could lead to the undesirable outcome where the presence of a female name on a resume might be weighted heavily against the candidate.
- bsanr2 7y ago
  Well, in many cases, they are indeed learning something false (or, to be more charitable, something that is not sufficiently true). For example, a facial recognition system that has trouble detecting dark-skinned faces. Black and brown people clearly have faces, but because the systems were not trained on their facial data, and the capability to adequately adjust exposure or other aspects of the sensor stack was not built into the system, it provides an objectively wrong solution given perfectly reasonable inputs. I agree that this process is one of resolving blind spots, but I disagree that the blind spots are simply areas devoid of light. AI/ML systems are frequently employed to augment or stand in for human perception, which is known to be necessarily incomplete with respect to reality. In other words, they can learn things that seem true to us but that are false from another perspective, or undesirable once exposed. What's exciting about them is that they provide an opportunity to interrogate the flaws in our individual perception with a systematized observation and analysis, in a much more sophisticated manner than in the past. But fulfilling that potential requires humility.
  - im3w1l 7y ago
    Yeah that's a good point, about having sufficient examples in the training set. I think it's not as simple as that though, several years after that made headlines people have still not been able to close the face recognition accuracy gap. Experts are debating different explanations for this. https://www.wired.com/story/best-algorithms-struggle-recognize-black-faces-equally/ https://www.wired.com/story/best-algorithms-struggle-recogni...
    - bsanr2 7y ago
      That wasn't entirely my point, however. The training set was only a part of the problem, which in its totality was that the developers failed to consider that the functioning of their system could be affected by the biases they didn't even know they had, in this case of being more likely to see light-skinned faces as belonging to a human. Their failure was not just in lacking diverse training sets, but diverse QA, or at least QA looking for those blindspots which eventually became evident. So, correct, it's not as simple as having "sufficient" data.
- loopz 7y ago
  You're correct. A statistical model could uncover "an inconvenient truth", and even make us question our biases and become more adamant in finding better solutions. However, ML is often sold as a solution for generating outcomes, not for finding truths, wether they be true or false. The distinction is huge. No matter what humans do, they will reap what they sow. Consequences and outcome matter more than "truth" (which may be in the eye and competence of the beholder).

by bsanr2 7y ago

im3w1l 7y ago

> The line between fit and overfit is arbitrary, though. What is happening is that the systems are revealing unconscious biases present in the more traditional systems the AI/ML are supposed to replicate, build upon, or speed up. When you say unconscious bias, you are kind of implying that the model learns something that is false. But more often it's the case that the model learns something true that we don't want it to learn. That's what makes the problem so hard, you are trying to hide the truth from a system you only half-understand processing data you only half-understand. There is a big risk the truth slips through the cracks if you aren't careful.

wan23 7y ago
> When you say unconscious bias, you are kind of implying that the model learns something that is false. It's more that the model learns something that is undesirable. It could be the case, for example, that the true thing that the AI learns is that your resume screening process tends to exclude women. This is true, sure, but it could lead to the undesirable outcome where the presence of a female name on a resume might be weighted heavily against the candidate.
bsanr2 7y ago
Well, in many cases, they are indeed learning something false (or, to be more charitable, something that is not sufficiently true). For example, a facial recognition system that has trouble detecting dark-skinned faces. Black and brown people clearly have faces, but because the systems were not trained on their facial data, and the capability to adequately adjust exposure or other aspects of the sensor stack was not built into the system, it provides an objectively wrong solution given perfectly reasonable inputs. I agree that this process is one of resolving blind spots, but I disagree that the blind spots are simply areas devoid of light. AI/ML systems are frequently employed to augment or stand in for human perception, which is known to be necessarily incomplete with respect to reality. In other words, they can learn things that seem true to us but that are false from another perspective, or undesirable once exposed. What's exciting about them is that they provide an opportunity to interrogate the flaws in our individual perception with a systematized observation and analysis, in a much more sophisticated manner than in the past. But fulfilling that potential requires humility.
- im3w1l 7y ago
  Yeah that's a good point, about having sufficient examples in the training set. I think it's not as simple as that though, several years after that made headlines people have still not been able to close the face recognition accuracy gap. Experts are debating different explanations for this. https://www.wired.com/story/best-algorithms-struggle-recognize-black-faces-equally/ https://www.wired.com/story/best-algorithms-struggle-recogni...
  - bsanr2 7y ago
    That wasn't entirely my point, however. The training set was only a part of the problem, which in its totality was that the developers failed to consider that the functioning of their system could be affected by the biases they didn't even know they had, in this case of being more likely to see light-skinned faces as belonging to a human. Their failure was not just in lacking diverse training sets, but diverse QA, or at least QA looking for those blindspots which eventually became evident. So, correct, it's not as simple as having "sufficient" data.
loopz 7y ago
You're correct. A statistical model could uncover "an inconvenient truth", and even make us question our biases and become more adamant in finding better solutions. However, ML is often sold as a solution for generating outcomes, not for finding truths, wether they be true or false. The distinction is huge. No matter what humans do, they will reap what they sow. Consequences and outcome matter more than "truth" (which may be in the eye and competence of the beholder).