4 ms·
One point that I think is under-discussed in the AI bias area: While it is true that using an algorithmic process to select candidates may introduce discrimina
by theptip 4y ago
One point that I think is under-discussed in the AI bias area:
While it is true that using an algorithmic process to select candidates may introduce discrimination against protected groups, it seems to me that it should be much easier to detect and prove than with previous processes with human judgement in the loop.
You can just subpoena the algorithm and then feed test data to it, and make observations. Even feed synthetic data like swapping in “stereotypically black” names for real resumes of other races, or in this case adding “uses a wheelchair” to a resume. (Of course in practice it’s more complex but hopefully this makes the point.)
With a human, you can’t really do an A/B test to determine if they would have prioritized a candidate if they hadn’t included some signal; it’s really easy to rationalize away discrimination at the margins.
So while most AI/ML developers are not currently strapping their models to a discrimination-tester, I think the end-state could be much better when they do.
(I think a concrete solution would be to regulate these models to require a certification with some standardized test framework to show that developers have actually attempted to control these potential sources of bias. Google has done some good work in this area: https://ai.google/responsibilities/responsible-ai-practices/?category=fairness https://ai.google/responsibilities/responsible-ai-practices/... - though there is nothing stopping model-sellers from self-regulating and publishing this testing first, to try to get ahead of formal regulation.)
- TimPC 4y agoI don’t think it’s adequate to attempt to prevent discrimination. Discrimination is core to our fundamental human rights. It’s necessary to succeed at preventing discrimination. “We applied best practices in the field to limit discrimination” should not be an adequate legal defence if the model can be shown to discriminate. To clarify further, just because you tried to prevent discrimination doesn’t mean you should be off the hook for the material harms of discrimination to a specific individual. Otherwise people don’t have a right to be protected against discrimination they only have a right to people ‘trying’ to prevent discrimination. We shouldn’t want to weaken rights that much even if it means we have to be cautious in how we adopt new technologies.
- MontyCarloHall 4y agoI agree that discrimination would be a lot easier to objectively prove after the fact, but it also would be far easier to occur in the first place, since many hiring managers would blindly "trust the AI" without a second thought.
- fshbbdssbbgdd 4y agoFrom my experience working on projects where we trained models, usually it’s obviously completely broken the first attempt and requires a lot of iteration to get to a decent state. “Trust the AI” is not a phrase anyone involved would utter. It’s more like: trust that it is wrong for any edge case we didn’t discover yet. Can we constrain the possibility space any more?
- pc86 4y agoMost hiring managers wouldn't make it to the end of the phrase "constrain the possibility space"
- MichaelBurge 4y ago"Trust the AI" could mean uploading a resume to a website and getting a "candidate score" from somebody else's model. Because I'll tell you, there's millions of landlords and they blindly trust FICO when screening candidates. Maybe not as the only signal, but they do trust it without testing it for edge cases.
- theptip 4y agoDefinitely could be so, particularly in these early days where frameworks and best-practices are very immature. Inasmuch as you think this is likely, I suspect you should favor regulation of algorithmic processes instead of voluntary industry best-practices.
- indymike 4y agoThe problem with AI is that when it does make discriminatory decisions on hiring, is that it does so systematically and mechanically. Incidentally, systematic and discrimination are two words you never want to see consecutively on a letter from the EEOC or OFCCP.
- TimPC 4y agoThe reason you never want to see those words together is that isolated discrimination may result in a single lawsuit but systemic discrimination is a basis for class action.
- slg 4y ago>With a human, you can’t really do an A/B test to determine if they would have prioritized a candidate if they hadn’t included some signal; it’s really easy to rationalize away discrimination at the margins. Which is part of the reason that discrimination doesn't have to be intentional for it to be punishable. This is a concept known as "disparate impact". The Supreme Court has issued decisions[1] that a policy which negatively impacts a protected class and has no justifiable business related reason for existing can be deemed discrimination regardless of the motivations behind that policy. [1] - https://en.wikipedia.org/wiki/Griggs_v._Duke_Power_Co https://en.wikipedia.org/wiki/Griggs_v._Duke_Power_Co.
- Ferrotin 4y agoEverything has a disparate impact, so now everything is illegal.
- pc86 4y agoMight I suggest: https://www.merriam-webster.com/dictionary/term%20of%20art https://www.merriam-webster.com/dictionary/term%20of%20art
- Ferrotin 4y agoNo, you’re wrong here, it’s not a term of art.
- pc86 4y agoIt sure looks like it is. > Disparate impact in United States labor law refers to ... https://en.wikipedia.org/wiki/Disparate_impact https://en.wikipedia.org/wiki/Disparate_impact
- Ferrotin 4y agoYour link is evidence for my side. It uses the plain definition. The plain meaning of the words.
- mjburgess 4y agoIt's under-discussed as with any discussion of an empirical study of ML systems, ie., treating them as targets of analysis. As soon as you do this, they're revealed to exploit only statistical coincidences and highly fragile heuristics embedded within the data provided. And likewise, pretty universally discriminatory when human data is inovlved
- Manuel_D 4y ago> With a human, you can’t really do an A/B test to determine if they would have prioritized a candidate if they hadn’t included some signal; it’s really easy to rationalize away discrimination at the margins. Not for individual candidates, no. But you can introduce a parallel anonymized interview process and compare the results.
- TimPC 4y agoActually you kind of can't. You don't have a legal basis for forcing the company to run that experiment.
- deleted 4y ago[deleted]
- TimPC 4y agoThere is a very real danger of models being biased in a way that doesn't show up when you apply these crude hacks to inputs. It seems to me we have to be much more deliberate, much more analytical, and much more thorough in testing models if we want to substantially reduce or even eliminate discrimination. Yes, you can A/B test the model if you can design reasonable experiments. You still don't have the general discrimination test because you have to define what a reasonable input distribution and what reasonable outputs are. If an employer is looking to hire an engineer with a CS degree from a top-tier university, and they use an AI model to evaluate resumes and it returns a number of successes on black people very similar to the population distribution of graduates from those programs is the model discriminatory? There are still hard problems here because any natural baseline you use for a model may in fact be wrong and designing a reasonable distribution of input data is almost impossibly hard as well.
- deleted 4y ago[deleted]
- theptip 4y agoYes, in practice it’s actually way more complex than I gestured at. The Google bias toolkit I linked does discuss in much more detail, but I am not a data scientist and haven’t used it; I’d be interested in expert opinions. (They also have some very good non-technical articles discussing the general problems of defining “fairness” in the first place.)