13 ms·
Employers’ Use of AI Tools Can Violate the Americans with Disabilities Act
- adolph 5y agoI wonder if this government guidance focuses on imperfections in products that on the whole may be a significant improvement over biases in traditional human screening.
- jaqalopes 5y agoCurrent headline is a bit misleading, the point of the article as made clear in the very first paragraph is that this is about AI hiring tools causing potential discrimination. This has nothing to do with AI workers somehow replacing disabled humans, which is what it sounds like.
- iso1631 5y agoThe first paragraph is exactly what I expected from the headline, ever since the amazon AI gender discrimination story a few years back. https://www.theguardian.com/technology/2018/oct/10/amazon-hiring-ai-gender-bias-recruiting-engine https://www.theguardian.com/technology/2018/oct/10/amazon-hi...
- deleted 5y ago[deleted]
- sneak 5y agohttp://www.tnellen.com/cybereng/harrison.html http://www.tnellen.com/cybereng/harrison.html
- alicesreflexion 5y agoThis doesn't matter until someone tries suing them for it, right? And as I understand it, you don't really have a case without evidence that the hiring algorithm is discriminating against people with disabilities. How would an individual even begin to gather that evidence?
- marian_ivanco 5y agoI am not sure, but if I remember correctly employer must prove they are not discriminating. And just because they are using AI they are not immune to litigation.
- danarmak 5y agoHow can the employer prove a negative? At most I imagine the plaintiff is allowed to do discovery, and then has to prove positive discrimination based on that.
- radu_floricica 5y ago
- vajrabum 5y agoIf you read the document again (?) maybe you'll see it's not about proving a negative. Instead, it's a standard of due care. Did you check whether using some particular tool illegally discriminates and document that consideration? From the document itself: "Clarifies that, when designing or choosing technological tools, employers must consider how their tools could impact different disabilities; Explains employers’ obligations under the ADA when using algorithmic decision-making tools, including when an employer must provide a reasonable accommodation;"
- HWR_14 5y agoIf it's a civil case, it's just the preponderance of the evidence. The jury just has to decide who they think is more likely to be correct.
- rascul 5y ago> I am not sure, but if I remember correctly employer must prove they are not discriminating. That seems backwards, at least in the US.
- dsr_ 5y agoThe process of gathering evidence after the suit has started is called discovery. There are three major kinds of evidence that would be useful here. Most useful but least likely: email inside the company in which someone says "make sure that this doesn't select too many people with disabilities" or "it's fine that the system isn't selecting people with disabilities, carry on". Useful and very likely: prima facie evidence that the software doesn't make necessary reasonable accomodations - a video captcha without an audio alternative, things like that. Fairly useful and of moderate likelihood: statistical evidence that whatever the company said or did, it has the effect of unfairly rejecting applicants with disabilities.
- timcavel 5y ago
- supergeek133 5y agoThis sort of reminds me of the story about an HR algorithm that ended up being discriminatory because it was trained using existing/past hiring data.. so it was biased toward white men. Was it Amazon? Anyway, this feels different to me, IIRC you can't ask disability related questions in hiring aside from the "self identify" types at the end? So how would a ML model find applicants with any kind of disability unless it was freely volunteered in a resume/CV? Or is that the advisory? "Don't do this?"
- padolsey 5y agohow would a ML model find applicants with any kind of disability unless it was freely volunteered in a resume/CV? A few off the top of my head: (1) Signals gained from ways that a CV is formatted or written (e.g. indicating dyslexia or other neurological variances, especially those comorbid with other physiological disabilities) (2) If a CV reports short tenure at companies with long breaks in between (e.g. chronic illnesses or flare-ups leading to burnout or medical leave) (3) There are probably many unintuitive correlates irt interests, roles acquired, skillsets. Consider what experiences, institutions, skillsets and roles are more or less accessible to disabled folk than others. (4) Most importantly: Disability is associated with lower education and lower economic opportunity, therefore supposed markers of success ("merit") in CVs may only reflect existing societal inequities. * * This is one of the reasons meritocratic "blind" hiring processes are not as equitable as they might seem; they can reflect + re-entrench the current inequitable distribution of "merit".
- MontyCarloHall 5y ago>If a CV reports short tenure at companies with long breaks in between (e.g. chronic illnesses or flare-ups leading to burnout or medical leave) This is a case where it may benefit a candidate to disclose any disabilities leading to such an erratic employment pattern. I don’t proceed with candidates who cannot explain excessively frequent job hops because it signals that they can’t hold a job due to factors I’d want to avoid hiring, like incompetence or a difficult personality. It’s a totally different matter if the candidate justified their erratic employment due to past medical issues that have since been treated.
- charcircuit 5y agoHow would this even happen. Why would people put their disabilities on their resume?
- michaelt 5y agoSkills: Fluent in American Sign Language. High School: Florida School for the Deaf and the Blind. Other Experience: President of Yale Disability Awareness Club (2009-2011).
- hansvm 5y agoThe AI learns proxy signals. Name, work experience, skills (e.g., an emphasis on A11Y) ... all have some predictive power for gender, for some sorts of disabilities, .... You can fix the problem by going nuclear and omitting any sort of data that could serve as a proxy for the discriminatory signals, but it's possible to explicitly feed the discriminatory signals into the model and enforce that no combination of other data amounting to knowledge about them can influence the model's predictions. There was a great paper floating around for a bit about how you could actually manage that as a data augmentation step for broad classes of models (constructing a new data set which removed implicit biases assuming certain mild constraints on the model being trained on it). I'm having a bit of trouble finding the original while on mobile, but they described the problem as equivalent to "database reconstruction" in case that helps narrow down your search.
- zw123456 5y agoOh, thank you, this was the question floating in my head as well, this explains it perfectly.
- treeman79 5y agoBecause you can’t always hide it. Nothing like giving a presentation on a whiteboard when all of a sudden your writing turns to gibberish. People have a limited tolerance. Then they start telling you to take care of yourself and strongly encouraging you to leave. It’s why I switched to remote work before the pandemic. If a bad episode starts up I can cover for it much more easily.
- annoyingnoob 5y agoI'll bet the same AI used in hiring decisions could also be biased against older workers.
- tj_h 5y ago"Using AI tools for hiring"...this is when i like to remind myself that google, basically at some point in the last 12-24 months was like "OH CRAP, we forgot to tell our robots about black people!". Like, I'm not saying google is at the forefront of ML - maybe it is, but it sure as hell is out in front somewhere and more to the point most companies are likely not gonna be using cutting edge technology for this stuff. EVEN GOOGLE, admitted their ML for images is poorly optimized for POC's, i hate to think what some random ML algorithm used by company X thinks about differently abled peoples
- temp8964 5y agoThis is unreasonable generalization. Dark skin does have direct impact on image processing. Nothing like this exists in hiring.
- lmkg 5y agoThere are companies selling products which screen hiring candidates based on video of them talking. Ostensibly for determining personality traits or whatever. So yes, this literally exists in hiring.
- htrp 5y agoMate. you ever hear about amazon's hiring ai?
- smiley1437 5y agoWhile image processing dark skin may not be germane to AIs doing hiring, the idea that unintentional discrimination from ML models could occur in the context of hiring is certainly worth considering and I believe it's the entire point of the technical assistance document released today.
- httpsterio 5y agoRacial bias does exist in ML based image recognition tools, there's a plethora of evidence to show that. https://www.theverge.com/2019/1/25/18197137/amazon-rekognition-facial-recognition-bias-race-gender https://www.theverge.com/2019/1/25/18197137/amazon-rekogniti... https://www.google.com/amp/s/www.wired.com/story/best-algorithms-struggle-recognize-black-faces-equally/amp https://www.google.com/amp/s/www.wired.com/story/best-algori... https://algorithmwatch.org/en/google-vision-racism/ https://algorithmwatch.org/en/google-vision-racism/ https://time.com/5520558/artificial-intelligence-racial-gender-bias/ https://time.com/5520558/artificial-intelligence-racial-gend...
- jijji 5y agoI encountered this recently on Facebook Marketplace. I post ads for houses for rent, and the ads say "no pets". This has been fine for 20+ years on craigslist, but on facebook marketplace the minute some guy writes that he "has a service animal" and you don't respond the right way, your ad gets blocked/banned.... You basically have to accept these people even though the law allows you to prohibit animals, service animals must be accepted otherwise you violate the ADA. I knew this guy when I was living in Sunnyvale he had a cat that was a registered service animal, and he would get kicked out of every hotel he went to, because they dont allow animals/pets, and then he would sue the owner under ADA laws and collect ~40k from each hotel owner. Its a real racket.
- deleted 5y ago[deleted]
- judge2020 5y ago> I knew this guy when I was living in Sunnyvale he had a cat that was a registered service animal, > Beginning on March 15, 2011, only dogs are recognized as service animals under titles II and III of the ADA. https://www.ada.gov/service_animals_2010.htm https://www.ada.gov/service_animals_2010.htm
- deleted 5y ago[deleted]
- golemotron 5y agoIsn't the entire point of an AI tool 'discrimination'?
- theptip 5y agoLike many words in the English language, “discrimination” has multiple meanings. From Webster: 1. The act of discriminating. 2. The ability or power to see or make fine distinctions; discernment. 3. Treatment or consideration based on class or category, such as race or gender, rather than individual merit; partiality or prejudice. You are talking about 2. The article is talking about 3. 3. is illegal in hiring. 2. is not.
- golemotron 5y agoIf you make a decision based on 2, you are doing 3. It's just that simple. 2 creates categories implicitly.
- theptip 5y agoThat is not how the courts interpret it.
- golemotron 5y agoIf the ultimate standard is disparate impact, that's where it goes.
- pipingdog 5y agoML is a technique for discovering and amplifying bias. Applying ML to hiring shows a profound lack of awareness of both ML and HR. Especially using previous hiring decisions as a training set. Like using a chainsaw to fasten trees to the ground.
- theptip 5y agoOne point that I think is under-discussed in the AI bias area: While it is true that using an algorithmic process to select candidates may introduce discrimination against protected groups, it seems to me that it should be much easier to detect and prove than with previous processes with human judgement in the loop. You can just subpoena the algorithm and then feed test data to it, and make observations. Even feed synthetic data like swapping in “stereotypically black” names for real resumes of other races, or in this case adding “uses a wheelchair” to a resume. (Of course in practice it’s more complex but hopefully this makes the point.) With a human, you can’t really do an A/B test to determine if they would have prioritized a candidate if they hadn’t included some signal; it’s really easy to rationalize away discrimination at the margins. So while most AI/ML developers are not currently strapping their models to a discrimination-tester, I think the end-state could be much better when they do. (I think a concrete solution would be to regulate these models to require a certification with some standardized test framework to show that developers have actually attempted to control these potential sources of bias. Google has done some good work in this area: https://ai.google/responsibilities/responsible-ai-practices/?category=fairness https://ai.google/responsibilities/responsible-ai-practices/... - though there is nothing stopping model-sellers from self-regulating and publishing this testing first, to try to get ahead of formal regulation.)
- TimPC 5y agoI don’t think it’s adequate to attempt to prevent discrimination. Discrimination is core to our fundamental human rights. It’s necessary to succeed at preventing discrimination. “We applied best practices in the field to limit discrimination” should not be an adequate legal defence if the model can be shown to discriminate. To clarify further, just because you tried to prevent discrimination doesn’t mean you should be off the hook for the material harms of discrimination to a specific individual. Otherwise people don’t have a right to be protected against discrimination they only have a right to people ‘trying’ to prevent discrimination. We shouldn’t want to weaken rights that much even if it means we have to be cautious in how we adopt new technologies.
- MontyCarloHall 5y agoI agree that discrimination would be a lot easier to objectively prove after the fact, but it also would be far easier to occur in the first place, since many hiring managers would blindly "trust the AI" without a second thought.
- TimPC 5y agoI think the general problem is the law says certain correlations are fair to use and others are not. If you can prove the AI model has no way to separate out which is which you have a fairly sizeable amount of evidence the AI is discriminating. Likely enough evidence for a civil case. Usually showing that input data is biased in some way or contains a potentially bad field will result in winning a discrimination case. If neither side can conclusively prove what the model is doing but the plaintiff shows it was trained on data that allows for discrimination and the model is designed to learn patterns in its training data then the defendant is on the hook for showing the model is unbiased. For the most part people design input data uncritically and some of the fields allow for discrimination.
- thaumasiotes 5y ago> but the plaintiff shows it was trained on data that allows for discrimination That's all data; there would be no need to show anything. There was a paper a while ago by a team of doctors who wanted to use classifiers on X-ray images and FREAKED OUT when they realized that the first thing the classifier did was categorize every image by the race of the patient. As they note, it will always do this regardless of whether the race of the patient is present in the input data. (Because obviously, the race of the patient is part of the information conveyed by the structure of their body, which is what an X-ray shows.)
- theptip 5y agoThe linked article gives some examples that I think are very useful clarifications: https://www.eeoc.gov/tips-workers-americans-disabilities-act-and-use-software-algorithms-and-artificial-intelligence https://www.eeoc.gov/tips-workers-americans-disabilities-act... > The format of the employment test can screen out people with disabilities [for example:] A job application requires a timed math test using a keyboard. Angela has severe arthritis and cannot type quickly. > The scoring of the test can screen out people with disabilities [for example:] An employer uses a computer program to test “problem-solving ability” based on speech patterns for a promotion. Sasha meets the requirements for the promotion. Sasha stutters so their speech patterns do not match what the computer program expects. Interestingly, I think the second one is problematic for common software interview practices. If your candidate asked for an accommodation (say, no live rapid-fire coding) due to a recognized medical condition, you would be legally required to provide it. This request hasn’t come up for me in all the (startup) hiring I’ve done, but it could be tough to honor this request fairly on short notice, so worth thinking about in advance.
- alar44 5y agoWhat if the job requires you to type quickly? Why would someone with arthritis even want a job where you have to type quickly? Is that really discrimination or is that the candidate simply not being able to perform the job?
- TimPC 5y agoIf the job actually requires typing quickly like a Court Recorder then there is a basis to require typing quickly. If the job doesn’t actually require it, like for example a programmer then enforcing the requirement anyway is discrimination.
- xyzzyz 5y agoMost jobs that involve typing benefit from being able to type quickly. For example, I am a frequent customer of U-Haul. I learned to not use the branch that’s closest to me, because some employees there are really slow with computers, which makes checking out equipment very slow, and frequently results in a long line of waiting customers. Driving 5 extra minutes saves me 20 minutes of waiting for employees to type in everything and click through the system. And this is freaking uhaul. If you’re a software engineer, slow typing is also a productivity drain: a 3 minutes email becomes 6 minutes one, a 20 minutes slack conversation becomes 30 minutes etc. It all adds up.
- bombcar 5y agoEven the very requirement to "apply online" has been quite effective at making it very difficult for a sub-section of the working population to succeed at applying. There are many (and I know quite a few) people who are quite capable at their jobs and entirely computer-ineffective. As they're forced more and more to deal with confusing two-factor requirements and other computer-related things that we're just "used" to, they get discouraged and give up. For now you can often help them fill it out, but at some point that's going to be unwieldy or insufficient.
- david-cako 5y agofor instance, I find mass surveillance intolerable and it makes me completely uninterested in my work.
- frankfrankfrank 5y agoIt is a bit of an aside, but not only do I find it interesting that 1) this issue is approaching an interesting nexus of computer based efficiency and human "adjustments" (I will just call them) like the ADA that are intentionally and even deliberately inefficient; and 2) that the efficiency and centralization based sector of computer "sciences"/development is so replete with extremely contrary types that demand all manner of exceptions, exemptions, and make special pleadings. I find it all very interestingly paradoxical, regardless of everything else.
- umvi 5y agoIs it actually possible to hire someone without some level of discrimination involved? Seems like this ideal world where candidates are hired purely on technical ability or merits without regard to any other aspects of their life is impossible. For example, if I were hiring a programmer, and the programmer was technically competent but spoke with such a thick accent that I couldn't understand them very well, I'd be tempted to pass on that candidate even though they meet all the job requirements. And if it happened every time I interviewed someone from that particular region, I'd probably develop a bias against similar future candidates.
- throwaway09223 5y agoNo, it's not possible. Humans have all kinds of inherent biases. The big difference is we can prove the bias in an AI. It's a very interesting curveball when it comes to demonstrating liability in the choice making process.
- Broken_Hippo 5y agoProbably not simply because we are human, but we can minimize some of it. You wouldn't screen out a person who cannot speak or who cannot speak clearly due to a disability of some sort. You'd use a different method of communication as would everyone else and it could really be the same for them. On the other hand, if communication was clearly impossible and/or they needed to be understood by the public (customers), the accent may very well mean they cannot do the job and not in the scope of things to teach someone like you can teach expectations about customer service.
- egberts1 5y agoIt is a strange twist but when filling out the many prescreening questions at Google, I was immediately sent an email asking for an interview. Sadly, they insist that they would only call me at an agreed time and not take any incoming call from me, even at their agreed upon timeframe. Why is this an important stance of AI bias against people with disability? I need to line up a voice interpreter who would do American Sign Language so I can understand what a group of Google interviewers would be saying. And voice interpreters requires a 3-way call that I can help setup. I am capable of understanding them perfectly through my lipreading given that I lost all high frequency discriminator due to a childhood high fever. I cannot tell the difference between B, V, P, D, Z, or E, but a lipreading skill can nails them all. I am an excellent speaker of the English language given being technically and governmentally assigned as Deaf. So, have I been wronged? I still think so … to this day.
- ordiel 5y agoIMHO people are not cogs, you cannot masure them all with only a few parameters, we are so kuch more complex than that, and there should be a absolute ban in the use of AI and software tools to measure (and spy) employees, only allowing those softare tools that facilitate capturing and keeping record of those observations perfomed by other humans. I am not saying we humans are perfect to handle that complexity, hence when humans are the ones behind such desition making it is rarely done by a single person
- lucidmote 5y agoI've been disabled since 17, finally got approved for SSDI in 2017 after almost two decades of struggling to work and make ends meet. I can say definitively that employment in the US does not follow the guidelines as they exist and those guidelines are too weighted in the favor of the employer in the first place. A simple example of this is that often part-time hours would allow me to continue working and there is nothing in the ADA/EEO or FMLA that guarantees a worker's right to keep their job at part-time indefinitely. The obligation to perform to the level of a typical worker is squarely on the shoulders of the disabled. Good luck finding an employer generous enough to deal with all of your needs. 85% of the people with my diagnosis are unemployed despite the fact most of us want to work. The algorithms will probably help make sure it's more like 90-95% of us. But at least the stakeholder involvement process makes people feel like they have a voice even as they are categorically excluded from ever being fully-functional human beings.