2 ms·
One of the things I'm rather worried about here is AI's ability to learn on the most unlikely (to human cognition) features. Take the image recognizer that find
by geofft 9y ago
One of the things I'm rather worried about here is AI's ability to learn on the most unlikely (to human cognition) features. Take the image recognizer that finds "sheep" by seeing pastures where it thinks sheep might graze http://aiweirdness.com/post/172894792687/when-algorithms-surprise-us http://aiweirdness.com/post/172894792687/when-algorithms-sur... , or take any of the adversarial perturbing examples like https://blog.openai.com/robust-adversarial-inputs/ https://blog.openai.com/robust-adversarial-inputs/ .
You're not allowed to discriminate on race in housing. But is an AI that determines your creditworthiness for mortgages allowed to discriminate on what you eat, where you go to church, what your favorite music is, etc.? Maybe it doesn't have that data, but it will have one level higher - what store credit cards you have and how much you use them and where you opened them.
If you train an AI on a segregated city with a history of actively discriminatory citizens, where the few people of race X who moved into a not-race-X neighborhood got harassed out and sold before they paid off their mortgages, how easily will the AI conclude that people born in certain neighborhoods are more likely to pay off their mortgage if they avoid certain other neighborhoods?
Is that illegal? (My guess is there's no way to prove to a court, to the court's usual standards, that the AI happened to learn the city's racial tensions.) Should it be illegal, if outright racial discrimination is illegal?