3 ms·

Those things are bad, and are worth thinking about, but right now we have more immediate concerns. AI systems are currently being used for bail eligibility, lik

Those things are bad, and are worth thinking about, but right now we have more immediate concerns. AI systems are currently being used for bail eligibility, likelihood of recurrence, mortgage approval, and hiring. These are things that literally change people's lives on an individual level, and the systems being used are currently indecipherable black boxes that return a result with no possibility of meaningful appeal. There's a sense of starting small - redlining is illegal. No company wants to redline - it cuts into their profits and dings their compliance scores. So they're willing to work with us. Once we get that right, then, maybe we can start dealing with the cases where there aren't millions of dollars of funding assisting with making the tech more ethical -- and then, cases where it may actually be working against the money.

valesco 6y ago
European law foresees those exact cases and IIRC prohibits automated treatment of personal information that leads to decisions that affects someone’s life, if no human takes the final decision and if the specific elements for the decision cannot be explained. In my opinion this forbids black boxes.
- cauthon 6y ago
  I think you’re referring to GDPR article 22, which prohibits decisions based solely on automated processing but imposes no constraints on the transparency of such processing. [1] [1] https://gdpr-text.com/read/article-22/ https://gdpr-text.com/read/article-22/
- dgellow 6y ago
  What about credit score systems like the German SCHUFA? https://en.wikipedia.org/wiki/Schufa https://en.wikipedia.org/wiki/Schufa. They have their proprietary algorithm to calculate a credit score, for example you almost cannot find an apartment without presenting your SCHUFA review and there is now way to know how their score is calculated (it has even been shown in the past that they use incorrect or outdated data for their calculation). That’s the definition of a black box in my book.
visarga 6y ago
> AI systems are currently being used for bail eligibility, likelihood of recurrence, mortgage approval, and hiring. Is there an analysis comparing the automated to manual approaches? I mean, people are biased, too. The models are more systematically biased, but can also be more systematically evaluated and de-biased. Which is better, the old or the imperfect new?
- freeone3000 6y ago
  Yes; automated approaches amplify bias[0]. Moreover, automated approaches launder bias; in the general population, people view computers and algorithms as "unbiased", so imagine biased results to be more accurate than humans, because a computer did them. 0: https://arxiv.org/abs/1809.07842 https://arxiv.org/abs/1809.07842 2: https://arxiv.org/abs/2004.07213 https://arxiv.org/abs/2004.07213 also has an overview of our current combative systems.

valesco 6y ago

European law foresees those exact cases and IIRC prohibits automated treatment of personal information that leads to decisions that affects someone’s life, if no human takes the final decision and if the specific elements for the decision cannot be explained. In my opinion this forbids black boxes.

cauthon 6y ago
I think you’re referring to GDPR article 22, which prohibits decisions based solely on automated processing but imposes no constraints on the transparency of such processing. [1] [1] https://gdpr-text.com/read/article-22/ https://gdpr-text.com/read/article-22/
dgellow 6y ago
What about credit score systems like the German SCHUFA? https://en.wikipedia.org/wiki/Schufa https://en.wikipedia.org/wiki/Schufa. They have their proprietary algorithm to calculate a credit score, for example you almost cannot find an apartment without presenting your SCHUFA review and there is now way to know how their score is calculated (it has even been shown in the past that they use incorrect or outdated data for their calculation). That’s the definition of a black box in my book.

visarga 6y ago

> AI systems are currently being used for bail eligibility, likelihood of recurrence, mortgage approval, and hiring. Is there an analysis comparing the automated to manual approaches? I mean, people are biased, too. The models are more systematically biased, but can also be more systematically evaluated and de-biased. Which is better, the old or the imperfect new?

freeone3000 6y ago
Yes; automated approaches amplify bias[0]. Moreover, automated approaches launder bias; in the general population, people view computers and algorithms as "unbiased", so imagine biased results to be more accurate than humans, because a computer did them. 0: https://arxiv.org/abs/1809.07842 https://arxiv.org/abs/1809.07842 2: https://arxiv.org/abs/2004.07213 https://arxiv.org/abs/2004.07213 also has an overview of our current combative systems.