3 ms·

So... the obvious Serious Issues here are privacy and child pronography. This blog though, is about an issue I don't know how to define concisely. It's about g

So... the obvious Serious Issues here are privacy and child pronography. This blog though, is about an issue I don't know how to define concisely. It's about getting labeled, for whatever reason as "high risk." I flagged some paypal thing years ago, when I moved countries. It was too much trouble to sort out, so I abandoned my $20 and moved on. Paypal has me in a no recourse bucket. Most transactions run through such risk assessments. False positives are tolerated. Child pronography scanning, as this post lays out, will have similar issues. We're already used to credit scores and insurance behaving like this. I'm not diminishing the privacy aspects, but this neokafka world of automated, risk accessing bureaucracy is also a world we're constantly walking towards. It's also bad.

tornato7 5y ago
Basically, these companies know that false positives happen. To them, it's just a statistic - they can afford to wrongly ban 0.1% of customers in the name of preventing abuse. But for the person they're wrongly banning, that's not so fun. I'm worried this will apply to more seriously some day. "This algorithm says you're guilty of a crime, and it's 99.8% accurate, which we've determined is acceptable, so there will be no way to appeal your verdict."