3 ms·
Tell HN: Search Marketplace for "Sony megastorage" get accused of child porno
You may think you want to buy a 300 CD player.
Facebook thinks your the worst sort of person, accuses you of searching for child porn and - somewhat strangely - offers to refer you to counseling.
I have to say I felt offended. I also wondered if I’d been added to some watchlist for the sin of searching for a 300 disc CD player.
Very strange.
- talhamahmood666 6mo agoThe "somewhat strangely offers to refer you to counseling" bit is the real tell. That's a human-review fallback UX. The classifier fired, they don't have enough signal to action it, so they route you to a soft warning that also doubles as a dark pattern if you're actually guilty. For normal users it reads as Kafka. Building detection systems I've seen this pattern a lot. False-positive rates on any NSFW or abuse classifier at Meta's scale are measured in millions of daily hits. "megastorage" tokenizing into something adjacent to a flagged term is the kind of thing that slips past ablation testing. The part worth worrying about isn't the accusation. It's that the appeal loop probably doesn't exist. If a classifier flags you, that signal persists in your profile's feature vector indefinitely. No one tells you, and nothing you do in the product removes it.
- andrewstuart 6mo agoYeah somewhere deep in Facebook they’ve put a black mark against my profile “filthy 300 CD player buyer, keep an eye on him”.