5 ms·
I would bet it is some ML model to detect fraud. This model will never be perfect – it will have false positives. My guess is – while picking the precision/reca
by vinay_ys 5y ago
I would bet it is some ML model to detect fraud. This model will never be perfect – it will have false positives. My guess is – while picking the precision/recall thresholds for blocking users, someone higher up would have argued it is okay to cause some false positives to prevent a lot of harm.
And they would have justified to themselves saying there's always recourse through customer support. But customer support tools to investigate and un-ban users would be slow and painful and lacks capabilities needed to check if the complaining user even passes the basic smell tests for a fraudster. And nobody can really explain why the model blocked the user in the first place. There's no well-lit path from CS to engineering on a case-by-case basis. Escalation would happen in bulk/batches – when lots of seemingly 'innocent' users complain to CS, CS may escalate to engineering.
Btw, the alternative of having to pick really conservative thresholds (with near-zero false-positives) causes more harm – harm that's more visible.
The slippery slope is this – over time the definition of 'trust and safety' would have been expanded to include interests of more and more stakeholders (including company's own business interests) – it is very easy to lose sight of serving the user (who may not be the paying customer).
- cube00 5y ago> it is very easy to lose sight of serving the user Every public corporation only serves its shareholders. If they annoy enough of the users and the shareholders take notice when it hits the bottom line maybe then something will be done.
- dymk 5y agoBoring take that shows up in every HN thread. I worked for years in Risk/Fraud and what you're saying is just not true. There are always tradeoffs to be made, we will always mistakes (hopefully rarely), and 99% of the time the people behind the curtain are trying very hard to reduce harm for the good actors.
- bbarn 5y agoI think most actors are genuinely acting in good faith. The problem comes when the machine gets so big that people don't understand that their good faith actions lead to overall bad outcomes. It's not so much evil corps as I think it sometimes is clueless corps.
- smugglerFlynn 5y agoThanks for that. Evil Corp narrative is strong on HN. Years working in corporate and I am yet to see people who are “serving shareholders” and “squeezing customers”. There are strategies that fail, and sometimes there are people who maximize personal gains at all cost. Exact same way as for any other kind of organization. Big problem is scale. Cost of mistake and collateral damages for big org will always be higher and more impactful due to size. 0.1% of customers for Amazon is 300 people.
- andi999 5y agoIsn't it more like Amazon has 300 Million customers so 1% is 3 Million, and 0,1% IS 300000 people?
- smugglerFlynn 5y agoHehe, that’s what I meant, but my math failed me
- berkes 5y agoSqueezing Customers does not automatically follow from Serving Shareholders. A healthy coorporation has these aligned: serving customers well, treating workers well results in shareholder value.
- midasuni 5y agogetting rid of 1% of customers to save 10% of expenditure is generally a good move. You can have 99% of customers are happy even if 1% are quietly given food poisoning and die, which is why we have laws to protect that 1%.
- berkes 5y agoAs Tim Ferris shows brilliantly in his famous book (four hour workweek) "Firing customers" can be greatly benefitial to all. Or, to stay in your analogy: striving to make a food that is 100% safe for 100% of the people will result in bland, poor food. It's fine to use gluten in your bread, or peanuts in your sause, even if x% of potential customers will get sick and can die, if they eat it. Fire those customers. Be clear: this is not for you. It makes the other customers happier. And consequently your stakeholders happier.
- raesene9 5y agoAnd this is why regulation of private corporations is a necessary evil, to provide an incentive to corporations to treat their customers better...
- lambertsimnel 5y ago> I would bet it is some ML model to detect fraud. This model will never be perfect – it will have false positives. My guess is – while picking the precision/recall thresholds for blocking users, someone higher up would have argued it is okay to cause some false positives to prevent a lot of harm. > And they would have justified to themselves saying there's always recourse through customer support. But customer support tools to investigate and un-ban users would be slow and painful and lacks capabilities needed to check if the complaining user even passes the basic smell tests for a fraudster. And nobody can really explain why the model blocked the user in the first place. There's no well-lit path from CS to engineering on a case-by-case basis. Escalation would happen in bulk/batches – when lots of seemingly 'innocent' users complain to CS, CS may escalate to engineering. Can such a model be trained effectively if isolated reports of false positives are rejected without meaningful investigation? In that case, wouldn't the model be trained with bad data? What if for each reported false positive there are more users affected who didn't report it (because their accounts were less valuable, because their time was more valuable, because they were too upset, because they were too timid, because they died (for unrelated reasons), etc)?
- tehwebguy 5y ago> Can such a model be trained effectively if isolated reports of false positives are rejected without meaningful investigation? No, and much like YouTube auto terminating accounts with 10 years of content there is absolutely no excuse for certain very obvious cases to be handled without human interaction. There absolutely must be flags that stop terminations without a human.