5 ms·
How to Lie and Distort with Statistics 101: 1. Use absolute numbers or percentages depending on whichever sounds higher; 2. Don't put absolute numbers in cont
by cletus 5y ago
How to Lie and Distort with Statistics 101:
1. Use absolute numbers or percentages depending on whichever sounds higher;
2. Don't put absolute numbers in context. For example, is that 119k out of 500k or 500M?
3. Only mention false positive rate if it's really low;
4. Only mention false negative rate if it's really low;
5. Don't be afraid to retroactively include data in your featured set. For example, classifying accounts banned for other reasons as spam accounts;
6. Only mention time frames if it helps your case. For example, if those accounts had been operating for a mean of 3 years then say nothing about that.
I'm not saying any of the above are true but after having seeing thousands of these pronouncements, my mind automatically goes to start asking what isn't being said here.
I remember seeing this first hand at Google with the cut backs in food (cafes and microkitchens). There were statements about "we saved $100m". That's an absolute number that sounds large. It actually amounted to something like $2/employee per day for a noticeable decrease in both quantity and quality to the point where people complained. Was that really worth it? Probably not.
- asdfasgasdgasdg 5y agoTbh, I think this blog post is pretty content free and I'm unsure what merit other people are seeing in it. It's a fluff piece put out by the play store team to toot their horn about the work that they've accomplished last year. I get why they'd be proud of it, but I'm not sure what actionable information the public is supposed to derive besides, "Hey, we are doing stuff." That being said, I don't think there's any intent to mislead here. The numbers could be more complete and detailed, but even if you had relative numbers (e.g. "we banned 119k, which is X% of all developer accounts") what could you do with that information? If it were 20% would that be too low? Too high? What if it were 1%? Since you don't know the ground truth fraction of spam accounts, and you don't know how well a similarly situated generic team of devs would do, I don't think there's much use you can get out of this knowledge.
- jsnell 5y agoThe merit people are seeing is the editorialized title, which lets us have yet another repetitive discussion about account bans.
- tyingq 5y agoThis funny stat stood out to me as well: "Additionally, in 2020, Google Play Protect scanned over 100B installed apps each day" Are there really 100B unique apps? Or are they saying they scanned some much smaller number of apps, that happen to be "installed" on a shite ton of phones? Why is the number of installs relevant to the how many unique apps were scanned on a cloud server somewhere? Edit: I'm assuming "Play Protect" just uploads a hash of the binary and makes sure that particular binary has been previously scanned?
- throwaways885 5y agoApp installs. I think Play Protect is like anti-virus. I think it also works on apps outside of the Play Store, which makes more sense since it can't just be a server-side thing.
- jpalomaki 5y agoFor me the value in seeing this kind actual numbers is that it puts into some kind of perspective the reports of falsely terminated accounts. If you are terminating accounts at this level, then it is - at least for me - a bit more understandable that you can't conduct a deep human review of each termination (and even if you do, there would be still likely some mistakes).
- fartcannon 5y agoIf they can't do it fairly, correctly, and quickly, they shouldn't be permitted to do it and profit from it at all.
- smabie 5y agoI mean, can anyone do things fairly, correctly, and quickly? I don't think that's even possible.
- pradn 5y agoThanks for this succinct and useful list. :)
- Lammy 5y ago7. Use common dictionary words like “spam” despite having a specific in-house definition for what that means. I’ve read way too many examples of “Google closed my account for <speculation> with no recourse and won’t provide any details” just here on HN alone.