3 ms·

Detecting and hiding abusive tweets algorithmically is NOT THAT DIFFICULT. These methods are employed widely by Facebook and Google. It's baffling to me that

by bishnu 11y ago

Detecting and hiding abusive tweets algorithmically is NOT THAT DIFFICULT. These methods are employed widely by Facebook and Google. It's baffling to me that Twitter doesn't do the same thing, and is willing to burn money on pointless bureaucracy like this. Remember the "Other" inbox in FB? Twitter abuse could have been a footnote on par with that. Instead it's the only thing anyone talks about (deservedly so).

tptacek 11y ago
When you say something "algorithmic" is "NOT THAT DIFFICULT" on HN, I feel like you shoulder some obligation to explain some of the algorithmic approaches. You know, if it really is "not that difficult". That sure as shit sounds like a more interesting discussion to have than a semantic debate about free speech.
- bishnu 11y ago
  A regression model trained from existing spam and abuse report data to twitter would cover ~99% of terrible at-replies on twitter. Look how effectively a hacked-together script that just uses a couple of heuristics like GGAutoBlocker works. [1] Something a little more principled, trained with unfettered access to user data, would effectively kill the problem. IMO. [1] https://github.com/freebsdgirl/ggautoblocker https://github.com/freebsdgirl/ggautoblocker
  - harryh 11y ago
    Your understanding of what is difficult is wrong because you're looking at it as a detection problem when it's actually a false positive problem. See: http://laughingmeme.org/2011/07/23/cost-of-false-positives/ http://laughingmeme.org/2011/07/23/cost-of-false-positives/
    - bishnu 11y ago
      That is an overly simplistic problem description. It's trying to identify spam/abuse in a universal way, which is a much harder (and barely coherent) problem, and then delete those tweets which makes the cost of a false positive much higher. What’s far more feasible is to compute the probability a given user would like to see a specific tweet, and then have those below a certain threshold hidden (IE, don’t ping the notification tab for it). This is actually a very well-studied problem, and the massively parallel ML infrastructure it would take exists. Eg, http://www.datanami.com/2014/07/17/inside-sibyl-googles-massively-parallel-machine-learning-platform/ http://www.datanami.com/2014/07/17/inside-sibyl-googles-mass...

by bishnu 11y ago

tptacek 11y ago

When you say something "algorithmic" is "NOT THAT DIFFICULT" on HN, I feel like you shoulder some obligation to explain some of the algorithmic approaches. You know, if it really is "not that difficult". That sure as shit sounds like a more interesting discussion to have than a semantic debate about free speech.

bishnu 11y ago
A regression model trained from existing spam and abuse report data to twitter would cover ~99% of terrible at-replies on twitter. Look how effectively a hacked-together script that just uses a couple of heuristics like GGAutoBlocker works. [1] Something a little more principled, trained with unfettered access to user data, would effectively kill the problem. IMO. [1] https://github.com/freebsdgirl/ggautoblocker https://github.com/freebsdgirl/ggautoblocker
- harryh 11y ago
  Your understanding of what is difficult is wrong because you're looking at it as a detection problem when it's actually a false positive problem. See: http://laughingmeme.org/2011/07/23/cost-of-false-positives/ http://laughingmeme.org/2011/07/23/cost-of-false-positives/
  - bishnu 11y ago
    That is an overly simplistic problem description. It's trying to identify spam/abuse in a universal way, which is a much harder (and barely coherent) problem, and then delete those tweets which makes the cost of a false positive much higher. What’s far more feasible is to compute the probability a given user would like to see a specific tweet, and then have those below a certain threshold hidden (IE, don’t ping the notification tab for it). This is actually a very well-studied problem, and the massively parallel ML infrastructure it would take exists. Eg, http://www.datanami.com/2014/07/17/inside-sibyl-googles-massively-parallel-machine-learning-platform/ http://www.datanami.com/2014/07/17/inside-sibyl-googles-mass...