4 ms·
> The researchers observed that the accuracy of news sources the Twitter users retweeted promptly declined by roughly 1 percent in the next 24 hours after being
by s17n 5y ago
> The researchers observed that the accuracy of news sources the Twitter users retweeted promptly declined by roughly 1 percent in the next 24 hours after being corrected. Similarly, evaluating over 7,000 retweets with links to political content made by the Twitter accounts in the same 24 hours, the scholars found an upturn by over 1 percent in the partisan lean of content, and an increase of about 3 percent in the “toxicity” of the retweets, based on an analysis of the language being used.
Rule of thumb, you can disregard any social science paper where the effect size is single digits.
- ChrisLomont 5y agoWhy would real effects either be 0 or >= 10%? What would cause such a discontinuity across every single effect?
- Guvante 5y agoIt isn't that simple. We want to know what percentage of tweets have property X given event Y. We can estimate that using a sample and error bars. Note that the quotes specifically avoid this inference because the error bars would be silly. 500 million total tweets a day can't be talked about based on 7,000 hand categorized tweets. But non staticians don't have any reference point for how powerful a predictor it is so a short hand is < 10% probably means in the error bars so less believable.
- a1369209993 5y agoTaking your question non-rhetorically: because the error bars are wider than 10%[eg, 0, which finds a floor of 4% for anything based on polls], so anything less than that is consistent with p-hacking/random noise[eg see 1], and this is pervasive enough[2] that social science papers should be assumed garbage until proven otherwise. 0: http://slatestarcodex.com/2013/04/12/noisy-poll-results-and-reptilian-muslim-climatologists-from-mars/ http://slatestarcodex.com/2013/04/12/noisy-poll-results-and-... 1: https://www.explainxkcd.com/wiki/index.php/882:_Significant https://www.explainxkcd.com/wiki/index.php/882:_Significant 2: https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis
- maze-le 5y agoIt's not that no effects occur in this range, but if they do, they are not discernable from random fluctuations, noise and other statistical (non-)effects.
- fuzxi 5y agoIt can be real but unimportant. An extra drink every week might increase my liver cancer risk by 4%, but that's negligible if it's the difference between a 1 in 1 million chance and a 1 in 960,000 chance.
- emsal 5y agoAs imperfect measures as they are, statistical significance measures like the p-value are more epistemologically sound than any arbitrary rule of thumb-threshold on the size of the effect.
- asdfasgasdgasdg 5y agoStatistical significance is a good measure of how sure we are there is a correlation, or in this case with an RCT, causation. But any layperson can look at a 3% effect and conclude, yeah, that's probably not that big of a deal. Or not, depending upon your preferences! No judgement. It's not something that requires a degree to determine, just an assessment of one's own values and the effect size in that context.
- BeetleB 5y ago> statistical significance measures like the p-value are more epistemologically sound than any arbitrary rule of thumb-threshold on the size of the effect. Keep in mind that the GP isn't saying the effect doesn't exist if it's in the single digits, but that it is inconclusive and/or insignificant. Insignificant in the human sense, not the statistical sense. A 1% increase in this behavior? Irrelevant to almost everyone. This, of course, is not even getting into the issue of the reproducibility crisis, much of which did rely on p-values. While I personally am happy to do p-tests, the skepticism of small effects is well founded. Were someone else to try to reproduce the effects and fail, the standard defense is that the results are sensitive to the methodology used. It's much easier to invoke that defense if your effect is 1% vs 20%.
- unishark 5y agoP-values are based on a normally-distributed sampling distribution which presumes the samples are randomly-chosen. It's hard to see how random sampling can apply very well here. Suppose we did a comparison between the p-values of biased researchers desperate to publish, versus a conservative heuristic that doesn't believe small effects. Particularly for social science experiments like this, which would you bet on being able to assess repeatability better?
- s17n 5y agoP-values don't work (https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis).
- wolverine876 5y agoMy rule of thumb is to disregard snarky know-it-all comments by random people on the Internet.
- d4mi3n 5y agoOddly enough, this comment seems to reinforce the findings we just read about.
- scythe 5y agoBut now you've created three Borromean competing claims.
- d4mi3n 5y agoI hadn't heard of Borromean Knots before you made this comment and got a great read out of it. Thanks! For others following up on this thread, Borromean Knots seem to be an extension of psychoanalytic theory that tries to describe social interactions through an object oriented model that highlights the differences between symbolic, real, and imaginary claims: 1. https://en.wikipedia.org/wiki/Borromean_clinic https://en.wikipedia.org/wiki/Borromean_clinic 2. https://larvalsubjects.wordpress.com/2009/12/08/borromean-knots-ooo-and-social-and-political-theory/ https://larvalsubjects.wordpress.com/2009/12/08/borromean-kn...
- temp8964 5y agoYour comment does not provide any useful information at all. A merely personal attack.
- voldacar 5y agorussell's paradox: the hn comment
- temp8964 5y agoMy guessing is that their measurement of partisan lean of content and toxicity of retweets may have measurement error way larger than 1% or 3%, not to mention other noises in the stastistical modeling.