4 ms·
Ask HN: Scientific Study of HN flag dynamics
Would there be any interest in a scientific study of the effectiveness of HN flagging, including:
- false positives
- false negatives
- brigading
- flagged topics vs off-HN contexts
- flagged topic trends over extended time
- content origin (domain, geo, org, author)
Pointers to prior research on distributed flagging and moderation would be welcome.
- WalterGR 5y agoDefinitely. Add to the facets: the submitted domain. Maybe posts by the armchair experts and quarterbacks on Substack get flagged more often than the WSJ, for example.
- walterbell 5y agoThanks, added "content origin".
- aaron695 5y ago
- brudgers 5y agoThere are no objective criteria for classifying flags as false positive/negative.
- walterbell 5y agoWhat does [flagged] mean? Users flagged the post as breaking the guidelines or otherwise not belonging on HN. Moderators sometimes also add [flagged] (though not usually on submissions), and sometimes turn flags off when they are unfair. A false positive is a user flag that was reversed on appeal, after manual review by HN human moderators. In a study, a list of false positive flags could be derived from public data. In a study, additional false positive candidates could be identified after manual review.
- brudgers 5y agoFlags add gravity. Sometimes, but only rarely, to the point of flagged-to-dead. Mostly flags just move submissions away from the top of the front page. It is very rare for those to be “deflagged.” Rare enough that there’s a moderation comment about turning flags off. And an appeal that the community be on its better behavior. They are editorial exceptions not rule following corrections.
- walterbell 5y ago> Flags add gravity. For the purpose of a study, gravity can be inferred from story rank. > Sometimes, but only rarely, to the point of flagged-to-dead. A study could examine the three categories of: - gravity - dead - unfair It would need to be long enough that all of those categories were non-zero.
- brudgers 5y agoThere’s no unfair…or fair. HN’s not a court of law.
- walterbell 5y agoThis sub-thread is about false positives, which has already been stated as a user-flagged event that was subsequently reversed by an HN moderator. That is a physically observable event that is visible in the HN public record, therefore it is an event that is subject to scientific study. It either exists or does not, so it is objective. In this sub-thread, "unfair" is quoted directly from the HN guidelines as the term for the objective event of a reversed flag, i.e. a "false positive", which we are discussing. It's not being used here as an adjective, it's being used to connect this discussion to the official HN FAQ.
- sjg007 5y agoIs this data publicly available?
- walterbell 5y agoRaw flag data is not public. Some studies are possible on the public data. Any interesting results could be investigated further, privately, by HN moderators.
- walterbell 5y agoResponse to aaron695: https://news.ycombinator.com/item?id=30496522 https://news.ycombinator.com/item?id=30496522 > The open data would tell an incomplete picture. It would still be an interesting to see what you can pull out, but you'd be guessing why a lot of the time. The current implementation is like the DMCA. Anything can be censored for any reason without input from HN moderators. If the submitter is aware of the process, they can appeal to HN moderators. If HN readers saw the submission before it was flagged, they can also appeal to HN moderators. If there were organized efforts to censor specific topics, one mitigation could be to deny the benefits of censorship. This could be done from public data by extracting a queue of flagged submissions and feeding them into another level of non-public analysis, automated + human. The surviving stories could then be posted to a dedicated site, e.g. using https://github.com/lobsters/lobsters https://github.com/lobsters/lobsters. If there were organized efforts to censor specific topics, a secondary public log of valid-but-censored articles could trigger the Streisand effect. Not only could this deny the benefit of censorship, it could cause censorship to have the opposite effect, reducing demand for organized flagging. This would have no impact on organic, non-coordinated flagging.
- WalterGR 5y agoWait - are you proposing to study flagging or “censorship” on HN? Flagging certainly exists. What is your evidence in favor of censorship existing here? If memory serves, there are website(s) that track which submission get flagged. If you’re concerned about a favored view or topic being flagged frequently, you could start there. Otherwise you risk exerting lots of effort to simply find out that a favored view or topic isn’t shared by lots of people here. And, well, not all of them are equally important, or equally important to the HN audience.
- walterbell 5y agoThe study would be on flagging, which is typically done by independent HN users. HN has non-public mechanisms to detect some coordinated activity (e.g. upvoting submissions) across registered usernames. > If there were organized efforts to censor specific topics Note the "if" at the beginning of the sentence. It's not about transient topics, but long-term patterns. Broadly speaking, "censorship" is only one possible motive for coordinated activity, flagging or otherwise. > If memory serves, there are website(s) that track which submission get flagged. Thanks for the pointer, found https://news.ycombinator.com/item?id=29144989 https://news.ycombinator.com/item?id=29144989 & https://news.ycombinator.com/item?id=29113079 https://news.ycombinator.com/item?id=29113079 & https://github.com/MoserMichael/flagged-hn https://github.com/MoserMichael/flagged-hn.
- musawo 5y agohttps://youtu.be/l2H99Pp9U6o https://youtu.be/l2H99Pp9U6o
- walterbell 5y agoAn larger and noisier category for study: recurring patterns of low-quality (downvoted, flagged) spam comments on some topics.