6 ms·

I think that incentive idea is great, and is a smart move to build a community, particularly when you're trying to draw subject matter experts. I like how some

I think that incentive idea is great, and is a smart move to build a community, particularly when you're trying to draw subject matter experts. I like how some of the Ask<X> reddits do it, by flagging people with verified advanced degrees. People think that news sites are afraid of conflicting opinions, but in my experience that's nonsense, it just has to be well thought out and not "DEATH TO <ISRAEL/ARABS/SUNNI/TURKS/AMERICA>", which is the vast majority. It still doesn't solve the problem that for someone to _find_ those great comments, they have to _read_ them, and stop them from getting buried. I'll err on the side of caution with revealing employee counts, but in my experience many of the FP/Atlantic/Mother Jones/Weekly Standard/Pick your midrange site are running on a single digit to low double-digit number of web production staff, many of whom are also trying to make a writing, article layout, or fact-checking quota. The suggestion that these magazines can either get those staffers to moderate tens of thousands of comments per day, or quadruple their web staff just to improve the comments ignores the business reality. User moderation in the normal HN/Reddit way doesn't work well on news sites, it's too easy to game or brigade, and news sites can't or won't give add unpaid moderators to be gatekeepers. That's what's hard; creating comments is scalable, filtering them is not. Leaving them unfiltered doesn't work either.

omginternets 10y ago
Thanks for your reply! >I think that incentive [is good], particularly when you're trying to draw subject matter experts. You bring up an excellent point. One of the fundamental problems with comments, I think, is that it creates a space in which ignorance and expertise are equally-weighted. In fact, it's often worse than that for reasons we all know: interesting issues are hard to distill into 300-or-so characters, and short, simple points are often more percussive. Vetting credentials is a very good option IMHO for certain forums but not for others. Reddit's /r/askscience is an example of a forum in which it works well. >It still doesn't solve the problem that for someone to _find_ those great comments, they have to _read_ them, and stop them from getting buried. I wonder if this problem can't be solved through the use of machine-learning to classify comments into high-versus-low quality by grammatical and semantic analysis. This kind of first-pass filtering could, at the very least, help throw out the obvious trash and pre-select candidates for recognition. Such a system can be tuned to minimize false-alarms (shitpost getting flagged as good), which I think represent the most problematic of classification errors. This is a nice problem-space for ML because the increase in misses implied by a bias against false-alarms doesn't degrade the service much: not having one's comment select for re-publication is unexceptional. Thoughts?
- showerst 10y ago
  RE:Machine learning: I think there are two problems with that approach, one cultural and one technological. The cultural issue is that many news orgs are still run by people for whom the idea that technology could accidentally censor a valid criticism or ban a decent voice is just too risky. I think this is changing, and many newsrooms today a little more fluid than when I really cared about the problem 4 years ago. The tech issue is a little bit of a cop out on my part. An ML approach is super attractive to me as a techie. Google (youtube), facebook, NYT, WaPO, and tons of other billion dollar orgs have this problem, and could loads of money by being seen as better communities. On the more guerrilla side, hundreds of subreddits have automoderaters written by savvy, caring moderators. They have terabytes of training data, already tagged, and world class ML experts on staff. If it was a tractable problem with business value, why wouldn't they have fixed it? I'm guessing it's the sort of thing that looks doable from the surface, but you get buried in the details. Again, cop out answer, so please go prove me wrong!!
  - omginternets 10y ago
    >cultural issue I understand, and I think that's probably the most difficult problem of the two. I'd just like to point out -- in the interest of discussion -- three things: 1. Pre-filtering for moderators is different (much safer) than auto-banning by a bot 2. It's valid both to filter informed opinions that are poorly expressed, and for a publisher to have a preferred "voice", i.e. a style of writing that it favors. 3. The argument can be made that machines are no more biased than human editors, and that in many cases, the biases of the former are known. As a corollary to this point, there exist certain ML techniques (e.g. randomized forrest classifiers) for which the decision process of an individual case can be retraced after the fact. How do you think publishers would respond to these counter-points? >technical problem Counter-cop-out: someone has to be the first! Somewhat-less-cop-outy-counter-cop-out: by your own admission, certain sites (e.g. Reddit) have high-quality automoderators. I would argue that the problem is "approximately solved" and that this is sufficient for the purposes of moderating an internet news publisher. Again, I would make the signal-detection-theoretic point of my previous comment: I can selectively bias my automoderators in favor of reducing either false-alarms or misses. Of course, this brings us back to the cultural problem you mentioned. By this I conclude that the bottleneck is cultural, which brings me to a follow-up question: what do you think is driving the increased tolerance towards accidentally censoring a "decent voice"? Is it the understanding that it doesn't matter so long as a critical mass of decent voices are promoted?
deleted 10y ago
[deleted]
LarbarBarking 10y ago
For news sites with a paid component, they could allow comments only from subscribers / donors. Having a gate which involves money will improve the conversation somewhat. I'd even go a step further and make comments invisible except for subscribers. People creating trial paid accounts could see the comments but not comment themselves. This latter step would prevent astroturfing from firms willing to pay $10 for a trial but not $100 for an annual subscription. Moderators would still be needed but their workload would be reduced. And there would be money available for them since many would subscribe / donate just to be part of the community, which would make moderation less of a drain and more of the core profit-making.

omginternets 10y ago

Thanks for your reply! >I think that incentive [is good], particularly when you're trying to draw subject matter experts. You bring up an excellent point. One of the fundamental problems with comments, I think, is that it creates a space in which ignorance and expertise are equally-weighted. In fact, it's often worse than that for reasons we all know: interesting issues are hard to distill into 300-or-so characters, and short, simple points are often more percussive. Vetting credentials is a very good option IMHO for certain forums but not for others. Reddit's /r/askscience is an example of a forum in which it works well. >It still doesn't solve the problem that for someone to _find_ those great comments, they have to _read_ them, and stop them from getting buried. I wonder if this problem can't be solved through the use of machine-learning to classify comments into high-versus-low quality by grammatical and semantic analysis. This kind of first-pass filtering could, at the very least, help throw out the obvious trash and pre-select candidates for recognition. Such a system can be tuned to minimize false-alarms (shitpost getting flagged as good), which I think represent the most problematic of classification errors. This is a nice problem-space for ML because the increase in misses implied by a bias against false-alarms doesn't degrade the service much: not having one's comment select for re-publication is unexceptional. Thoughts?

showerst 10y ago
RE:Machine learning: I think there are two problems with that approach, one cultural and one technological. The cultural issue is that many news orgs are still run by people for whom the idea that technology could accidentally censor a valid criticism or ban a decent voice is just too risky. I think this is changing, and many newsrooms today a little more fluid than when I really cared about the problem 4 years ago. The tech issue is a little bit of a cop out on my part. An ML approach is super attractive to me as a techie. Google (youtube), facebook, NYT, WaPO, and tons of other billion dollar orgs have this problem, and could loads of money by being seen as better communities. On the more guerrilla side, hundreds of subreddits have automoderaters written by savvy, caring moderators. They have terabytes of training data, already tagged, and world class ML experts on staff. If it was a tractable problem with business value, why wouldn't they have fixed it? I'm guessing it's the sort of thing that looks doable from the surface, but you get buried in the details. Again, cop out answer, so please go prove me wrong!!
- omginternets 10y ago
  >cultural issue I understand, and I think that's probably the most difficult problem of the two. I'd just like to point out -- in the interest of discussion -- three things: 1. Pre-filtering for moderators is different (much safer) than auto-banning by a bot 2. It's valid both to filter informed opinions that are poorly expressed, and for a publisher to have a preferred "voice", i.e. a style of writing that it favors. 3. The argument can be made that machines are no more biased than human editors, and that in many cases, the biases of the former are known. As a corollary to this point, there exist certain ML techniques (e.g. randomized forrest classifiers) for which the decision process of an individual case can be retraced after the fact. How do you think publishers would respond to these counter-points? >technical problem Counter-cop-out: someone has to be the first! Somewhat-less-cop-outy-counter-cop-out: by your own admission, certain sites (e.g. Reddit) have high-quality automoderators. I would argue that the problem is "approximately solved" and that this is sufficient for the purposes of moderating an internet news publisher. Again, I would make the signal-detection-theoretic point of my previous comment: I can selectively bias my automoderators in favor of reducing either false-alarms or misses. Of course, this brings us back to the cultural problem you mentioned. By this I conclude that the bottleneck is cultural, which brings me to a follow-up question: what do you think is driving the increased tolerance towards accidentally censoring a "decent voice"? Is it the understanding that it doesn't matter so long as a critical mass of decent voices are promoted?

deleted 10y ago

[deleted]

LarbarBarking 10y ago

For news sites with a paid component, they could allow comments only from subscribers / donors. Having a gate which involves money will improve the conversation somewhat. I'd even go a step further and make comments invisible except for subscribers. People creating trial paid accounts could see the comments but not comment themselves. This latter step would prevent astroturfing from firms willing to pay $10 for a trial but not $100 for an annual subscription. Moderators would still be needed but their workload would be reduced. And there would be money available for them since many would subscribe / donate just to be part of the community, which would make moderation less of a drain and more of the core profit-making.