4 ms·

FWIW I made a website for our yearly project conference that allowed anybody to create an account and post material. But 1) all postings had to be moderated un

by gwd 6y ago

FWIW I made a website for our yearly project conference that allowed anybody to create an account and post material. But 1) all postings had to be moderated until the account was verified (either manually by me or by a code sent only to conference attendees) 2) I was pretty active in monitoring new posts and deleting posts and accounts that were clearly spam. And of course the normal stuff, links all prefixed with "nofollow", only whitelisted markdown content / HTML, &c. The first year we had a handful of spammers try to post stuff, but it didn't take much time at all to filter it out. This year we didn't have any spam accounts at all. So my suspicion is that getting ahead of the "spam problem" by doing heavy manual moderation early on is an investment. If he'd just spent 5 minutes a day looking at the accounts and deleting rubbish, he would never have been swarmed by spammers, and the moderation load would have remained relatively low, until he got popular enough that he could afford to actually spend some real effort automating / crowdsourcing the spam-fighting capabilities.

julianlam 6y ago
The key here is manual moderation. At NodeBB we also have had our fair share of spam companies trying to build scripts to post things, and the only foolproof solution is manual moderation via a post queue for new users. The downside, of course, is that they takes effort to maintain, and is a barrier to entry for new accounts.
- 101008 6y ago
  Unrelated, but seeing NodeBB reminded me of the old days when installing forums for communities. phpBB2, vBulletin, downloading plugins and themes, etc. Oh, the nostalgy for the old web is amazing...
  - vorticalbox 6y ago
    I recently got a raspberry pi and set up phpbb using docker just for a fun side project. I looked at nodebb might give that ago next
- hinkley 6y ago
  Do you get a lot of people reposting the same thing because they think it didn’t work the first time? I wonder if you treated new content from unverified users like a temporary shadowban (they see it, nobody else does) how that would affect behavior.
  - birdyrooster 6y ago
    Great idea! Edit: Why does shadow banning feel like such an elegant solution? Everything has tradeoffs and I feel like shadow banning has tons of upside and very little downside. What am I missing?
    - keenmaster 6y ago
      The only risk I can think of is that someone will upload the video and then livestream its playback on your platform with your branding using a separate livestreaming tool.
      - kazinator 6y ago
        I.e. shadow banning semantically cannot work with a site that is geared toward publishing content, because the content is supposed to be visible to visitors who are not authenticated. If it isn't visible, that is painfully obvious. Shadow banning only works when only authenticated users can see any content at all; then we arrange for only the offender to see content they have created. This works as long as the offender doesn't create multiple accounts. (You need something slightly more clever, like allowing the content to be viewed from the same IP address as the last known address for that offender, plus some surrounding range (e.g. IPv4 class C subnet)).
    - lostlogin 6y ago
      The only one I can think of is that accounts may have extremely objectionable content and it remains there. That could be a legal liability.
    - dredmorbius 6y ago
      There are cases in which explicit and obvious moderation results in retaliation, including DDoS and hacking attacks. Various soft fail mechanisms, including shadowbanning, degraded performance, errors, authentication failures, etc., may help avoid this. The question is what adversarial model you're facing: is it some random pr0n / SEO / affiliate / fraudster, or is it a "nice website you gots heyah, be a shame if anyting happen' to it" squad? Former can be modded away. Latter takes nuance.
freedomben 6y ago
Interesting, thanks for sharing. I'll be launching a service that allows users to upload content and I've been thinking about how to handle anti-spam. I really appreciate you sharing your experience.
stevenicr 6y ago
Manual moderation in this way could open you up to other forms of liability that you may have easier defense against without it. I don't think it'd be an issue for a hardware forum, although mass approving comments and finding that one is defamatory about person or company X could get some lawyer's drool glands working for example.. Also section 230 appears to be under threat by the current administration and how things are interpreted - and I just read where Biden has been talking about removing it completely.. So this situation is in a state of flux at the moment I believe. /not a lawyer / doctor yada yada
- thenanyu 6y ago
  This is a common misconception of section 230. Section 230 absolves platform companies from responsibility over speech on the platform. It does not care about what is moderated out. No one is responsible for speech that isn’t said.
  - triceratops 6y ago
    People complaining about "censorship" have been scarily effective in spreading this misconception of Section 230.
    - stevenicr 6y ago
      I am trying to understand this comment. I do not claim to know everything about 230 - but have read several news stories about it, in relation to it- and have sat in an a seminar by a high dollar lawyer once.. Since I did not stay in a Holiday Inn express I can't claim that I know it all - haha which is also why I put not a lawyer / doctor / yada yada - so people would research it more. I believe this would be more of an issue for some kinds of sites and much less an issue for others. (I doubt tom's hardware has ever had anything defamatory or libelous in posts to worry about ) - other places that moderate may do so in ways that are over-zealous in protecting the publishers and therefore that would create a different issue, but that's a different discussion. Anyhow if you are referring to an anti-censor group or some kind of misinformation army like the bots and confangled news stories around the net neutrality debate, I would like to learn more about this/them and see why / how etc. thanks
Pick-A-Hill2019 6y ago
100% this. As soon as an IP / Service appears for the first time it is scanned and hammered to look for low hanging fruit. Once the initial wave of exploit bots have found out they don't get far, your 'thing' is marked as 'too much faff' and malicious traffic drops off and is left alone (usually, y.m.m.v etc etc etc).
rstuart4133 6y ago
My experience is different. I manage a wiki. It was a wonderful success in terms of getting people to contribute, however the wiki itself (moinmoin) has lousy authorisation and authentication tools, so I wrote my own. A few versions later changed to moinmoin broke those tools, but by that time most of time content has settled down so I just reverted locking down the system by making the user database read only (literally: "chmod -R a-w"). That worked well enough. However every so often you would have to create a new account. To do that the chmod had to be undone, but only for a minute or two. In that minute or two typically 2 or 3 spam accounts where created, and maybe 20 wiki pages spammed or defaced. In short: defending the site had no effect whatsoever for us. The bots were always probing, every few seconds, and it never stopped even years after the site was totally locked down. PS: SPAM wise the bots were just an annoyance. But every page hit runs moinmoin's Python code, and it's not the fastest thing. We were running on low end VPS's that took a dim view of anybody using too much CPU. Our VM regularly got shut down because of those bloody bots.
- julianlam 6y ago
  Perhaps you could hand-tailor a fail2ban rule that automatically jailed anyone who even _attempted_ to log in, for a whole month, and leave that running for a week prior to the chmod, and disable it during the editing process?
- gwd 6y ago
  > however the wiki itself (moinmoin) Right, so I think the key think spammers need to make spam "pay" is to make it scalable: only 0.1% of people will click on your link, but if you can generate a million new "views" per month, then that's 1000 clicks per month. So the key thing for a spammer is to be able to automate as much as possible. One key element in preventing spam, therefore, is to try to make it so that a human has to be in the loop somewhere. In my case, I wrote the website; I suspect any login bot would have to be customized to my specific website to be effective. I'm sure it doesn't take much time, but you'd still have to start an automatic bot specifically for my site. Why would you do that if you saw that all of your attempts to get spam up had failed? In your case, you're using a standard tool. Someone's already written a bot that can log into any moinmoin instance; and almost certainly someone's written tools to scan all websites for new moinmoin instances and try to create accounts. It's probably unlikely anyone has specifically thought about your site at all; they'd probably have to write special code to remove it from their automatic scans. Same thing as before -- even if their bot fails on 99% of moinmoin sites, that 1% makes it worth keeping it going. > PS: SPAM wise the bots were just an annoyance. But every page hit runs moinmoin's Python code and it's not the fastest thing. Golang FTW. :-D

by gwd 6y ago

julianlam 6y ago

The key here is manual moderation. At NodeBB we also have had our fair share of spam companies trying to build scripts to post things, and the only foolproof solution is manual moderation via a post queue for new users. The downside, of course, is that they takes effort to maintain, and is a barrier to entry for new accounts.

101008 6y ago
Unrelated, but seeing NodeBB reminded me of the old days when installing forums for communities. phpBB2, vBulletin, downloading plugins and themes, etc. Oh, the nostalgy for the old web is amazing...
- vorticalbox 6y ago
  I recently got a raspberry pi and set up phpbb using docker just for a fun side project. I looked at nodebb might give that ago next
hinkley 6y ago
Do you get a lot of people reposting the same thing because they think it didn’t work the first time? I wonder if you treated new content from unverified users like a temporary shadowban (they see it, nobody else does) how that would affect behavior.
- birdyrooster 6y ago
  Great idea! Edit: Why does shadow banning feel like such an elegant solution? Everything has tradeoffs and I feel like shadow banning has tons of upside and very little downside. What am I missing?
  - keenmaster 6y ago
    The only risk I can think of is that someone will upload the video and then livestream its playback on your platform with your branding using a separate livestreaming tool.
    - kazinator 6y ago
      I.e. shadow banning semantically cannot work with a site that is geared toward publishing content, because the content is supposed to be visible to visitors who are not authenticated. If it isn't visible, that is painfully obvious. Shadow banning only works when only authenticated users can see any content at all; then we arrange for only the offender to see content they have created. This works as long as the offender doesn't create multiple accounts. (You need something slightly more clever, like allowing the content to be viewed from the same IP address as the last known address for that offender, plus some surrounding range (e.g. IPv4 class C subnet)).
  - lostlogin 6y ago
    The only one I can think of is that accounts may have extremely objectionable content and it remains there. That could be a legal liability.
  - dredmorbius 6y ago
    There are cases in which explicit and obvious moderation results in retaliation, including DDoS and hacking attacks. Various soft fail mechanisms, including shadowbanning, degraded performance, errors, authentication failures, etc., may help avoid this. The question is what adversarial model you're facing: is it some random pr0n / SEO / affiliate / fraudster, or is it a "nice website you gots heyah, be a shame if anyting happen' to it" squad? Former can be modded away. Latter takes nuance.

freedomben 6y ago

Interesting, thanks for sharing. I'll be launching a service that allows users to upload content and I've been thinking about how to handle anti-spam. I really appreciate you sharing your experience.

stevenicr 6y ago

Manual moderation in this way could open you up to other forms of liability that you may have easier defense against without it. I don't think it'd be an issue for a hardware forum, although mass approving comments and finding that one is defamatory about person or company X could get some lawyer's drool glands working for example.. Also section 230 appears to be under threat by the current administration and how things are interpreted - and I just read where Biden has been talking about removing it completely.. So this situation is in a state of flux at the moment I believe. /not a lawyer / doctor yada yada

thenanyu 6y ago
This is a common misconception of section 230. Section 230 absolves platform companies from responsibility over speech on the platform. It does not care about what is moderated out. No one is responsible for speech that isn’t said.
- triceratops 6y ago
  People complaining about "censorship" have been scarily effective in spreading this misconception of Section 230.
  - stevenicr 6y ago
    I am trying to understand this comment. I do not claim to know everything about 230 - but have read several news stories about it, in relation to it- and have sat in an a seminar by a high dollar lawyer once.. Since I did not stay in a Holiday Inn express I can't claim that I know it all - haha which is also why I put not a lawyer / doctor / yada yada - so people would research it more. I believe this would be more of an issue for some kinds of sites and much less an issue for others. (I doubt tom's hardware has ever had anything defamatory or libelous in posts to worry about ) - other places that moderate may do so in ways that are over-zealous in protecting the publishers and therefore that would create a different issue, but that's a different discussion. Anyhow if you are referring to an anti-censor group or some kind of misinformation army like the bots and confangled news stories around the net neutrality debate, I would like to learn more about this/them and see why / how etc. thanks

Pick-A-Hill2019 6y ago

100% this. As soon as an IP / Service appears for the first time it is scanned and hammered to look for low hanging fruit. Once the initial wave of exploit bots have found out they don't get far, your 'thing' is marked as 'too much faff' and malicious traffic drops off and is left alone (usually, y.m.m.v etc etc etc).

rstuart4133 6y ago

My experience is different. I manage a wiki. It was a wonderful success in terms of getting people to contribute, however the wiki itself (moinmoin) has lousy authorisation and authentication tools, so I wrote my own. A few versions later changed to moinmoin broke those tools, but by that time most of time content has settled down so I just reverted locking down the system by making the user database read only (literally: "chmod -R a-w"). That worked well enough. However every so often you would have to create a new account. To do that the chmod had to be undone, but only for a minute or two. In that minute or two typically 2 or 3 spam accounts where created, and maybe 20 wiki pages spammed or defaced. In short: defending the site had no effect whatsoever for us. The bots were always probing, every few seconds, and it never stopped even years after the site was totally locked down. PS: SPAM wise the bots were just an annoyance. But every page hit runs moinmoin's Python code, and it's not the fastest thing. We were running on low end VPS's that took a dim view of anybody using too much CPU. Our VM regularly got shut down because of those bloody bots.

julianlam 6y ago
Perhaps you could hand-tailor a fail2ban rule that automatically jailed anyone who even _attempted_ to log in, for a whole month, and leave that running for a week prior to the chmod, and disable it during the editing process?
gwd 6y ago
> however the wiki itself (moinmoin) Right, so I think the key think spammers need to make spam "pay" is to make it scalable: only 0.1% of people will click on your link, but if you can generate a million new "views" per month, then that's 1000 clicks per month. So the key thing for a spammer is to be able to automate as much as possible. One key element in preventing spam, therefore, is to try to make it so that a human has to be in the loop somewhere. In my case, I wrote the website; I suspect any login bot would have to be customized to my specific website to be effective. I'm sure it doesn't take much time, but you'd still have to start an automatic bot specifically for my site. Why would you do that if you saw that all of your attempts to get spam up had failed? In your case, you're using a standard tool. Someone's already written a bot that can log into any moinmoin instance; and almost certainly someone's written tools to scan all websites for new moinmoin instances and try to create accounts. It's probably unlikely anyone has specifically thought about your site at all; they'd probably have to write special code to remove it from their automatic scans. Same thing as before -- even if their bot fails on 99% of moinmoin sites, that 1% makes it worth keeping it going. > PS: SPAM wise the bots were just an annoyance. But every page hit runs moinmoin's Python code and it's not the fastest thing. Golang FTW. :-D