3 ms·

I've run a few large vBulletin forums for years. The biggest problem with internet commenting isn't the software, it's the spam. I wish I had a system that I

I've run a few large vBulletin forums for years. The biggest problem with internet commenting isn't the software, it's the spam. I wish I had a system that I could train with 10 years of user registration and comment moderation history and have it do the tedious, daily pointy-clicky grind for me.

Harelin 9y ago
The forum spam problem is easily solved if you ask a question during registration like: "What are the last four letters in the name of this website?" Anything that requires momentary thought from an actual user, and is specific to your website, will throw bots off.
- nodja 9y ago
  This only mitigates spam if your forum is small enough for spammers to not care. Those questions are a mere extra parameter on the registration post request. Spam software automatically detects those and asks a human to fill it. If the question is static it's even easier than a captcha solver.
  - ryacko 9y ago
    Spam is done by massive botnets that crawl the web at a rate nearly equal to Google's.
    - AFNobody 9y ago
      And they have their equivalents of Amazon's Mechanical TURK to learn how to bypass security questions. https://2captcha.com/public_statistics https://2captcha.com/public_statistics <- this for example You can find humans willing to work for $0.001/captcha.
      - littlestymaar 9y ago
        I've also read that some torrent or other illegal download website display captchas that are in fact captchas from other websites, to crowdsource the captcha bypass.
        AFNobody 9y ago
        Correct. There is a long list of human farming options they employ. They frequently use that one then "pretend" they have workers.
- AFNobody 9y ago
  As someone who uses stuff like that to reduce spam, it cuts spam by about ~50%. Many of them rely on human operator intervention for forum registration when the automated method(s) fail these days.
lobo_tuerto 9y ago
Have you tried using reCAPTCHA? It even has an "invisible" mode: https://developers.google.com/recaptcha/docs/invisible https://developers.google.com/recaptcha/docs/invisible
ioddly 9y ago
When I moderated a mid-sized forum, pretty much 90% of our spam had a URL in the first post, whereas normal users were pretty much the opposite. It seemed like sending those direct to moderation would have removed a lot of the work. In the end we just moderated all first posts since it wasn't that much volume and I didn't want to modify the software myself. Is your spam more complex? I've noticed that it seems to be getting fancier.
- always_good 9y ago
  A moderation queue for everyone's first post is among the best solutions if you can minimize approval time. But you need some impressive manpower for that, like mods willing to get notifications for each first post. Right now I post registrations and first-posts to a #forum-activity channel on my forum's Discord.
stevekemp 9y ago
I setup https://blogspam.net/ https://blogspam.net/ to offer an API to do exactly that kind of real-time spam detection for blog/forum comments. It works. But it is pretty coarse, and doesn't support training. (It used to have a bayasian filter setup, and you could train ham/spam, but I found too many reports of false-positives so this was removed.) I've seen plugins written for wordpress, drupal ,etc, but it never really took off the way I hoped it might. Still it does mean my own blog is spam-free.
- ploggingdev 9y ago
  Interesting, a few questions : * Is it like an open source alternative to Akismet? * Is it supported by the debian project or is it a personal project? * What are the specs of the server that hosts this service?
  - stevekemp 9y ago
    Yes it is like akismet, which is open-source. It is a personal project, although I did use to run it upon the https://debian-administration.org/ https://debian-administration.org/ website (which I created and ran as a personal site). I'm not sure why you'd ask about a link to Debian though! The system was recently rewritten in golang to improve performance, and it will run happily with 1Gb of RAM. It's running on a host that does a lot of other stuff, but basically it is an idle host. The biggest issue is that we store "known-bad" IPs in redis as a cache, so RAM can grow just due to that.
    - ploggingdev 9y ago
      > The biggest issue is that we store "known-bad" IPs in redis as a cache, so RAM can grow just due to that. How many requests do you serve per month? I see 50 million total rejected comments, but even if you serve a few million requests per month you should be fine with a traditional database like Postgres or MySQL.
      - stevekemp 9y ago
buro9 9y ago
Forcing sign-in to be "log in using email" wiped out +90% of spam for the ~300 forums I run. I use auth0 now in a passwordless setup, and it requires either Google Auth, Windows (Hotmail) Auth or a one-time-token being emailed to the person (with a 15 minute expiry). It has required that I pay for Sendgrid to ensure the email delivery... but it's worth it. I deal with virtually no spam nowadays and the biggest cure was this one thing.

Harelin 9y ago

The forum spam problem is easily solved if you ask a question during registration like: "What are the last four letters in the name of this website?" Anything that requires momentary thought from an actual user, and is specific to your website, will throw bots off.

nodja 9y ago
This only mitigates spam if your forum is small enough for spammers to not care. Those questions are a mere extra parameter on the registration post request. Spam software automatically detects those and asks a human to fill it. If the question is static it's even easier than a captcha solver.
- ryacko 9y ago
  Spam is done by massive botnets that crawl the web at a rate nearly equal to Google's.
  - AFNobody 9y ago
    And they have their equivalents of Amazon's Mechanical TURK to learn how to bypass security questions. https://2captcha.com/public_statistics https://2captcha.com/public_statistics <- this for example You can find humans willing to work for $0.001/captcha.
    - littlestymaar 9y ago
      I've also read that some torrent or other illegal download website display captchas that are in fact captchas from other websites, to crowdsource the captcha bypass.
      - AFNobody 9y ago
        Correct. There is a long list of human farming options they employ. They frequently use that one then "pretend" they have workers.
AFNobody 9y ago
As someone who uses stuff like that to reduce spam, it cuts spam by about ~50%. Many of them rely on human operator intervention for forum registration when the automated method(s) fail these days.

lobo_tuerto 9y ago

Have you tried using reCAPTCHA? It even has an "invisible" mode: https://developers.google.com/recaptcha/docs/invisible https://developers.google.com/recaptcha/docs/invisible

ioddly 9y ago

When I moderated a mid-sized forum, pretty much 90% of our spam had a URL in the first post, whereas normal users were pretty much the opposite. It seemed like sending those direct to moderation would have removed a lot of the work. In the end we just moderated all first posts since it wasn't that much volume and I didn't want to modify the software myself. Is your spam more complex? I've noticed that it seems to be getting fancier.

always_good 9y ago
A moderation queue for everyone's first post is among the best solutions if you can minimize approval time. But you need some impressive manpower for that, like mods willing to get notifications for each first post. Right now I post registrations and first-posts to a #forum-activity channel on my forum's Discord.

stevekemp 9y ago

I setup https://blogspam.net/ https://blogspam.net/ to offer an API to do exactly that kind of real-time spam detection for blog/forum comments. It works. But it is pretty coarse, and doesn't support training. (It used to have a bayasian filter setup, and you could train ham/spam, but I found too many reports of false-positives so this was removed.) I've seen plugins written for wordpress, drupal ,etc, but it never really took off the way I hoped it might. Still it does mean my own blog is spam-free.

ploggingdev 9y ago
Interesting, a few questions : * Is it like an open source alternative to Akismet? * Is it supported by the debian project or is it a personal project? * What are the specs of the server that hosts this service?
- stevekemp 9y ago
  Yes it is like akismet, which is open-source. It is a personal project, although I did use to run it upon the https://debian-administration.org/ https://debian-administration.org/ website (which I created and ran as a personal site). I'm not sure why you'd ask about a link to Debian though! The system was recently rewritten in golang to improve performance, and it will run happily with 1Gb of RAM. It's running on a host that does a lot of other stuff, but basically it is an idle host. The biggest issue is that we store "known-bad" IPs in redis as a cache, so RAM can grow just due to that.
  - ploggingdev 9y ago
    > The biggest issue is that we store "known-bad" IPs in redis as a cache, so RAM can grow just due to that. How many requests do you serve per month? I see 50 million total rejected comments, but even if you serve a few million requests per month you should be fine with a traditional database like Postgres or MySQL.
    - stevekemp 9y ago

buro9 9y ago

Forcing sign-in to be "log in using email" wiped out +90% of spam for the ~300 forums I run. I use auth0 now in a passwordless setup, and it requires either Google Auth, Windows (Hotmail) Auth or a one-time-token being emailed to the person (with a 15 minute expiry). It has required that I pay for Sendgrid to ensure the email delivery... but it's worth it. I deal with virtually no spam nowadays and the biggest cure was this one thing.