4 ms·

Ask HN: 2 karma minimum to lock out spam bots

52 points by jauco 18y ago

I just flagged two submissions that were blatant spam, both submitted by accounts created two seconds before the post. Wouldn't a 2 point karma minimum to url submission and voting fix a lot of spam posts? For legitimate users it's pretty easy to get one upvote on a comment and as a nice bonus it would raise the chance that the user has a basic grasp of community values. For spam bots OTOH it's impossible to submit links before they get one upvote and it's impossible to upvote themselves before they get one upvote. I think that it effectively locks them out.

DarkShikari 18y ago
I saw an interesting method used by spambots on the Doom9 forum: they would take old threads and repost them (with a link to their ad in their signature or such). Some variant of this method might work to circumvent such a policy; find a similar thread, pick a comment from it, and post it, perhaps? It might only work 20% of the time, but that's good enough to get some spam accounts with URLs approved. Of course this would all take effort, and might be enough to lock out a lot of would-be spammers, or at least convince them to go somewhere else.
- Tichy 18y ago
  Reminds me of what I see on my Wordpress Blog: lot's of 0-content comments like "this is a cool article". At first they didn't make sense (blocked them anyway), but then I remembered the Wordpress settings for comments. By default, it is set to "commentors must have been approved at least once". So I guess if I would let one of those innocent comments through, they would be back with more serious stuff. Spammers are crafty...
ashleyw 18y ago
Good idea; though I'd argue 2 is too low, if these aren't "bots" but real humans, it wouldn't be hard for them to adapt by commenting and then upvoting each other. 10+ would be nicer, anyone truly interested in HN would understand why! :)
- chanux 18y ago
  Well.. I don't really agree with you. 10+ would be a bit painful I guess. Let's first keep the bots out. Then let's find a way to keep lamers away. There's always "the erlang week" to scare away lamers anyway.
- talison 18y ago
  It's a good point that some spam "bots" could be human. I was running a free email site when we saw a lot of strange account creation. We had recaptcha enabled and knew it hand't been cracked. It turned out (based on IP address analysis) that the accounts were being created by humans in the Philippines and then handed over to spammers in Dubai. Ah globalization... If you have an efficient spam vector, it's not unusual to see low-wage humans manipulating the system to get around captcha.
  - jasonkester 18y ago
    This is a lot bigger than you'd expect. Nearly all the spam that makes it into the database on my sites is human-powered. It's maybe only 1% of the total attack volume, but because simple checks knock out nearly all the noise, it becomes the most significant fraction that you have to deal with.
    - ErrantX 18y ago
      limit it to 1 account creation per IP per hour :) Yeh they can use a ton of proxies to get round that but I bet it cuts the account creation right down. And it shouldnt affect 99.999999% of "real" users.
      - jasonkester 18y ago
        Ah, but there's the rub. Extrapolating from my comment above, your number one job is to make spammers feel successful when they fail. If you reject new accounts like this, you'll force them to write those little proxies to get around your system. The better thing to do is to simply notice what they're doing and flip the IsSpammer bit on all those new accounts (including the first one.) That way you can correctly classify any content they may post from those accounts in the future.
ComputerGuru 18y ago
That would encourage them to try to generate spam comments in the slim hope that they'd get upvoted.... which would lead to a much worse spam situation. For instance, they'd employ Markov Chains to re-hash comments in a post, or from older posts, or from the text of the article. One thing I've learned: spammers don't give up. If you give them an easy way to submit that's even easier for you to clean up, that's better than starting an all-out war that'll just increase the spam content and make it more difficult to filter out.
- CalmQuiet 18y ago
  Thank you: I needed to hear that. Part of my brain likes to keep scheming about "more absolute" ways to keep them out (and thus avoid any clean up). Your points suggest that my utopia-philic thinking is probably less useful than accepting that some clean up is just a necessary overhead (cf: "Eternal vigilance is the price of liberty."). Thank you.
- mixmax 18y ago
  Would a spammer really go through that just to submit a spam link that probably wouldn't get beyond the new page anyway? It's quite a bit of work, even if you have the same framework that you can use across multiple sites. My experience is that a post that doesn't get upvoted will receive around 30 hits. Surely there are easier ways of getting 30 hits.
  - mooism2 18y ago
    They only have to write the code once. Then they can get 30 hits to a different site every day.
    - mixmax 18y ago
      I don't think it's feasible to make "write once run anywhere" code for this. Input fields will be named differently, rules will vary and spam filters will trigger on different things. Even if you only have to spend an hour grooming your script for HN it seems like a lot of work to get 30 hits. You can of course reuse the script on the site once you've made it, but it won't take much before PG figures it out and bans IP-addresses, or some other measure.
trickjarrett 18y ago
I suggest we just add a recaptcha for people posting links below a certain karma threshold.
- chanux 18y ago
  I too was thinking of a captcha to block spam on HN. But then again thought it'd be a pain to the users. +1 for captcha for users under certain karma level. +1 for 2 karma to be eligible to post urls.
  - frossie 18y ago
    Also, since the OP said the spam was from accounts just created, you could add a third threshold (time since account creation before you can submit a story). By the way there is no reason to pick these thresholds out of thin air - presumably one can look at say the karma distribution of link submitters and figure out what the highest threshold that would inconvenience the least number of people is. For example, I would be truly surprised if 95% of stories were not submitted by people with a karma of 5 or more.
- froo 18y ago
  Instead of a captcha, what about something that is like a puzzle to complete, that way you can also avoid low quality submissions which might come from people not intelligent enough to solve said puzzles and would also be a delightful little game for the rest of us who like to solve puzzles/problems?
  - bendotc 18y ago
    First, I don't think I would be delighted by an automated puzzle on HN. Second, the idea that puzzle-solving-ability is somehow closely related to interesting/insightful writing is specious. People are smart in different ways, and as a programmer myself, the kind of smart that I want most is the kind I don't have, which is often not the problem-solving kind. All that having been said, I'm not against a captcha. I just don't think it'd be delightful and I don't believe that it'd somehow raise the quality of posts around here (beyond from removing some spam).
slater 18y ago
How about we keep with the intellectual stuff around here, and make 'em answer math captchas. Using latin numerals. eg, what's LV + IV? Answer has to be given in latin numerals, too.
- dhimes 18y ago
  We'd have to agree on how to write II + II We could ask for a basic derivative (for example, of a polynomial), or the value of 'x' in a simple algebraic equation x+5 = 2x + 10
  - phildawes 18y ago
    Cool - a challenge bots can't hope to beat. Oh wait... http://72.3.253.76/webMathematica3/quickmath/page.jsp?s1=equations&s2=solve&s3=basic http://72.3.253.76/webMathematica3/quickmath/page.jsp?s1=equ...
    - sketerpot 18y ago
      The point is to try to get the spammers to go someplace that's less work to spam.
  - cperciva 18y ago
    Problems like that are far too easy for computer programs to solve. Clearly the right solution here is to present new users with a set of Turing machines and ask them which of the specified machines halt. This would probably be very effective at limiting the growth rate of HN, too.
    - dhimes 18y ago
      I've never written a bot. I thought perhaps that the process of parsing instructions (we could put more than one variable in the equation), then writing code to solve, might be more effort than it's worth to them. Especially if we're doing something that is somewhat unique so their reward would only be one page. However, phildawes is apparently implying that once the bots have parsed the problem and know what to solve for they could go to a page that implements Mathematica and submit the problem to be solved, therefore saving the time required to write problem-solving code. I didn't realize they worked like that.
seejay 18y ago
Hope HN won't introduce anything similar to the burying system on digg which the powerful users on the site use for their advantage.
rythie 18y ago
There is an article on slate.com about CAPTCHAs http://www.slate.com/id/2216837/ http://www.slate.com/id/2216837/ (submitted here: http://news.ycombinator.com/item?id=579051 http://news.ycombinator.com/item?id=579051 )

52 points by jauco 18y ago

DarkShikari 18y ago

I saw an interesting method used by spambots on the Doom9 forum: they would take old threads and repost them (with a link to their ad in their signature or such). Some variant of this method might work to circumvent such a policy; find a similar thread, pick a comment from it, and post it, perhaps? It might only work 20% of the time, but that's good enough to get some spam accounts with URLs approved. Of course this would all take effort, and might be enough to lock out a lot of would-be spammers, or at least convince them to go somewhere else.

Tichy 18y ago
Reminds me of what I see on my Wordpress Blog: lot's of 0-content comments like "this is a cool article". At first they didn't make sense (blocked them anyway), but then I remembered the Wordpress settings for comments. By default, it is set to "commentors must have been approved at least once". So I guess if I would let one of those innocent comments through, they would be back with more serious stuff. Spammers are crafty...

ashleyw 18y ago

Good idea; though I'd argue 2 is too low, if these aren't "bots" but real humans, it wouldn't be hard for them to adapt by commenting and then upvoting each other. 10+ would be nicer, anyone truly interested in HN would understand why! :)

chanux 18y ago
Well.. I don't really agree with you. 10+ would be a bit painful I guess. Let's first keep the bots out. Then let's find a way to keep lamers away. There's always "the erlang week" to scare away lamers anyway.
talison 18y ago
It's a good point that some spam "bots" could be human. I was running a free email site when we saw a lot of strange account creation. We had recaptcha enabled and knew it hand't been cracked. It turned out (based on IP address analysis) that the accounts were being created by humans in the Philippines and then handed over to spammers in Dubai. Ah globalization... If you have an efficient spam vector, it's not unusual to see low-wage humans manipulating the system to get around captcha.
- jasonkester 18y ago
  This is a lot bigger than you'd expect. Nearly all the spam that makes it into the database on my sites is human-powered. It's maybe only 1% of the total attack volume, but because simple checks knock out nearly all the noise, it becomes the most significant fraction that you have to deal with.
  - ErrantX 18y ago
    limit it to 1 account creation per IP per hour :) Yeh they can use a ton of proxies to get round that but I bet it cuts the account creation right down. And it shouldnt affect 99.999999% of "real" users.
    - jasonkester 18y ago
      Ah, but there's the rub. Extrapolating from my comment above, your number one job is to make spammers feel successful when they fail. If you reject new accounts like this, you'll force them to write those little proxies to get around your system. The better thing to do is to simply notice what they're doing and flip the IsSpammer bit on all those new accounts (including the first one.) That way you can correctly classify any content they may post from those accounts in the future.

ComputerGuru 18y ago

That would encourage them to try to generate spam comments in the slim hope that they'd get upvoted.... which would lead to a much worse spam situation. For instance, they'd employ Markov Chains to re-hash comments in a post, or from older posts, or from the text of the article. One thing I've learned: spammers don't give up. If you give them an easy way to submit that's even easier for you to clean up, that's better than starting an all-out war that'll just increase the spam content and make it more difficult to filter out.

CalmQuiet 18y ago
Thank you: I needed to hear that. Part of my brain likes to keep scheming about "more absolute" ways to keep them out (and thus avoid any clean up). Your points suggest that my utopia-philic thinking is probably less useful than accepting that some clean up is just a necessary overhead (cf: "Eternal vigilance is the price of liberty."). Thank you.
mixmax 18y ago
Would a spammer really go through that just to submit a spam link that probably wouldn't get beyond the new page anyway? It's quite a bit of work, even if you have the same framework that you can use across multiple sites. My experience is that a post that doesn't get upvoted will receive around 30 hits. Surely there are easier ways of getting 30 hits.
- mooism2 18y ago
  They only have to write the code once. Then they can get 30 hits to a different site every day.
  - mixmax 18y ago
    I don't think it's feasible to make "write once run anywhere" code for this. Input fields will be named differently, rules will vary and spam filters will trigger on different things. Even if you only have to spend an hour grooming your script for HN it seems like a lot of work to get 30 hits. You can of course reuse the script on the site once you've made it, but it won't take much before PG figures it out and bans IP-addresses, or some other measure.

trickjarrett 18y ago

I suggest we just add a recaptcha for people posting links below a certain karma threshold.

chanux 18y ago
I too was thinking of a captcha to block spam on HN. But then again thought it'd be a pain to the users. +1 for captcha for users under certain karma level. +1 for 2 karma to be eligible to post urls.
- frossie 18y ago
  Also, since the OP said the spam was from accounts just created, you could add a third threshold (time since account creation before you can submit a story). By the way there is no reason to pick these thresholds out of thin air - presumably one can look at say the karma distribution of link submitters and figure out what the highest threshold that would inconvenience the least number of people is. For example, I would be truly surprised if 95% of stories were not submitted by people with a karma of 5 or more.
froo 18y ago
Instead of a captcha, what about something that is like a puzzle to complete, that way you can also avoid low quality submissions which might come from people not intelligent enough to solve said puzzles and would also be a delightful little game for the rest of us who like to solve puzzles/problems?
- bendotc 18y ago
  First, I don't think I would be delighted by an automated puzzle on HN. Second, the idea that puzzle-solving-ability is somehow closely related to interesting/insightful writing is specious. People are smart in different ways, and as a programmer myself, the kind of smart that I want most is the kind I don't have, which is often not the problem-solving kind. All that having been said, I'm not against a captcha. I just don't think it'd be delightful and I don't believe that it'd somehow raise the quality of posts around here (beyond from removing some spam).

slater 18y ago

How about we keep with the intellectual stuff around here, and make 'em answer math captchas. Using latin numerals. eg, what's LV + IV? Answer has to be given in latin numerals, too.

dhimes 18y ago
We'd have to agree on how to write II + II We could ask for a basic derivative (for example, of a polynomial), or the value of 'x' in a simple algebraic equation x+5 = 2x + 10
- phildawes 18y ago
  Cool - a challenge bots can't hope to beat. Oh wait... http://72.3.253.76/webMathematica3/quickmath/page.jsp?s1=equations&s2=solve&s3=basic http://72.3.253.76/webMathematica3/quickmath/page.jsp?s1=equ...
  - sketerpot 18y ago
    The point is to try to get the spammers to go someplace that's less work to spam.
- cperciva 18y ago
  Problems like that are far too easy for computer programs to solve. Clearly the right solution here is to present new users with a set of Turing machines and ask them which of the specified machines halt. This would probably be very effective at limiting the growth rate of HN, too.
  - dhimes 18y ago
    I've never written a bot. I thought perhaps that the process of parsing instructions (we could put more than one variable in the equation), then writing code to solve, might be more effort than it's worth to them. Especially if we're doing something that is somewhat unique so their reward would only be one page. However, phildawes is apparently implying that once the bots have parsed the problem and know what to solve for they could go to a page that implements Mathematica and submit the problem to be solved, therefore saving the time required to write problem-solving code. I didn't realize they worked like that.

seejay 18y ago

Hope HN won't introduce anything similar to the burying system on digg which the powerful users on the site use for their advantage.

rythie 18y ago

There is an article on slate.com about CAPTCHAs http://www.slate.com/id/2216837/ http://www.slate.com/id/2216837/ (submitted here: http://news.ycombinator.com/item?id=579051 http://news.ycombinator.com/item?id=579051 )