4 ms·

Ask.YC: Spammers have finally hit News.YC -- what do we do about them?

• mixmax 19y ago
  I was just about to submit an entry on this but you beat me to it :-) If it persists it should definitely be adressed - one of the great things about this place is the high signal to noise ratio and I would hate to see that drop. But since PG basically invented bayesian spam filtering I am sure that it can be solved. If he isn't too busy doing arc that that is. ;-)
  • deleted 19y ago
    [deleted]
  • hhm 19y ago
    Did he really? I thought he just proposed an improvement to it or something like that.
    • jcl 19y ago
      Wikipedia says that while other people worked on it earlier, PG's essay really popularized it: http://en.wikipedia.org/wiki/Bayesian_spam_filtering http://en.wikipedia.org/wiki/Bayesian_spam_filtering
      • deleted 19y ago
        [deleted]
    • mynameishere 19y ago
      When I first heard the term "Bayesian spam filtering" and its description, my response was, "Oh. I thought that's how they always did it." I didn't realize it was a big innovation.
• msteigerwalt 19y ago
  Easiest way is to just ask really simple technical questions in the submission area as a captcha. Not sure which technical questions would be 100% answerable to everyone around here, but it is Hacker News.
  • wallflower 19y ago
    Great idea. And not a bad one that would be applicable to CAPTCHAS as a whole. If non-technical. E.g. ask the user like 'What color is a US Dollar Bill'
    • ojbyrne 19y ago
      I've lived in the US, and I had to look at the dollar bills I happen to have in my wallet. Green, yes, but did they add other colors recently? Not a very foreigner-friendly question. The problem with technical questions, is that no matter how simple, people here will argue about the correct answer ;-). Unless we just give up and say that "It depends" is always correct.
  • shiro 19y ago
    What CAPTCHA emphasizes is that it is automated. Not only the verification process, but also the process of generating problems. If the problem space is fixed, eventually the list of answers is formulated if it is worth. In practice if spammer's think it's not worth we can outperform them by more problems, but that's kind of passive, giving up control to them. In general, what matters is asymptotic ratio of our cost vs spammer's cost---if it is both linear, it's not very promising. If the ratio reaches zero asymptotically, it means the more we and spammers compete the better we do, hence the control is ours.
• rms 19y ago
  pg's pretty good about killing spammer accounts as they come... if it becomes a problem, new users could have a probationary period of one approved submission before their submissions show up. and a reddit style filter on submission volume below a certain karma would prevent a spammer from filling up the new page.
• nreece 19y ago
  Btw, is there human moderation at News YC? If not, maybe provide the ability to the top 10 leaders to delete such spam links.
  • ijoshua 19y ago
    See those arrows? That’s human moderation. Just don’t click on the up arrow if you think it’s spam.
  • danw 19y ago
    Yup, I think pg mentioned before that there's a list of trusted users who can edit missleading titles, delete posts or change urls. (In the early days pdf links got changed to scribd links instead by moderators)
• fiaz 19y ago
  The simplest technique would be for the YC News gods to grant us a "report SPAM" button. The site would then be self moderating. I would also suggest something like granting karma points for sniffing out SPAM in case the YC News gods agree that a reported link does indeed turn out to be spam. Alternatively, a system of "open moderation" would be good as well (to lessen PG's load). In this case, moderation of the site would fall upon a rotating schedule between users that have reached a certain level of karma points. Like jury duty, you have a queue of sites that have been flagged for review, and the member's job is to merely approve of something as spam or not and have it dealt with. A link would have to be scrutinized by the random "committee" (which ideally should be a random percentage of YC News members on a regular schedule) and a majority vote would determine collectively if a link is spam or not. --------------- Apologies for making this post so long.... 1) have a "report" button 2) who ever is on "spam patrol" would see a reported link highlighted 3) "spam patrol" would have their "report" button substituted by a "confirm" button 4) after enough of the "spam patrol" confirms a link as spam, then the link is automatically marked for deletion The advantage of such a system is that there would be some filtration of the links and spammers would (hopefully) be discouraged. An "emergent" advantage would be to uncover spam sites. I can even imagine a number of social sites adopting a scheme creating a collaborative spam filter (ok, yes I'm spinning this out of control, but what the hell, I'm sure everybody here thinks like this!). This would be a social solution to a technical problem. If reddit had such a system, then a collaboration between YC News and reddit would pre-emptively eliminate confirmed spam sites from one another. Using this example, a number of social sites could immediately benefit from an open database of spam filtering (akin to the use of OpenID). So a confirmed spam site on reddit would automatically translate into a confirmed spam site on YC News, or whoever else subscribes to such an open database. Is this too complex of an idea???
  • jcl 19y ago
    It could be tricky to get right. For example, someone could censor a political blog from a social site interested in politics by spamming it to sites that are not interested in politics. And, of course, if this kind of filtering gets popular, spammers can simply set up a separate website for each submission to a social site.
• andr 19y ago
  negative karma should lead to a user ban.
  • thorax 19y ago
    Ouch-- I can't wait until the first YC lynch mobs begin.
• icey 19y ago
  Maybe this is a good time to start allowing negative karma and downvoting of submissions. If someone has negative karma, then they can't submit until they start making good comments at least. I know pg is worried about mass downvoting, and I think a good way around that is that downvoting anything costs one karma point for the person doing the downvoting. That way you only downvote things that really deserve it... And if you're a chronic downvoter, you wouldn't be able to do it for long as you'd hit Karma 0.
  • tjr 19y ago
    That will work only if the worthless submissions don't overpower us...
  • fiaz 19y ago
    The problem with down-voting is that there is no distinction between a spam report and judgmental calls on the content of a link. It changes the dynamic and the quality of links that make it to the top. Having recently come from reddit (and I must say I absolutely LOVE the energy of YC News!!!), I noticed that the down-voting makes submission more competitive and it makes the site less varied and more homogeneous.
    • icey 19y ago
      I disagree (to a degree). If enough people want to spend karma on a downvote then maybe the link doesn't really belong in hacker news anways (I'm looking at you, XKCD posts). If a submitter regularly submits garbage, how much different is it than spam? I mean, if a link is of low quality, maybe it ought to be downvoted & buried.
      • fiaz 19y ago
        But the fact that other quality links can be liberally up-voted by others would prevent low quality links from rising. The only advantage I can see with the down-voting is that it will make (subjectively) quality links rise faster and poor links sink faster - but this is not necessarily an advantage, at least I don't think it is (I could be TOTALLY missing your point here!).
• jsjenkins168 19y ago
  How about restricting the number of submissions allowed in a given period, scaled by karma? This would encourage people to only submit meaningful news, and it would also offer proactive protection against spam from low karma users. Just a suggestion.
• rob 19y ago
  What about adding a hidden field to the submission form where if it contains a value upon submission, it's obviously spam filled in by the bot (if these are bots and not actual people doing the submitting). Ryan Grove uses this technique on his blog and said it eliminated a lot of spam: "Yes. Riposte uses a very simple but very effective form of comment spam prevention: the comment form contains extra fields that are hidden from normal browsers via CSS but look like comment fields to spambots. When the form is submitted, if any of these fields contain content, Riposte denies the request. This technique has been in use for over a year at wonko.com and has been effective at stopping virtually all automated comment spam attempts."
  • dpapathanasiou 19y ago
    I also use this technique for SeekSift.com (I first saw it on Ned Batchelder's blog http://nedbatchelder.com/text/stopbots.html http://nedbatchelder.com/text/stopbots.html ), with good results (no spam bot signups so far).
  • paulgb 19y ago
    These are likely human spam posts, since a user account is required to submit. (edit: reading the comments, maybe spam bots are getting more sophisticated than I thought)
• edw519 19y ago
  PermissionToSubmit = (Karma > x) * (AccountDaysOld > y)
  • fiaz 19y ago
    This would incentivize some of the insidious viral marketers that leverage social news sites to create fake personas out there and eventually masquerade submissions for profit... that is if YC News faces the problem of large numbers. Also makes YC News elitist as well as shuts out new members, which I think would defeat the purpose of such a site. Fresh blood helps the mental juices flow as do varied ideas.
    • wastedbrains 19y ago
      you could only be allowed to submit X number of stories a day based on your Karma... So as you earn some respect you can do more. If you start with 1 you could only post 1 article a day which would make it less worth while to spammers. Or add flagging on users and if it passes some threshold it has the admins review the account and take action if it really is spam. Just two thoughts off the top of my head
      • fiaz 19y ago
        Some users would get too powerful in the long run. These users would be the head of a long tail of submissions and could have disproportionate influence. Of course, this depends upon YC News having the problem of a large user base. Also, I could be totally wrong in this imagination simulation...
    • edw519 19y ago
      Good points. I just assumed that the natural progression was: lurker, voter, commenter, submitter, addict.
      • fiaz 19y ago
        You are actually right on the money with this progression. Also, YC News is small enough (thankfully) that we don't have much of a spam problem. I'm just attacking our current situation as a global/general spam problem that social sites have. Having an opportunity to brainstorm gives me SO much enjoyment!
• johnrob 19y ago
  Spam blocking is probably a 1 liner in Arc.
  • davidw 19y ago
    "When Arc gets ahold of spam, it turns it into prose worthy of Shakespeare". "Arc doesn't need to block spam, because spam is too afraid to get anywhere near Arc".
    • kirubakaran 19y ago
      Arc creates classic lit. spam? :-) http://www.google.com/search?q=classic+lit+spam http://www.google.com/search?q=classic+lit+spam
• dkokelley 19y ago
  Short term solution: Ban account 1podgirl. I like the idea of prohibiting posting of articles until karma reaches a certain threshold, and limiting the amount of comments/hour by karma. So, for karma < 10 comments/hour = 1 for karma >= 10 comments/hour = karma/10
  • pchristensen 19y ago
    Heaven help us when nickb goes sour and hits us with 15 spam articles a minute!
    • iamelgringo 19y ago
      Muahahahaha... MUahahahaha. * wrings hands evilly *.
  • curi 19y ago
    i don't think you should force people to post comments. just rate limit their link submissions until they get 50 karma.
    • kajecounterhack 19y ago
      I agree...actually nobody should be allowed to submit 10 in an hour!
  • dmoney 19y ago
    An alternative to a normal ban would be to put identified spam accounts into a sandbox mode, where it looks to the account holder like its submissions are working, but everybody else doesn't see them. That way it doesn't start seeing errors and think, "Oh, time to create another account."
• jdavid 19y ago
  what about posts costing you karma? maybe the post is 1 karma, and each link in the post is an additional karma point. if it is a post of value, then you should earn at least more than 1 karma to replace it. it makes your post an investment of karma. I think its a good lesson for entrepreneurs too.
  • wastedbrains 19y ago
    I think that is a really interesting idea. The only issue would be if you hit 0 on your first post you could never post again. So if your at 0 you can post but only once per day or something until you post something that earns you karma. It is a simple system but seems like it could be very effective. Kudos
    • cstejerean 19y ago
      Best suggestion I've heard so far.
  • manvsmachine 19y ago
    Good idea, but how about instead of posts costing you karma, spam costs you karma. Then if your karma goes negative, you cannot post until it becomes positive again. That way it doesn't penalize new users with low karma, but it will only take two bad posts to render a spam account ineffective.
  • tlrobinson 19y ago
    I like this idea, it's very much like the #xkcd-signal experiment, designed to force interesting conversation (especially a problem in certain IRC channels...) http://blag.xkcd.com/2008/01/14/robot9000-and-xkcd-signal-attacking-noise-in-chat/ http://blag.xkcd.com/2008/01/14/robot9000-and-xkcd-signal-at... It will also make non-spammers think twice about posting useless comments or submissions (assuming they either have low karma or actually care about their karma level)
  • hooande 19y ago
    The problem with this is that it's too easy for spammers to make new accounts for every post.Then all you do is limit the number of posts (which may or not be a good thing) without doing much to deter spam.
• yters 19y ago
  I think pg has something in place already, though it takes a little time to work: http://news.ycombinator.com/item?id=99975 http://news.ycombinator.com/item?id=99975
• Kaizyn 19y ago
  We're only in trouble here if PG and the others used to implicitly moderate traffic are likely to upvote those links. Otherwise, we can safely ignore them. Adding a 'report submission' button might be prudent as it would make the process of eliminating junk links more efficient.
• misterbwong 19y ago
  This was probably mentioned in some form before, but what about a combination user spam button/automatic temp-ban? If the last X posts by user were marked as spam, ban account for X days. Two temp bans = perm ban.
• pg 19y ago
  Spammers hit News.YC the day we launched. There is spam more days than not. Usually the editors catch these before anyone sees them.
  • fiaz 19y ago
    Is spamming much of a problem for the editors? I suppose I should have asked this when you posted instead of going on my (useful?) brainstorm below (and I really was just throwing out some ideas)...
    • pg 19y ago
      Not huge. There's not a lot of it, and we have several tools for dealing with it. It was unusual for those spams to sit for 3 hours on the new page without getting cleaned up.
      • deleted 19y ago
        [deleted]
• alaskamiller 19y ago
  I don't know... don't vote them up?
• edu 19y ago
  A captcha on the register screen?
• DanielBMarkham 19y ago
  When I first started blogging, I wanted people to come read my great words of wisdom (naturally. Why am I not a god of publishing yet?!?) So I hit on the idea of tagging my Slashdot posts with a line at the bottom plugging whatever I was blogging about that day. I made it a point to post interesting relevant comments to Slashdot, but at the bottom I would always ham it up with stuff like "Giant Sponge Moon Found Orbiting Saturn!" People gave my comments 5 ratings almost all the time, but it really, really pissed off a group of \. readers. Was that spam? I'm still not sure the answer to that question. I know IpodGirl is a spammer, but there are all kinds of edge cases too.
• jgrahamc 19y ago
  Fundamentally, this sort of spam is about getting links to their sites (first for direct traffic and second for PageRank). So, you should attack the links and not the accounts: 0. Check your URLs against blog spam blacklists. Exclude recently registered domains (all of the ones mentioned are less than one month old). 1. Links on stories that have been become popular should be real hyperlinks. 2. Links on unpopular stories should go through a news.yc redirector (since this means they don't get PageRank) 3. Stories that go nowhere (likely spam) have their links removed and a text version of the link placed next to the story name for anyone who really wants to follow that link. 4. A 'spam' button could be handy.
• deleted 19y ago
  [deleted]