9 ms·

A painful tale of SEO, spam and Google's role in all

203 points by slaven 14y ago

phasevar 14y ago
This is a great post that I'm sure rings true with many 'a webmaster. Time is money. There's no two ways about it. If you want a webmaster to modify their site in your favor, you should be willing to compensate them for their time in doing so.
speeder 14y ago
This is sad, sad, sad. I wonder how much awesome communities died that way. I stopped visiting Orkut when communities there got overran with spam. Also the same apply to some Usenet groups and Google Groups I used to like.
mutagen 14y ago
I saw the SEO post he's referring to (it was one of the articles in SEOMoz's top 10 monthly email). Site owners don't realize the amount of work and headache they cause forum communities when they contract out SEO work without an understanding of what that work entails. Or they do and just don't care. Google's Penguin update didn't deter the spammers, either. Here we are nearly a year later and I'm still cleaning out accounts created en masse by XRumer or other bots.
jedireza 14y ago
You could shoot the hostage and add rel='nofollow' to all outbound links. You should be able to programmatically do this. Then with an auto-reply (or 'link policy' page) inform 3rd party sites that the link-juice (good or bad) is no longer flowing.
- Scramblejams 14y ago
  Does this constitute a long-term solution to the problem for active communities? That is, default to rel='nofollow' for an untrusted class of users, which presumably includes all the spammers?
  - 33degrees 14y ago
    that only works if the spammers realize their links are no longer effective; I'm not certain that that will actually happen...
    - tracker1 14y ago
      If you have a relatively small site.. curating content isn't too bad... limiting the tail of articles so that comments can only be added for X days helps with that. Only allowing logged in users to comment (and making login easy via oauth sites like facebook/twitter/g+ etc) I kind of like Disqus, but would rather have something more home-grown.
  - tracker1 14y ago
    I'd say rel=nofollow ALL user generated content.. but hey, that's just me.
  - jedireza 14y ago
    Not at all. I was addressing the fact that when a 3rd party site asks for links to be removed (and are un-willing to pay for 'time' it takes to take links down) and then threaten to turn you into the google police for link spam... you can defend your position and show that you indeed are not participating in link shenanigans.
- kalleboo 14y ago
  We did this, it made no difference. What finally worked was a combo of: custom javascript on the signup form, disallowing new users from posting links in posts or signatures (and auto-banning ones who continually made attempts), insta-banning users who filled out non-linkedin links in the "linked-in link" profile field. We still get spam, but now it's maybe one post a week, instead of 10-20 a day, so it's at a level that moderators/user reports can handle it.
- tronalddump 14y ago
  Rel=no follow discourages spam due to PR flow, but won't stop it. It still has an influence, however diminished. Take an hour or two and write a script that selects all users with less than 100 posts and iterates through their posts removing links. The integrity of actual users is maintained while 99% of spam links eliminated. You spend a total of 2 hours, but don't have to deal with the emails any more - which could save many more hours.
  - ozi 14y ago
    No decent spammer cares about whether or not the links are nofollow. Targeting just dofollow links and then having a backlink portfolio to your site that is 99%+ dofollow links looks completely unnatural.
arn 14y ago
I've received similar emails. I've also been tempted to charge a fee for removal. I just ignore the emails.
javajosh 14y ago
The problem here is the work. The solution is to write a program. Have the white hat SEO people write a program that spiders the site with admin privileges and removes offending posts. It should come with a "dry run" mode that lets you spot check it. When they get it right, you can run it for them. It's a win-win: your forums get cleaned up and they did the work.
- bradleyjg 14y ago
  It's not really a win-win. Any time he spends on this is a loss from his point of view since the SEO's client already ruined his forum. It's some chutzpah to get annoyed when your victim refuses to help you hide your misdeeds.
  - javajosh 14y ago
    If they do the work to undo the sins of the past, it helps the OP recover the value of his forums for far less effort than if he'd undertaken the job himself. I'd call that a win.
    - argonaut 14y ago
      No, removing all the spam is not going to recover the community that fled long ago.
    - TeMPOraL 14y ago
      But why would the owner want to do that? By keeping those links up, the author is providing value to the Internet by lowering PR of folks who paid to spam him in the first place. I see no incentive for the author to clean this up.
kkowalczyk 14y ago
Blaming google for seo spam is not productive - spammers are legion and in constant battle to game google's ranking system, whatever it currently is. This won't change as long as google is used to search for things. There is a solution to his particular problem: better forum software. I don't want to trivialize the problem of writing spam-resistant forum software but it's not such an insurmontable problem (this forum being a proof positive for that). For the reference: I've been running a fairly popular forum (http://forums.fofou.org/sumatrapdf http://forums.fofou.org/sumatrapdf) for several years, using forum software that I wrote. I don't even require the user to log in, I don't require moderation for posts and yet I had zero automatic spam (I attribute this to my unusual captcha http://blog.kowalczyk.info/article/16fw/Best-captcha-is-exotic-captcha.html http://blog.kowalczyk.info/article/16fw/Best-captcha-is-exot...). I occasionally get human spam i.e. someone writes a post with the only purpose of linking to some other website. I just hellban them after I see the post in my rss reader. And I didn't even write any special anti-spam code (other than hell-banning), because I'm lazy. I can easily come up with simple ideas e.g. putting all posts that contain links in moderation queue. To reiterate: his problems were caused by a crappy forum software that didn't do much to protect from seo spam. I don't really know how current best off-the-shelf forums fare in this respect. I would rather not spend my time maintaining my own forum software so I have high hopes for http://www.discourse.org http://www.discourse.org. I'm sure StackOverflow had plenty of spam problems so discourse people should understand the problem.
- greggman 14y ago
  I'd attribute it to your custom software. Break phpBB and you can infect thousands of sites. Break your custom software you get to infect 1 site. I ran a forum using SimpleMachines. Turned on all the spam fighting stuff. Didn't matter. Bots or Mechanical Turks made 10-30 new accounts a day. I tried moderating the accounts but who wants to spend time every day to delete the fefRFGR34@foobar.com account. I gave up and shut it down.
  - skore 14y ago
    To be fair, though, SimpleMachines is a very atrocious piece of software (not to mention their equally atrocious licensing, although it seems they're in the process of changing that[0]...), so it's no wonder. I could say similar things about phpBB, but that's quite a dead horse to beat (hey, at least they use a proper license). [0] http://www.simplemachines.org/about/opensource.php http://www.simplemachines.org/about/opensource.php
coderdude 14y ago
One way to combat this problem is to deny new users the ability to post links by replacing the link text with [removed]. HN has a karma system with thresholds that must be passed in order to get additional privileges (like down-voting). If you have a site that assigns karma to users (even if it's a secret number) then you can set a threshold for allowing links to pass through. It's not perfect but it's better than letting brand new accounts post links, IMO. Another solution is to hide all links when a page is viewed by a user who isn't logged in.
- markdown 14y ago
  > Another solution is to hide all links when a page is viewed by a user who isn't logged in. No, this is almost as bad as what medium.com does with all its links (rewriting them to point inwards towards redirects for tracking purposes). A web link should be respected.
  - coderdude 14y ago
    > (rewriting them to point inwards towards redirects for tracking purposes) I like this idea a lot. It has more than one use. Tracking is one but eliminating the "PR bleed" is another and it should deter spammers who are looking for link weight. I understand the sentiment you're conveying but a link isn't sacred. If you allow users to post content to your site then you should be taking measures to ensure that new users aren't spamming. I don't think the "go in and clean it up later" method cuts it.
    - markdown 14y ago
      > I understand the sentiment you're conveying but a link isn't sacred. It damn well should be. The link is the basis of the www. > If you allow users to post content to your site then you should be taking measures to ensure that new users aren't spamming. Of course, but breaking legitimate links as a form of spam control is just lazy. We're going OT here, but... > eliminating the "PR bleed" There is no such thing. PR doesn't work this way, and the existence of outward links doesn't negatively impact the linking website in any way.
Nikolas0 14y ago
I had the same problem myself. In fact I still run a couple of forums. And I get at least 3 "remove my links" mails per day. I guess you can't ask them money, but they already cost you money (to maintain their spam and since the panda update read their emails asking you to remove it) so I guess the best solution is just ignore those mails. Regarding anti spam I am afraid there is nothing you can do. Real humans will create an account for 0.01$ and they will post anything (I tried adding custom code in the post code as well, but those signups are not necessarily bots) In fact they even post if you don't allow them to post links (they post the urls with no http) Next solution I'll try is social integration. Maybe that would work for some time but even this way spammers will find their way to create thousands of crap accounts in FB, twitter, etc.
venus 14y ago
My god, the quote from that "white hat" SEO guy just defies belief: > There will certainly be webmasters out there who will strip you down to the bone asking for money in exchange of link removals. These are the most soulless snake oil salesmen on earth To say that about webmasters, already victims of years of abusive SEO spamming, when they then refuse to help an abusive site clean up its own mess for free .. I have no words. Could anyone possibly be more of a self-interested, myopic, egocentric prick? My rock-bottom opinion of pretty much anyone who has anything to do with active SEO is re-confirmed for the thousandth time.
- taf2 14y ago
  it's easy to judge
  - JoachimSchipper 14y ago
    Yes. Yes, it is.
- halcyondaze 14y ago
  Don't hate on all SEOs just because of the overall clown-ness of the space :( there are some good guys!
  - GhotiFish 14y ago
    Sounds like their job descriptions requires they not be "good guys" Googles heuristic would work be much more effective if people didn't screw with it.
    - halcyondaze 14y ago
      Not true, there's some very cool stuff you can do that is entirely within Google's guidelines AND gives your site a large advantage over other sites that aren't using them. See http://www.quicksprout.com/2013/01/24/the-advanced-guide-to-seo/ http://www.quicksprout.com/2013/01/24/the-advanced-guide-to-...
codezero 14y ago
I'd be tempted to ask for money as well, but I feel like at the very least, you should ask for an apology. They may not have known that their SEO agency was using shady tactics, but since times changed, and it's obvious that they put bad links out there, it seems like an apology would be more than a token gesture.
- deleted 14y ago
  [deleted]
8ig8 14y ago
How about taking inspiration from D&B[1]? Don't charge for link removal; charge for priority link removal. Same day service: $1,000 per link. One week: $500. Etc. Free link removal: First-come, first-serve at your own leisurely pace. Sell the old forum to someone else and let them handle the requests. BTW, PocoMail was a godsend back in the day. Thanks. [1]:http://news.ycombinator.com/item?id=4994246 http://news.ycombinator.com/item?id=4994246
- slaven 14y ago
  Props on PocoMail comment :)
halcyondaze 14y ago
I use XenForo and it's relatively simple to stop all forum spam for the most part. I just set up a custom captcha that only people who are interested in the topic would know the answer to, and then use XenForo's built in spam catching system for the rest...though I haven't had anything get through yet once I made the changes. Also deleted forum footprint to dodge people scraping my forum off of Google, so that takes care of about 99% of all spam attempts. Someone would have to custom register accounts on my forum to get in, after which they would be destroyed by the spam catcher :)
hakaaaaak 14y ago
If your business relies on SEO or is affected by Google search rank in a critical way, it may not be a viable business long-term. Yes, that means I'm talking about a lot of online businesses. The intelligent thing to do is sell a product or service that has value on its own and neither relies on SEO nor is it likely to be blames by other sites or companies for lowering their SEO. I feel bad for these folks, but if you are planning on starting a business that doesn't really provide much value on its own that is identifyable outside of the roach motel of SEO, then you are headed into the ocean in a dingy with a small outboard motor, imo.
Father 14y ago
Link removal requests can also be malicious. A blackhat seo will check his competitors domain to see if there's a spf record, if they're signing their mail, and if there's a catch-all (simply by checking if random mail is accepted). If there's neither they get a list of backlinks, from public web-crawls or sites like ahref, and request these links be taken down by sending mail with spoofed email addresses.
BashiBazouk 14y ago
Here is a question for those who know: is there anything forum regulars can do when we see spam posts beyond letting mods know about it? I was active in a now dead forum that would get hit once or twice a week. The mods would clean it up with in a half a day but until then those posts would just sit. Does the Google web crawler consider words in replies to the spam post? I used to reply occasionally with words like: scam, fraud, got ripped off. I have always wondered if I was wasting my time. A few times I checked the link to see if it went some where legitimate (Google the base url), if so, I then searched the site for an informational web form. If there was one, left a message that their SEO company was using sleazy methods with a link to the forum post. Google ought to make a code phrase that forum users can use to red flag spam posts. Though some of the posts were for Japanese and Chinese sites. The spam might not have been meant for Google but other search engines...
rurounijones 14y ago
Would love to hear feedback from the white-hat to this article. Maybe get a decent discussion going from both sides. It does seem that the white^hat's remarks are very harsh, I wonder if he has honestly never thought about it from the other side's point of view.
nkuttler 14y ago
So this "SEO" who helping his clients to "clean up their profiles" doesn't even know how to disavow links [1]. Well, it's not surprising that somebody who hired spammers once would hire another idiot later. [1] http://www.youtube.com/watch?v=393nmCYFRtA http://www.youtube.com/watch?v=393nmCYFRtA and https://www.google.com/webmasters/tools/disavow-links-main https://www.google.com/webmasters/tools/disavow-links-main (login required)
- JimWestergren 14y ago
  Well Google requires you to contact the website owners and do everything you can to get the links removed BEFORE you submit a list of disavow links. Google are supposed to check that you really did remove a large part of the links before disavowing.
bambax 14y ago
There must be a programmatic solution to this problem. Do some outbound links have value on this forum? If not, then you could remove all links, or remove the "link" part of the link (change @href to text). If some outbound links have value you need to identify those, and it's more complex, but a Bayesian analysis of posts should be able to score posts on their "spaminess" and remove the links on only the most spam-like comments. There may be some false-positive doing this, but since no information is actually removed (only the links, not the content) it should be quite ok.
- robryan 14y ago
  Random links on forums already have pretty low value. I would assume that forum spam has come down now that this is the case. As the article states, websites are looking the clean up the spammy inbound links as google has threatened to deindex sites that don't make an effort to do this.
will_critchlow 14y ago
I can't put words in the mouth of the SEO quoted, but (unless you know he was one of the people making removal requests against your site) I suspect he is not talking about you or sites like yours. I should add that I don't agree with the rhetoric btw, but I think he is targeting a different kind of webmaster. I think he is referring to webmasters who sold links (knowingly outside the guidelines) for years. They would previously have instantly removed the links if someone stopped paying. As soon as Google stepped up their game and removing those links was important, those same webmasters started charging to take down the same links. I would personally point to the irony of this (google creating a market that enriches people who have been abusing their system for years) rather than calling it immoral. Ymmv. Hope that helps clarify some things and I hope I'm not distorting the guy's real meaning. (written on my phone. Please excuse typos).
- dhimes 14y ago
  I'm wondering about the graph on Gary's site: what type of business' fortune is linked this tightly to Google's rankings? Is it a particular category, or are there several categories of businesses that suffer from (or leverage) this? That's a pretty frightening situation to be in, IMO.
  - will_critchlow 14y ago
    Oh, there are many businesses at least that exposed. I personally wouldn't particularly relish running one, but that's another story...
lutusp 14y ago
The solution is obvious -- write a routine that automatically goes through the entire forum database and disables all the links -- leave the names, but rewrite the links so they're just text, not hyperlinks. Sort of like: s!http://!!g http://!!g The above deletes the "http:// http:// prefix, but leaves the original destination name, in case anyone wants to object that their post has been edited after the fact. So technically, it's no longer a link back to the originating site, but it's otherwise unchanged. No human intervention required. Problem solved.
ajenner 14y ago
Wow, so Google penalizes a site if a link to that site shows up in spammy pages? That seems like a new business model for black-hat SEOers: "hey, nice site you have there - it'd be a shame if links to it started appearing all over my spammy network - $$$ will make sure that doesn't happen." Really search engines should just give zero weight rather than negative weight to links from spammy sites.
- Isofarro 14y ago
  "hey, nice site you have there - it'd be a shame if links to it started appearing all over my spammy network" Good way to clue Google in to this network, and then drop it from the index. "Really search engines should just give zero weight rather than negative weight to links from spammy sites" That's what was happening before Penguin/Panda - and it wasn't working. There's no risk to link spamming. Now there is.
iuguy 14y ago
Does anyone else think that the author should set up something like mailchimp with an autoresponder to deal with the spam? Something that explains his policy, why he doesn't respond to emails, why he won't remove the link without charging them? Many of the people who will have these spammy links on his forum will have paid one of the more scummy SEOs out there to raise their SERPs. The people emailing may not have originally been aware that the links have been put up. By having an autoresponder address for people to mail to this should alleviate the spam for all but the dumbest of people (that can't read what to sign up to to get a response, or those that fail to read a clearly defined policy via email).
rizz0 14y ago
There's a difference between buying links, which Google actively started prohibiting at a certain point, and the automated forum spam you are mentioning. The latter is done using software like XRumer, and has always been penalized by Google. The former wasn't prohibited in the beginning, when link deals were often mutual agreements, most of the time involving a traffic component as well. The snake oil salesmen that are mentioned, are the ones who actively participated in the scheme by selling links and making a buck, and are trying to make another quick buck now that the rules have changed. Moreover, some links aren't even paid at all, but just look manipulative. For example, if you developed a wordpress theme, and your link is in the footer of tons of blogs, you might get penalized for manipulating the anchor text of your links in a non-natural way. In those legitimate situations, webmasters do have a moral obligation to cooperate. I don't think anyone would think that of an honest entrepreneur being spammed to death by link spamming software. On a side note, there are plenty of forums on the web that have survived the spam wave, if it were core to your business, you could have protected yourself.
Tichy 14y ago
Maybe it could be scripted? Site owners would have to place some proof of ownership on their site (to be generated on the site of the forum owner - see Google web admins site verification), then a script could detect the faulty links on the forum and remove them automatically.
JimWestergren 14y ago
Place a meta noindex, nofollow on the whole forum or at least the infected areas such as memberlist and user profiles. That could be done within 15 minutes and then you can answer with a standard reply that you did that to all emails you receive about removing links.
mangostache2 14y ago
Why are people getting bent out of shape over some bad links? Can't you just disavow them in Google Webmaster Tools?
Aardwolf 14y ago
There exist very popular forums that receive no such spam. Hacker News comments also seem free from it. Any idea why it is that some forums get targeted with 10000's of spam accounts, while others don't? What is the trick to protecting if even capchas don't work?
- mesozoic 14y ago
  They mainly use a program called xrumer to spam and it mostly works with your typically install of vbulletin or other popular forum software so custom software won't get hit. Kinda security by obscurity.
edwinyzh 14y ago
Maybe we should not blame search engines, but SEO really has bad effects on the Internet... PS, due to spams I've closed this forum (http://writingoutliner.com/forum http://writingoutliner.com/forum), a online forum for my 'Addon documents organizer for MS Word'. And I regret I use BBPress as the forum software, because new 2.x versions are not compatible with the old 1.x versions...
mesozoic 14y ago
Oh man that is some sweet irony. Just wait till the next waive of spam from SEOs trying to get competitors penalized then the competitors firing back and contacting you. Google stirred up a hornets nest because they couldn't figure out how to effectively devalue these links completely.

203 points by slaven 14y ago

This is a great post that I'm sure rings true with many 'a webmaster. Time is money. There's no two ways about it. If you want a webmaster to modify their site in your favor, you should be willing to compensate them for their time in doing so.

speeder 14y ago

This is sad, sad, sad. I wonder how much awesome communities died that way. I stopped visiting Orkut when communities there got overran with spam. Also the same apply to some Usenet groups and Google Groups I used to like.

mutagen 14y ago

I saw the SEO post he's referring to (it was one of the articles in SEOMoz's top 10 monthly email). Site owners don't realize the amount of work and headache they cause forum communities when they contract out SEO work without an understanding of what that work entails. Or they do and just don't care. Google's Penguin update didn't deter the spammers, either. Here we are nearly a year later and I'm still cleaning out accounts created en masse by XRumer or other bots.

jedireza 14y ago

You could shoot the hostage and add rel='nofollow' to all outbound links. You should be able to programmatically do this. Then with an auto-reply (or 'link policy' page) inform 3rd party sites that the link-juice (good or bad) is no longer flowing.

Scramblejams 14y ago
Does this constitute a long-term solution to the problem for active communities? That is, default to rel='nofollow' for an untrusted class of users, which presumably includes all the spammers?
- 33degrees 14y ago
  that only works if the spammers realize their links are no longer effective; I'm not certain that that will actually happen...
  - tracker1 14y ago
    If you have a relatively small site.. curating content isn't too bad... limiting the tail of articles so that comments can only be added for X days helps with that. Only allowing logged in users to comment (and making login easy via oauth sites like facebook/twitter/g+ etc) I kind of like Disqus, but would rather have something more home-grown.
- tracker1 14y ago
  I'd say rel=nofollow ALL user generated content.. but hey, that's just me.
- jedireza 14y ago
  Not at all. I was addressing the fact that when a 3rd party site asks for links to be removed (and are un-willing to pay for 'time' it takes to take links down) and then threaten to turn you into the google police for link spam... you can defend your position and show that you indeed are not participating in link shenanigans.
kalleboo 14y ago
We did this, it made no difference. What finally worked was a combo of: custom javascript on the signup form, disallowing new users from posting links in posts or signatures (and auto-banning ones who continually made attempts), insta-banning users who filled out non-linkedin links in the "linked-in link" profile field. We still get spam, but now it's maybe one post a week, instead of 10-20 a day, so it's at a level that moderators/user reports can handle it.
tronalddump 14y ago
Rel=no follow discourages spam due to PR flow, but won't stop it. It still has an influence, however diminished. Take an hour or two and write a script that selects all users with less than 100 posts and iterates through their posts removing links. The integrity of actual users is maintained while 99% of spam links eliminated. You spend a total of 2 hours, but don't have to deal with the emails any more - which could save many more hours.
- ozi 14y ago
  No decent spammer cares about whether or not the links are nofollow. Targeting just dofollow links and then having a backlink portfolio to your site that is 99%+ dofollow links looks completely unnatural.

arn 14y ago

I've received similar emails. I've also been tempted to charge a fee for removal. I just ignore the emails.

javajosh 14y ago

The problem here is the work. The solution is to write a program. Have the white hat SEO people write a program that spiders the site with admin privileges and removes offending posts. It should come with a "dry run" mode that lets you spot check it. When they get it right, you can run it for them. It's a win-win: your forums get cleaned up and they did the work.

bradleyjg 14y ago
It's not really a win-win. Any time he spends on this is a loss from his point of view since the SEO's client already ruined his forum. It's some chutzpah to get annoyed when your victim refuses to help you hide your misdeeds.
- javajosh 14y ago
  If they do the work to undo the sins of the past, it helps the OP recover the value of his forums for far less effort than if he'd undertaken the job himself. I'd call that a win.
  - argonaut 14y ago
    No, removing all the spam is not going to recover the community that fled long ago.
  - TeMPOraL 14y ago
    But why would the owner want to do that? By keeping those links up, the author is providing value to the Internet by lowering PR of folks who paid to spam him in the first place. I see no incentive for the author to clean this up.

kkowalczyk 14y ago

Blaming google for seo spam is not productive - spammers are legion and in constant battle to game google's ranking system, whatever it currently is. This won't change as long as google is used to search for things. There is a solution to his particular problem: better forum software. I don't want to trivialize the problem of writing spam-resistant forum software but it's not such an insurmontable problem (this forum being a proof positive for that). For the reference: I've been running a fairly popular forum (http://forums.fofou.org/sumatrapdf http://forums.fofou.org/sumatrapdf) for several years, using forum software that I wrote. I don't even require the user to log in, I don't require moderation for posts and yet I had zero automatic spam (I attribute this to my unusual captcha http://blog.kowalczyk.info/article/16fw/Best-captcha-is-exotic-captcha.html http://blog.kowalczyk.info/article/16fw/Best-captcha-is-exot...). I occasionally get human spam i.e. someone writes a post with the only purpose of linking to some other website. I just hellban them after I see the post in my rss reader. And I didn't even write any special anti-spam code (other than hell-banning), because I'm lazy. I can easily come up with simple ideas e.g. putting all posts that contain links in moderation queue. To reiterate: his problems were caused by a crappy forum software that didn't do much to protect from seo spam. I don't really know how current best off-the-shelf forums fare in this respect. I would rather not spend my time maintaining my own forum software so I have high hopes for http://www.discourse.org http://www.discourse.org. I'm sure StackOverflow had plenty of spam problems so discourse people should understand the problem.

greggman 14y ago
I'd attribute it to your custom software. Break phpBB and you can infect thousands of sites. Break your custom software you get to infect 1 site. I ran a forum using SimpleMachines. Turned on all the spam fighting stuff. Didn't matter. Bots or Mechanical Turks made 10-30 new accounts a day. I tried moderating the accounts but who wants to spend time every day to delete the fefRFGR34@foobar.com account. I gave up and shut it down.
- skore 14y ago
  To be fair, though, SimpleMachines is a very atrocious piece of software (not to mention their equally atrocious licensing, although it seems they're in the process of changing that[0]...), so it's no wonder. I could say similar things about phpBB, but that's quite a dead horse to beat (hey, at least they use a proper license). [0] http://www.simplemachines.org/about/opensource.php http://www.simplemachines.org/about/opensource.php

coderdude 14y ago

One way to combat this problem is to deny new users the ability to post links by replacing the link text with [removed]. HN has a karma system with thresholds that must be passed in order to get additional privileges (like down-voting). If you have a site that assigns karma to users (even if it's a secret number) then you can set a threshold for allowing links to pass through. It's not perfect but it's better than letting brand new accounts post links, IMO. Another solution is to hide all links when a page is viewed by a user who isn't logged in.

markdown 14y ago
> Another solution is to hide all links when a page is viewed by a user who isn't logged in. No, this is almost as bad as what medium.com does with all its links (rewriting them to point inwards towards redirects for tracking purposes). A web link should be respected.
- coderdude 14y ago
  > (rewriting them to point inwards towards redirects for tracking purposes) I like this idea a lot. It has more than one use. Tracking is one but eliminating the "PR bleed" is another and it should deter spammers who are looking for link weight. I understand the sentiment you're conveying but a link isn't sacred. If you allow users to post content to your site then you should be taking measures to ensure that new users aren't spamming. I don't think the "go in and clean it up later" method cuts it.
  - markdown 14y ago
    > I understand the sentiment you're conveying but a link isn't sacred. It damn well should be. The link is the basis of the www. > If you allow users to post content to your site then you should be taking measures to ensure that new users aren't spamming. Of course, but breaking legitimate links as a form of spam control is just lazy. We're going OT here, but... > eliminating the "PR bleed" There is no such thing. PR doesn't work this way, and the existence of outward links doesn't negatively impact the linking website in any way.

Nikolas0 14y ago

I had the same problem myself. In fact I still run a couple of forums. And I get at least 3 "remove my links" mails per day. I guess you can't ask them money, but they already cost you money (to maintain their spam and since the panda update read their emails asking you to remove it) so I guess the best solution is just ignore those mails. Regarding anti spam I am afraid there is nothing you can do. Real humans will create an account for 0.01$ and they will post anything (I tried adding custom code in the post code as well, but those signups are not necessarily bots) In fact they even post if you don't allow them to post links (they post the urls with no http) Next solution I'll try is social integration. Maybe that would work for some time but even this way spammers will find their way to create thousands of crap accounts in FB, twitter, etc.

venus 14y ago

My god, the quote from that "white hat" SEO guy just defies belief: > There will certainly be webmasters out there who will strip you down to the bone asking for money in exchange of link removals. These are the most soulless snake oil salesmen on earth To say that about webmasters, already victims of years of abusive SEO spamming, when they then refuse to help an abusive site clean up its own mess for free .. I have no words. Could anyone possibly be more of a self-interested, myopic, egocentric prick? My rock-bottom opinion of pretty much anyone who has anything to do with active SEO is re-confirmed for the thousandth time.

taf2 14y ago
it's easy to judge
- JoachimSchipper 14y ago
  Yes. Yes, it is.
halcyondaze 14y ago
Don't hate on all SEOs just because of the overall clown-ness of the space :( there are some good guys!
- GhotiFish 14y ago
  Sounds like their job descriptions requires they not be "good guys" Googles heuristic would work be much more effective if people didn't screw with it.
  - halcyondaze 14y ago
    Not true, there's some very cool stuff you can do that is entirely within Google's guidelines AND gives your site a large advantage over other sites that aren't using them. See http://www.quicksprout.com/2013/01/24/the-advanced-guide-to-seo/ http://www.quicksprout.com/2013/01/24/the-advanced-guide-to-...

codezero 14y ago

I'd be tempted to ask for money as well, but I feel like at the very least, you should ask for an apology. They may not have known that their SEO agency was using shady tactics, but since times changed, and it's obvious that they put bad links out there, it seems like an apology would be more than a token gesture.

deleted 14y ago
[deleted]

8ig8 14y ago

How about taking inspiration from D&B[1]? Don't charge for link removal; charge for priority link removal. Same day service: $1,000 per link. One week: $500. Etc. Free link removal: First-come, first-serve at your own leisurely pace. Sell the old forum to someone else and let them handle the requests. BTW, PocoMail was a godsend back in the day. Thanks. [1]:http://news.ycombinator.com/item?id=4994246 http://news.ycombinator.com/item?id=4994246

slaven 14y ago
Props on PocoMail comment :)

halcyondaze 14y ago

I use XenForo and it's relatively simple to stop all forum spam for the most part. I just set up a custom captcha that only people who are interested in the topic would know the answer to, and then use XenForo's built in spam catching system for the rest...though I haven't had anything get through yet once I made the changes. Also deleted forum footprint to dodge people scraping my forum off of Google, so that takes care of about 99% of all spam attempts. Someone would have to custom register accounts on my forum to get in, after which they would be destroyed by the spam catcher :)

hakaaaaak 14y ago

If your business relies on SEO or is affected by Google search rank in a critical way, it may not be a viable business long-term. Yes, that means I'm talking about a lot of online businesses. The intelligent thing to do is sell a product or service that has value on its own and neither relies on SEO nor is it likely to be blames by other sites or companies for lowering their SEO. I feel bad for these folks, but if you are planning on starting a business that doesn't really provide much value on its own that is identifyable outside of the roach motel of SEO, then you are headed into the ocean in a dingy with a small outboard motor, imo.

Father 14y ago

Link removal requests can also be malicious. A blackhat seo will check his competitors domain to see if there's a spf record, if they're signing their mail, and if there's a catch-all (simply by checking if random mail is accepted). If there's neither they get a list of backlinks, from public web-crawls or sites like ahref, and request these links be taken down by sending mail with spoofed email addresses.

BashiBazouk 14y ago

Here is a question for those who know: is there anything forum regulars can do when we see spam posts beyond letting mods know about it? I was active in a now dead forum that would get hit once or twice a week. The mods would clean it up with in a half a day but until then those posts would just sit. Does the Google web crawler consider words in replies to the spam post? I used to reply occasionally with words like: scam, fraud, got ripped off. I have always wondered if I was wasting my time. A few times I checked the link to see if it went some where legitimate (Google the base url), if so, I then searched the site for an informational web form. If there was one, left a message that their SEO company was using sleazy methods with a link to the forum post. Google ought to make a code phrase that forum users can use to red flag spam posts. Though some of the posts were for Japanese and Chinese sites. The spam might not have been meant for Google but other search engines...

rurounijones 14y ago

Would love to hear feedback from the white-hat to this article. Maybe get a decent discussion going from both sides. It does seem that the white^hat's remarks are very harsh, I wonder if he has honestly never thought about it from the other side's point of view.

nkuttler 14y ago

So this "SEO" who helping his clients to "clean up their profiles" doesn't even know how to disavow links [1]. Well, it's not surprising that somebody who hired spammers once would hire another idiot later. [1] http://www.youtube.com/watch?v=393nmCYFRtA http://www.youtube.com/watch?v=393nmCYFRtA and https://www.google.com/webmasters/tools/disavow-links-main https://www.google.com/webmasters/tools/disavow-links-main (login required)

JimWestergren 14y ago
Well Google requires you to contact the website owners and do everything you can to get the links removed BEFORE you submit a list of disavow links. Google are supposed to check that you really did remove a large part of the links before disavowing.

bambax 14y ago

There must be a programmatic solution to this problem. Do some outbound links have value on this forum? If not, then you could remove all links, or remove the "link" part of the link (change @href to text). If some outbound links have value you need to identify those, and it's more complex, but a Bayesian analysis of posts should be able to score posts on their "spaminess" and remove the links on only the most spam-like comments. There may be some false-positive doing this, but since no information is actually removed (only the links, not the content) it should be quite ok.

robryan 14y ago
Random links on forums already have pretty low value. I would assume that forum spam has come down now that this is the case. As the article states, websites are looking the clean up the spammy inbound links as google has threatened to deindex sites that don't make an effort to do this.

will_critchlow 14y ago

I can't put words in the mouth of the SEO quoted, but (unless you know he was one of the people making removal requests against your site) I suspect he is not talking about you or sites like yours. I should add that I don't agree with the rhetoric btw, but I think he is targeting a different kind of webmaster. I think he is referring to webmasters who sold links (knowingly outside the guidelines) for years. They would previously have instantly removed the links if someone stopped paying. As soon as Google stepped up their game and removing those links was important, those same webmasters started charging to take down the same links. I would personally point to the irony of this (google creating a market that enriches people who have been abusing their system for years) rather than calling it immoral. Ymmv. Hope that helps clarify some things and I hope I'm not distorting the guy's real meaning. (written on my phone. Please excuse typos).

dhimes 14y ago
I'm wondering about the graph on Gary's site: what type of business' fortune is linked this tightly to Google's rankings? Is it a particular category, or are there several categories of businesses that suffer from (or leverage) this? That's a pretty frightening situation to be in, IMO.
- will_critchlow 14y ago
  Oh, there are many businesses at least that exposed. I personally wouldn't particularly relish running one, but that's another story...

lutusp 14y ago

The solution is obvious -- write a routine that automatically goes through the entire forum database and disables all the links -- leave the names, but rewrite the links so they're just text, not hyperlinks. Sort of like: s!http://!!g http://!!g The above deletes the "http:// http:// prefix, but leaves the original destination name, in case anyone wants to object that their post has been edited after the fact. So technically, it's no longer a link back to the originating site, but it's otherwise unchanged. No human intervention required. Problem solved.

ajenner 14y ago

Wow, so Google penalizes a site if a link to that site shows up in spammy pages? That seems like a new business model for black-hat SEOers: "hey, nice site you have there - it'd be a shame if links to it started appearing all over my spammy network - $$$ will make sure that doesn't happen." Really search engines should just give zero weight rather than negative weight to links from spammy sites.

Isofarro 14y ago
"hey, nice site you have there - it'd be a shame if links to it started appearing all over my spammy network" Good way to clue Google in to this network, and then drop it from the index. "Really search engines should just give zero weight rather than negative weight to links from spammy sites" That's what was happening before Penguin/Panda - and it wasn't working. There's no risk to link spamming. Now there is.

iuguy 14y ago

Does anyone else think that the author should set up something like mailchimp with an autoresponder to deal with the spam? Something that explains his policy, why he doesn't respond to emails, why he won't remove the link without charging them? Many of the people who will have these spammy links on his forum will have paid one of the more scummy SEOs out there to raise their SERPs. The people emailing may not have originally been aware that the links have been put up. By having an autoresponder address for people to mail to this should alleviate the spam for all but the dumbest of people (that can't read what to sign up to to get a response, or those that fail to read a clearly defined policy via email).

rizz0 14y ago

There's a difference between buying links, which Google actively started prohibiting at a certain point, and the automated forum spam you are mentioning. The latter is done using software like XRumer, and has always been penalized by Google. The former wasn't prohibited in the beginning, when link deals were often mutual agreements, most of the time involving a traffic component as well. The snake oil salesmen that are mentioned, are the ones who actively participated in the scheme by selling links and making a buck, and are trying to make another quick buck now that the rules have changed. Moreover, some links aren't even paid at all, but just look manipulative. For example, if you developed a wordpress theme, and your link is in the footer of tons of blogs, you might get penalized for manipulating the anchor text of your links in a non-natural way. In those legitimate situations, webmasters do have a moral obligation to cooperate. I don't think anyone would think that of an honest entrepreneur being spammed to death by link spamming software. On a side note, there are plenty of forums on the web that have survived the spam wave, if it were core to your business, you could have protected yourself.

Tichy 14y ago

Maybe it could be scripted? Site owners would have to place some proof of ownership on their site (to be generated on the site of the forum owner - see Google web admins site verification), then a script could detect the faulty links on the forum and remove them automatically.

JimWestergren 14y ago

Place a meta noindex, nofollow on the whole forum or at least the infected areas such as memberlist and user profiles. That could be done within 15 minutes and then you can answer with a standard reply that you did that to all emails you receive about removing links.

mangostache2 14y ago

Why are people getting bent out of shape over some bad links? Can't you just disavow them in Google Webmaster Tools?

Aardwolf 14y ago

There exist very popular forums that receive no such spam. Hacker News comments also seem free from it. Any idea why it is that some forums get targeted with 10000's of spam accounts, while others don't? What is the trick to protecting if even capchas don't work?

mesozoic 14y ago
They mainly use a program called xrumer to spam and it mostly works with your typically install of vbulletin or other popular forum software so custom software won't get hit. Kinda security by obscurity.

edwinyzh 14y ago

Maybe we should not blame search engines, but SEO really has bad effects on the Internet... PS, due to spams I've closed this forum (http://writingoutliner.com/forum http://writingoutliner.com/forum), a online forum for my 'Addon documents organizer for MS Word'. And I regret I use BBPress as the forum software, because new 2.x versions are not compatible with the old 1.x versions...

mesozoic 14y ago

Oh man that is some sweet irony. Just wait till the next waive of spam from SEOs trying to get competitors penalized then the competitors firing back and contacting you. Google stirred up a hornets nest because they couldn't figure out how to effectively devalue these links completely.