5 ms·

I first learned about this after starting https://neocities.org https://neocities.org and seeing a bunch of really garbage pages that were full of random text t

I first learned about this after starting https://neocities.org https://neocities.org and seeing a bunch of really garbage pages that were full of random text that linked to a derpie site somewhere. We get pagerank SEO spam from time to time, and it's pretty annoying. I have the tools to take care of it within 5 minutes every day, but I do worry that if we grow to a certain point it may no longer be possible for me to handle the problem alone. I'm sure many other sites have similar problems with comment spam, and I'd love to hear some advice on how to deal with this from sites that have the same problem. Right now our main lines of defense are a recaptcha (our last remaining third party embed, ironically sending user data to Google I'd rather not send to deal with a problem Google largely created), and a daily update of an IP blacklist we get from Stop Forum Spam. I tried to do some Bayesian classification, but didn't make much progress unfortunately. And nofollow really isn't an option for me, as it would involve me manipulating other people's web sites and I don't want to do that.

mmanfrin 11y ago
This might be very heavy handed, but could you build something that added rel="nofollow" to all links on user pages until trust is verified?
- softawre 11y ago
  > I tried to do some Bayesian classification, but didn't make much progress unfortunately. And nofollow really isn't an option for me, as it would involve me manipulating other people's web sites and I don't want to do that. Did you read the comment?
- kyledrake 11y ago
  I've pondered doing something like this, but it kindof crosses a line I'm not yet comfortable crossing (changing users' HTML without them noticing), and would require a way for me to convert from/back when the user edits their site. Technically it's pretty difficult: HTML/XML parsers like Nokogiri like to "fix" html when they parse it, so they can do a lot of changes to your document you didn't want to make. I've said I would decide to cross it if I ever ran into a major security issue, but so far that hasn't been the case. I've even decided against web page auto optimization ala things like ngx_pagespeed for this reason. One place I do cross this line is for our https://neocities.org/browse https://neocities.org/browse gallery in order to deal with iframe issues (I make the A links on a site open a new tab with a javascript inject), but that passive model won't work for pagerank issues because it uses the browser to make the changes. And honestly, let's be real, even if I put on nofollow and Google stops using pagerank, they're probably still going to do it because they're basically shady pagerank scam artists in the end, fooling gullible people into sending them money. They don't understand they're feeding into a botnet meets army of underpaid pagerank spammers, and it's basically impossible to fix this with education. It would be excellent if Google gave me an API to report pagerank spam. For all the money they've made on pagerank, it would be nice if they could defer some of that money into helping us deal with this, and would definitely help to improve their search results.

mmanfrin 11y ago

This might be very heavy handed, but could you build something that added rel="nofollow" to all links on user pages until trust is verified?

softawre 11y ago
> I tried to do some Bayesian classification, but didn't make much progress unfortunately. And nofollow really isn't an option for me, as it would involve me manipulating other people's web sites and I don't want to do that. Did you read the comment?
kyledrake 11y ago
I've pondered doing something like this, but it kindof crosses a line I'm not yet comfortable crossing (changing users' HTML without them noticing), and would require a way for me to convert from/back when the user edits their site. Technically it's pretty difficult: HTML/XML parsers like Nokogiri like to "fix" html when they parse it, so they can do a lot of changes to your document you didn't want to make. I've said I would decide to cross it if I ever ran into a major security issue, but so far that hasn't been the case. I've even decided against web page auto optimization ala things like ngx_pagespeed for this reason. One place I do cross this line is for our https://neocities.org/browse https://neocities.org/browse gallery in order to deal with iframe issues (I make the A links on a site open a new tab with a javascript inject), but that passive model won't work for pagerank issues because it uses the browser to make the changes. And honestly, let's be real, even if I put on nofollow and Google stops using pagerank, they're probably still going to do it because they're basically shady pagerank scam artists in the end, fooling gullible people into sending them money. They don't understand they're feeding into a botnet meets army of underpaid pagerank spammers, and it's basically impossible to fix this with education. It would be excellent if Google gave me an API to report pagerank spam. For all the money they've made on pagerank, it would be nice if they could defer some of that money into helping us deal with this, and would definitely help to improve their search results.