3 ms·

I've ran a large scale web crawler for a decade: http://www.datastreamer.io/ http://www.datastreamer.io/ The main issues and complaints we run into are mostly

I've ran a large scale web crawler for a decade: http://www.datastreamer.io/ http://www.datastreamer.io/ The main issues and complaints we run into are mostly unusual requests for indexing content. People see the total number of requests and think it's a significant burden and just have an irrational need to protect their content. It's almost a form of paranoia at some point. I mean we have NO problem removing someone that doesn't want to be indexed and 99/100 we do so without a problem. The only times we push back are when government organizations contact us as we feel we have a legal right to index the content. We haven't had to really fight over this because they usually agree and that's the end of the discussion.

zzzcpan 8y ago
I see hundreds of thousands of requests from search engines daily without any burden, but I do block all other bots. It has nothing to do with paranoia, a lot of bots are just evil making it impossible to act towards bots in good faith. For example there are forensic bots, copyright protection bots, brand protections bots and similar bots with the whole purpose of using crawled data to send you threats and "abuse" reports or drag you to court. There are bots that scan for vulnerabilities to exploit and bots that scan pages to find something bad to blacklist somewhere. There are bots that generate spammy links or even grab content for black hat seo, something search engines punish you for. There are bots that belong to companies that sell your data or try to gain competitive advantage through this without giving you anything in return, something you may or may not be comfortable with. There are even bots that gather data for political and intelligence purposes, literally causing death and destruction.
- brennebeck 8y ago
  Given the wide range of bots you mentioned, I’m very curious how you’re just blanket blocking all other bots? Could you elaborate a bit? Edit: add other; ambiguous.

zzzcpan 8y ago

I see hundreds of thousands of requests from search engines daily without any burden, but I do block all other bots. It has nothing to do with paranoia, a lot of bots are just evil making it impossible to act towards bots in good faith. For example there are forensic bots, copyright protection bots, brand protections bots and similar bots with the whole purpose of using crawled data to send you threats and "abuse" reports or drag you to court. There are bots that scan for vulnerabilities to exploit and bots that scan pages to find something bad to blacklist somewhere. There are bots that generate spammy links or even grab content for black hat seo, something search engines punish you for. There are bots that belong to companies that sell your data or try to gain competitive advantage through this without giving you anything in return, something you may or may not be comfortable with. There are even bots that gather data for political and intelligence purposes, literally causing death and destruction.

brennebeck 8y ago
Given the wide range of bots you mentioned, I’m very curious how you’re just blanket blocking all other bots? Could you elaborate a bit? Edit: add other; ambiguous.