10 ms·
Crawlers and spambots are the scourge of medium to small websites. I run a small wiki that gets just a few thousand human hits a day. But according to the ser
by nostromo 13y ago
Crawlers and spambots are the scourge of medium to small websites.
I run a small wiki that gets just a few thousand human hits a day. But according to the server logs 90% of server hits are crawlers and spambots, so I'm using 10 times the resources I really need to serve customers.
I finally resorted to blocking entire data centers and companies that crawl constantly but send no traffic.
I feel like search engines should crawl websites in proportion to the traffic they send. For example, Yandex, Baidu, and Bing were all crawling my website hundreds or thousands of times a day, but never sending a single visitor (or in the case of Bing sending single-digit visitors). It's an absurd waste of resources, so I blocked them completely.
- ryandrake 13y agoChicken and egg. Should they 1. crawl you first, or 2. send you traffic first? If 1, how long until they should either send traffic or stop, and if 2, how?
- toomuchtodo 13y agoCould they not run webhooks where my site can call to them when to do a full or partial crawl? PubSub for web crawling, if you will.
- ceejayoz 13y agoSitemaps (http://www.sitemaps.org/ http://www.sitemaps.org/) help a bit in that regard, as crawlers can check the sitemap and only crawl updated content.
- unhappyhippie 13y agoThanks.
- sergiosgc 13y agoPUsH: http://en.wikipedia.org/wiki/PubSubHubbub http://en.wikipedia.org/wiki/PubSubHubbub Google is the only crawler I see using the protocol on my site. It does make Google updates occur within minutes, so that alone is reason to implement push.
- leephillips 13y agoI thought this was only being used for RSS - interesting!
- pavel_lishin 13y agoWhy not adjust the crawl rate based on how often the site's content changes? If DuckDuckGoogle crawls me once an hour for a week, and notices that I only updated my content twice, why not scale the crawling back to a more reasonable rate, like once a day?
- lostlogin 13y agoThis seems logical, however what about the person who updates rarely (once or twice a month) but get a lot of notice when they do. Search engines wouldn't want to miss the opportunity to get that traffic. I wonder if there is some sort of reverse process you could opt into - ie never crawl me until I ask to be crawled. Disclaimer: I know little about this sort of thing.
- gizmo686 13y agoYou could probably do this by having a robots.txt file that blocks all crawling. When you want to be re-crawled, you can edit it to allow the relevant spiders to crawl you. I would imagine that spiders do not automatically re-check sites which disallow everything often, but Google (and probably others) let you manually submit URLs to be crawled.
- walshemj 13y agoBut a disallow all robots.txt will kill your sites traffic - its not going to work like you think it is.
- pavel_lishin 13y agoPresumably you could set a minimum interval - an hour, or a day. This would make sure that big updates are noticed, without putting undue stress on the server.
- deleted 13y ago[deleted]
- victorf 13y agoAdd a time limit to robots.txt and block anyone who violates it. Maybe also add a nonexistent honeypot (I disallow /wpadmin on my static website haha). I've been fail2banning a lot of shitbirds and my traffic logs before and after are quite different.
- guard-of-terra 13y agoDo you/did you receive traffic from DDG? Because DDG uses Yandex for [some] organic search results and if you block its spider you will also partially block DDG.
- nostromo 13y agoI did not know that! Thanks, I'll look.
- PaulHoule 13y agoIt's actually a lot worse if your site is large since crawlers will make even more requests crawling a large site. If I see that a crawler is sending me any traffic at all, I will accept that, but if the amount of traffic is zero, I put them in robots.txt and in the IP block list. Although I try hard to make sure my images are all clean creative common images, I block the robots that are there to find copyrighted images because these ones exist to give me nothing but trouble.
- leephillips 13y agoIs there a list of good IPs to block, similar to the hosts files that people pass around? EDIT: chrsstrm mentions something below
- gabemart 13y ago>I block the robots that are there to find copyrighted images because these ones exist to give me nothing but trouble. I've never heard of these bots before. Where did you hear about them, and how do you block them?
- greglindahl 13y agoFortunately for us new(er) search engines (I work at blekko), only a small fraction of websites actively block crawlers from new search engines. It's a chicken and egg problem: we're going to have a big index for a long time before we have big traffic. I also have a bunch of hobby websites, and I agree that bing crawls a lot more than Google and sends me a lot less traffic. And Yandex and Baidu have very little English-speaking usage, so they aren't sending me much traffic.
- sergiosgc 13y agoI don't block completely, but have for a long time configured servers to react like this: - If it's cached content, there's no limit - If it's non cached content and it is a bot, direct to the pool of servers assigned to uninteresting bots - If it is an interesting bot (Google, bing, a couple others), use instead a pool of servers for interesting bots Some bots are not only non-traffic generating, they are absurdly aggressive, consuming 99% of server resources for the hour or so needed to pull the content, regular site users be damned. A configuration like this is reasonably low maintenance while limiting the damage by bots.
- mrgreenfur 13y agoI'm late, but for this there are companies that sit in the middle and block crawlers, spambots and known-malicious addresses. Here's one I use: http://www.incapsula.com/ http://www.incapsula.com/ So far I've been happy with it!