2 ms·
Don't prevent them. The same data you let humans access for free should be accessible via bots. If you only want to give out a "reasonable" amount of data, that
by Rjevski 9y ago
Don't prevent them. The same data you let humans access for free should be accessible via bots. If you only want to give out a "reasonable" amount of data, that humans wouldn't usually exceed but bots would, then define a rate-limit that wouldn't inconvenience humans and then apply it for everyone - bot or not. That way you're discriminating based on the amount of data instead of whether it's a bot or not. It will thwart people simply paying humans to scrape the data (which would happen if you magically found a way to block bots) while not inconveniencing humans who use a bot to make their job easier while scraping a reasonable amount of data.
- alecco 9y agoThey kill your bandwidth. For a client's catalog site we discovered crawlers were more than half of the used bandwidth costs.
- Rjevski 9y agoDon't humans kill your bandwidth too? In fact a properly written bot would use less bandwidth as it doesn't care about CSS or images. My solution would solve the issue in a fair way by having a "reasonable usage" limit applied to everyone, bot or not. This also means it can't be defeated by someone paying humans to do the dirty work to bypass bot restrictions.