3 ms·
They kill your bandwidth. For a client's catalog site we discovered crawlers were more than half of the used bandwidth costs.
by alecco 9y ago
They kill your bandwidth. For a client's catalog site we discovered crawlers were more than half of the used bandwidth costs.
- Rjevski 9y agoDon't humans kill your bandwidth too? In fact a properly written bot would use less bandwidth as it doesn't care about CSS or images. My solution would solve the issue in a fair way by having a "reasonable usage" limit applied to everyone, bot or not. This also means it can't be defeated by someone paying humans to do the dirty work to bypass bot restrictions.