3 ms·
In theory you are right, but in reality 99% of the rogue bots are actually just some scraper tools were ignorant users changed sane defaults to "make it go fast
by jlg23 10y ago
In theory you are right, but in reality 99% of the rogue bots are actually just some scraper tools were ignorant users changed sane defaults to "make it go faster".
They usually don't have enough knowledge to even understand that they are routed into a black hole - not to speak of being able to do something about it.
Disclaimer: Getting rid of those idiots^wmisguides poor souls is part of my job description.
- throwasehasdwi 10y agoThen why not block based on bots that ignore robots text or make requests too quickly like I suggested? Looking for sub-strings in the UA header is a hack at best.
- jlg23 10y agoThey all require one to keep state information on the server side and with overall little benefit. Yes, I do what you proposed, but in hindsight it was a waste of time to set that up: 99% simply don't know what they are doing and therefore easy to catch. I don't care about this one person who knows to change the user agent sent, those are the ones you can usually talk to and they'll happily throttle their crawlers. The other 99 are the problem - and they are a problem that can be solved pretty well by simple string matching.