3 ms·
This is why web crawlers are built with upper boundaries on everything! Nobody malicious brings down crawlers. It's just unexpected things you find out on the
by matt_wulfeck 9y ago
This is why web crawlers are built with upper boundaries on everything!
Nobody malicious brings down crawlers. It's just unexpected things you find out on the internet.
- jacquesm 9y ago> Nobody malicious brings down crawlers. You're wrong about that. I've more than once brought down crawlers on purpose, especially the ones that didn't respect robots.txt.