3 ms·
A tar file would be better if the crawlers would use it, but even sites with well-publicised options for bulk downloads (like wikipedia) are getting hammered by
by masfuerte 1y ago
A tar file would be better if the crawlers would use it, but even sites with well-publicised options for bulk downloads (like wikipedia) are getting hammered by the bots.
The bot operators DNGAF.
- HankStallone 1y agoRight. I don't care if AI (or anything else) indexes or learns from my sites. That's what they're there for. But yesterday I blocked an IP that hit one of my sites 82000 times in an hour, or 22/second. And apparently it's a very stupid bot, because it kept redownloading CSS and other asset files every time it saw a link to them. There's no way the people behind that bot are going to follow any suggestions to make it behave better. After all, adding things like caching and rate-limiting to your web crawler might take a few hours, and who's got time for that.