4 ms·

I wonder if we're doing the wrong thing blocking them with invasive tools like cloudflare? If all you're concerned about is server load, wouldn't it be better

I wonder if we're doing the wrong thing blocking them with invasive tools like cloudflare? If all you're concerned about is server load, wouldn't it be better to just offer a tar file containing all of your pages they can download instead? The models are months out of date, so a monthly dumb would surely satisfy them. There could even be some coordination for this. They're going to crawl anyway. We can either cooperate or turn it into some weird dark market with bad externalities like drugs.

recallingmemory 1y ago
Yeah, I am in the opposing camp too - I don't use Cloudflare's bot fight tooling on any of our high traffic websites. I'm not seeing the issue with allowing bots to crawl our websites other than some additional spend for bandwidth. Agent mode is pretty powerful when paired with a website that cooperates, and if people want to use AI to interact with our data then what's wrong with that?
masfuerte 1y ago
A tar file would be better if the crawlers would use it, but even sites with well-publicised options for bulk downloads (like wikipedia) are getting hammered by the bots. The bot operators DNGAF.
- HankStallone 1y ago
  Right. I don't care if AI (or anything else) indexes or learns from my sites. That's what they're there for. But yesterday I blocked an IP that hit one of my sites 82000 times in an hour, or 22/second. And apparently it's a very stupid bot, because it kept redownloading CSS and other asset files every time it saw a link to them. There's no way the people behind that bot are going to follow any suggestions to make it behave better. After all, adding things like caching and rate-limiting to your web crawler might take a few hours, and who's got time for that.
zzo38computer 1y ago
I also do not like Cloudflare. If the crawlers were aware of these archive files, and would be willing to use it, then that would help, but it isn't. (It would also help to know which dynamic files are worthless for archiving and mirroring, but they will often ignore that.)

recallingmemory 1y ago

Yeah, I am in the opposing camp too - I don't use Cloudflare's bot fight tooling on any of our high traffic websites. I'm not seeing the issue with allowing bots to crawl our websites other than some additional spend for bandwidth. Agent mode is pretty powerful when paired with a website that cooperates, and if people want to use AI to interact with our data then what's wrong with that?

masfuerte 1y ago

A tar file would be better if the crawlers would use it, but even sites with well-publicised options for bulk downloads (like wikipedia) are getting hammered by the bots. The bot operators DNGAF.

HankStallone 1y ago
Right. I don't care if AI (or anything else) indexes or learns from my sites. That's what they're there for. But yesterday I blocked an IP that hit one of my sites 82000 times in an hour, or 22/second. And apparently it's a very stupid bot, because it kept redownloading CSS and other asset files every time it saw a link to them. There's no way the people behind that bot are going to follow any suggestions to make it behave better. After all, adding things like caching and rate-limiting to your web crawler might take a few hours, and who's got time for that.

zzo38computer 1y ago

I also do not like Cloudflare. If the crawlers were aware of these archive files, and would be willing to use it, then that would help, but it isn't. (It would also help to know which dynamic files are worthless for archiving and mirroring, but they will often ignore that.)