4 ms·
> If only we had some way to teach computers that when you smash a site with so much traffic that it becomes unresponsive that it's bad for that site Yes, that
by Kuinox 2y ago
> If only we had some way to teach computers that when you smash a site with so much traffic that it becomes unresponsive that it's bad for that site
Yes, that's called following the http specs and return http 429, too many requests, the crawler would probably throttle in this case.
- matt_heimer 2y agoI am aware and have a rate limit rule for the AI bot category configured in Cloudflare which defaults to returning a 429. But I was mostly being sarcastic. No other bots are this aggressive. I see tens of thousands of hits per day from other bots, Amazon, Apple, etc. OpenAI is the only one generating hundreds of thousands of hits per day to a small niche website. Rate limiting the AI bot category ends up impacting the better behaving bots also. For comparison it takes Google about 45 days to make the same number of requests that OpenAI makes per day.