4 ms·
I don't. Just 3 AI spiders put more load on our servers than all search engine spiders and all human traffic combined. Some numbers I have handy from before I
by LukeShu 2y ago
I don't.
Just 3 AI spiders put more load on our servers than all search engine spiders and all human traffic combined.
Some numbers I have handy from before I blocked the bots:
ClaudeBot drove more requests through our Redmine in a month than it saw in the combined 5 years prior to ClaudeBot.
Bytespider accounted for 59% of the total traffic to our Git server.
Amazonbot accounted for 21% of the total traffic to our Git server.
Google has never even been close to breaking out of the single-digit-percentages of any metric.
- DamonHD 2y agoGenerally Googlebot is well behaved and efficient these days, though I have discovered that it is currently horribly broken around 429 / 503 response codes... And pays no attention to Retry-After either... Also Google-Podcast which is meant to have been turned off!
- o11c 2y agoSomeone needs to start adding all these AI's homepages to the browser "malware" lists.