2 ms·
Their web crawler / scraper "ClaudeBot" is the worst semi-legitimate crawler I've ever encountered. Extremely aggressive scraping from a ton of IP addresses wit
by 0x0 2y ago
Their web crawler / scraper "ClaudeBot" is the worst semi-legitimate crawler I've ever encountered. Extremely aggressive scraping from a ton of IP addresses with a ton of concurrent accesses, often getting lost in an infinite maze of dynamic url path combinations. Do better, please.