20 ms·
On one of my sites I received a million requests from Googlebot before lunchtime today. Those requests were spread over hundreds of IPs. I get similar traffic f
by aorth 2y ago
On one of my sites I received a million requests from Googlebot before lunchtime today. Those requests were spread over hundreds of IPs. I get similar traffic from them and other hyper scalers daily. I'm just one guy. I don't have load balancers and endless engineering budget. I'm sick of this.
- phreeza 2y agoDo they not respect the crawl-delay directive in robots.txt?