4 ms·
Even the main Google search crawler does not respect Crawl-Delay from robots.txt (as of today), so it's a moot point anyway. I suspect it's just a cultural thi
by AshamedCaptain 4y ago
Even the main Google search crawler does not respect Crawl-Delay from robots.txt (as of today), so it's a moot point anyway.
I suspect it's just a cultural thing inside Google.
- kevincox 4y agoI don't think so. When I worked at Google most things that talked to the outside world were carefully rate limited with quite generous backoffs if there was signs of the receiving side being harmed (errors or slowdowns). Especially crawling and mail sending. But Google is a big company and it is very likely that the Go team didn't think of these things that are probably top-of-mind for someone on the crawling, ads or mail teams.