4 ms·

> The Go team holds that this service is not a crawler, and thus they do not obey robots.txt This seems wrong. I guess I always assumed that robots.txt applied

by dmm 4y ago

> The Go team holds that this service is not a crawler, and thus they do not obey robots.txt This seems wrong. I guess I always assumed that robots.txt applied to non-humans.

jaywalk 4y ago
I'm sure the reasoning would be something along the lines of these requests being made in response to user activity of some sort.
kevincox 4y ago
I think they are right not to obey robots.txt in this case. If I tell Go to download a module it shouldn't follow robots.txt because I am a human and I requested it. This is similar to if you had private iCal URLs on your server the right think would be to deny them in robots.txt (in case a crawler found a leaked link) but a service that is monitoring a specific iCal calendar should still fetch it. Basically robots.txt is more about what should be crawled than how. In this problem it appears that the traffic is desired in general, but it is being done far too often. robots.txt does have primative rate limiting configs but that seems to be a minor part of the file. Of course like anything there is nuance and there is definitely some middle ground between crawlers and humans.
- forty 4y ago
  Precisely, here the service is fetching stuff regularly even if no one is asking anything, if I understand correctly.
  - willcipriano 4y ago
    It apparently pulled a repo only used by the author 500 times in a day.
AshamedCaptain 4y ago
Even the main Google search crawler does not respect Crawl-Delay from robots.txt (as of today), so it's a moot point anyway. I suspect it's just a cultural thing inside Google.
- kevincox 4y ago
  I don't think so. When I worked at Google most things that talked to the outside world were carefully rate limited with quite generous backoffs if there was signs of the receiving side being harmed (errors or slowdowns). Especially crawling and mail sending. But Google is a big company and it is very likely that the Go team didn't think of these things that are probably top-of-mind for someone on the crawling, ads or mail teams.

by dmm 4y ago

> The Go team holds that this service is not a crawler, and thus they do not obey robots.txt This seems wrong. I guess I always assumed that robots.txt applied to non-humans.

jaywalk 4y ago

I'm sure the reasoning would be something along the lines of these requests being made in response to user activity of some sort.

kevincox 4y ago

I think they are right not to obey robots.txt in this case. If I tell Go to download a module it shouldn't follow robots.txt because I am a human and I requested it. This is similar to if you had private iCal URLs on your server the right think would be to deny them in robots.txt (in case a crawler found a leaked link) but a service that is monitoring a specific iCal calendar should still fetch it. Basically robots.txt is more about what should be crawled than how. In this problem it appears that the traffic is desired in general, but it is being done far too often. robots.txt does have primative rate limiting configs but that seems to be a minor part of the file. Of course like anything there is nuance and there is definitely some middle ground between crawlers and humans.

forty 4y ago
Precisely, here the service is fetching stuff regularly even if no one is asking anything, if I understand correctly.
- willcipriano 4y ago
  It apparently pulled a repo only used by the author 500 times in a day.

AshamedCaptain 4y ago

Even the main Google search crawler does not respect Crawl-Delay from robots.txt (as of today), so it's a moot point anyway. I suspect it's just a cultural thing inside Google.

kevincox 4y ago
I don't think so. When I worked at Google most things that talked to the outside world were carefully rate limited with quite generous backoffs if there was signs of the receiving side being harmed (errors or slowdowns). Especially crawling and mail sending. But Google is a big company and it is very likely that the Go team didn't think of these things that are probably top-of-mind for someone on the crawling, ads or mail teams.