3 ms·
Why should a git client respect an http standard such as robots.txt?
by jslql 4y ago
Why should a git client respect an http standard such as robots.txt?
- cmatthias 4y agoRead the OP; it's obvious based on the references to robots.txt, the User-Agent header, returning a 429 response, etc, that most (all?) of Google's requests are doing git clones over http(s).
- yamtaddle 4y agoGoogle began pushing for it to become an Internet standard—explicitly to be applicable to any URI-driven Internet system, not just the Web—in 2019, and it was adopted as an Internet standard in 2022. https://developers.google.com/search/blog/2019/07/rep-id https://developers.google.com/search/blog/2019/07/rep-id
- cmatthias 4y agoThis is true but irrelevant to the parent's question -- in the article, it's made clear that Google's requests are happening over HTTP, which is the most obvious reason why robots.txt should be respected.
- yamtaddle 4y agoIt's relevant because it attacks the premise of their objection.
- trulyrandom 4y agoBecause it uses HTTP.