3 ms·
Yeah, the GitHub robots.txt is surprisingly restrictive: https://github.com/robots.txt https://github.com/robots.txt User-agent: * Disallow: /*/pulse
by simonw 4y ago
Yeah, the GitHub robots.txt is surprisingly restrictive:
https://github.com/robots.txt https://github.com/robots.txt
User-agent: *
Disallow: /*/pulse
Disallow: /*/tree/
That "/*/tree" rule means that search engine crawlers are allowed to hit the README file of a repo but effectively NONE of the other files in it.
Which means that if you keep your project documentation on GitHub in a docs/ folder it won't be indexed!
You need to publish it to a separate site via GitHub Pages, or use https://readthedocs.org/ https://readthedocs.org/
(Side note: I just noticed https://github.com/ekansa/Open-Context-Data https://github.com/ekansa/Open-Context-Data is explicitly listed in the robots.txt for GitHub - the only repo that gets a mention like that. I'd love to know the story behind that!)
- burkaman 4y agoThat repo apparently used to be the largest on GitHub: https://news.ycombinator.com/item?id=5912922 https://news.ycombinator.com/item?id=5912922. I bet Google was repeatedly scraping the entire thing and putting too much strain on their servers at the time it was added. It's been 10 years, what are the odds nobody at GitHub today remembers why it was added? Also, very relatable to see a decade old "I'll update this shortly" comment that was never updated. We all have a few of those.
- snowycat 4y agoIt appears that the creator of the repo actually confirmed this: https://twitter.com/ekansa/status/1137052076062650368 https://twitter.com/ekansa/status/1137052076062650368
- knute 4y ago/*/tree is only for directory listings. File contents will be under a /blob/ path, e.g. https://github.com/facebook/react/blob/main/AUTHORS https://github.com/facebook/react/blob/main/AUTHORS, and should be, AFAIK, indexable. (mandatory disclaimer: I'm a GitHub employee, not speaking on behalf of the company)
- staplung 4y agoIf they can't hit `/*/tree` is there a way to know the URLs of the files?
- simonw 4y agoI asked about this on the support forum a while ago and never got a satisfactory response: https://github.com/community/community/discussions/20958 https://github.com/community/community/discussions/20958
- blep_ 4y agoThere's also two users: Disallow: /account-login Disallow: /Explodingstuff/ The first for obvious reasons, the second probably because they've uploaded nothing of substance besides a copy of WannaCry.