> It’s a little surprising that Google doesn’t just index everything public in Github, gitlab, etc. GitHub excludes a few patterns in robots.txt that means a l

> It’s a little surprising that Google doesn’t just index everything public in Github, gitlab, etc. GitHub excludes a few patterns in robots.txt that means a lot of the public data is never seen by crawlers that respect robots.txt, for example: Disallow: /*/*/tree/* Disallow: /*/*/blob/* and Disallow: /*/*/commits/*/* Disallow: /*/*/commits/*?author Disallow: /*/*/commits/*?path Disallow: /*/*/branches Disallow: /*/*/tags https://github.com/robots.txt https://github.com/robots.txt They probably want people landing mainly on - user profiles - repo roots - individual issues - individual pull requests And I am glad they do, because when I search on Google, the above mentioned kinds of results are exactly what I am looking for, not individual source files etc.