3 ms·
GitHub is hella tricky to crawl too due to its sheer size and single entry point (meaning slow crawl speed). I've been looking at the problem as well, and so fa
by marginalia_nu 4y ago
GitHub is hella tricky to crawl too due to its sheer size and single entry point (meaning slow crawl speed). I've been looking at the problem as well, and so far just ignored it as un-crawlable, but I might do something like crawl only the about pages for repos that are linked to externally some time in the future.
- mdaniel 4y agoThere's an asterisk to that: they serve the underlying content through two different APIs so one can side-step the HTML wrapper around the bytes: the discovery phase has a formal API (both REST and GraphQL) for finding repos, and then the in-repo content can be git-cloned and one can locally index every branch, commit, and blob, without issuing hundreds of thousands of http requests to GH. One would still need to hit GH for the issues, if that's in scope, but it'd be way less http requests unless your repo is named kubernetes or terraform.
- marginalia_nu 4y agoWe're still talking about git clone:ing a hundred thousand github repos. Git Repos get big very fast. That's a lot of data when you're realistically only interested in is a few markdown files per repo.