3 ms·
There's an asterisk to that: they serve the underlying content through two different APIs so one can side-step the HTML wrapper around the bytes: the discovery
by mdaniel 4y ago
There's an asterisk to that: they serve the underlying content through two different APIs so one can side-step the HTML wrapper around the bytes: the discovery phase has a formal API (both REST and GraphQL) for finding repos, and then the in-repo content can be git-cloned and one can locally index every branch, commit, and blob, without issuing hundreds of thousands of http requests to GH. One would still need to hit GH for the issues, if that's in scope, but it'd be way less http requests unless your repo is named kubernetes or terraform.
- marginalia_nu 4y agoWe're still talking about git clone:ing a hundred thousand github repos. Git Repos get big very fast. That's a lot of data when you're realistically only interested in is a few markdown files per repo.