2 ms·
> using GitHub APIs to download the data yourself isn't possible Is it? Data storage would be prohibitive, but I can see ways to download the entirety of Githu
by Aeolun 4y ago
> using GitHub APIs to download the data yourself isn't possible
Is it? Data storage would be prohibitive, but I can see ways to download the entirety of Github in a few weeks/months (assuming my size estimate is accurate).
- Aperocky 4y agoIt's probably the reverse - the data can probably fit in a few commercially available hard drives, the API calls to discover and download all repository without running afoul of the rate limiters and whatever other anti-crawling strategies can take years if you're running it single threaded (a single IP can effectively be considered single threaded).
- XorNot 4y agoThere are publicly published datasets, updated regularly, and you can stream all the events happening on Github (which is how secret leaks happen) so you can stay abreast of new repositories. https://www.gharchive.org/ https://www.gharchive.org/ http://ghtorrent.org/ http://ghtorrent.org/ also available on GCP as a dataset, provided by Github itself: https://console.cloud.google.com/marketplace/details/github/github-repos?filter=solution-type:dataset&id=46ee22ab-2ca4-4750-81a7-3ee0f0150dcb https://console.cloud.google.com/marketplace/details/github/...