3 ms·
It's probably the reverse - the data can probably fit in a few commercially available hard drives, the API calls to discover and download all repository without
by Aperocky 4y ago
It's probably the reverse - the data can probably fit in a few commercially available hard drives, the API calls to discover and download all repository without running afoul of the rate limiters and whatever other anti-crawling strategies can take years if you're running it single threaded (a single IP can effectively be considered single threaded).
- XorNot 4y agoThere are publicly published datasets, updated regularly, and you can stream all the events happening on Github (which is how secret leaks happen) so you can stay abreast of new repositories. https://www.gharchive.org/ https://www.gharchive.org/ http://ghtorrent.org/ http://ghtorrent.org/ also available on GCP as a dataset, provided by Github itself: https://console.cloud.google.com/marketplace/details/github/github-repos?filter=solution-type:dataset&id=46ee22ab-2ca4-4750-81a7-3ee0f0150dcb https://console.cloud.google.com/marketplace/details/github/...