3 ms·
You can't, unless your question is "how do I mirror a github repo?" I'm currently indexing thousands of GitHub repos for analytics reasons, and if you don't pl
by sdesol 6y ago
You can't, unless your question is "how do I mirror a github repo?"
I'm currently indexing thousands of GitHub repos for analytics reasons, and if you don't play nice (fetch infrequently), GitHub will throttle/terminate your connection.
- threeseed 6y agoGitHub isn't the first company to do this. You just need to use one of the many scraping proxy services which will give you a HTTP proxy that cycles through millions of IP addresses. It's impossible to block since they are largely residential IPs controlled by malware. It definitely works but you are encouraging them which isn't a nice feeling.