3 ms·
A case of github not limiting a particular api endpoint? I don't see any real harm in this guy sorta running amok. It does make the feature of staring repos les
by stephenitis 11y ago
A case of github not limiting a particular api endpoint? I don't see any real harm in this guy sorta running amok. It does make the feature of staring repos less... something.
There was a good talk I would recommend watching by Jeremy Edberg (Architect from Netflix/Eeddit that said to put limits on everything)
http://www.heavybit.com/library/video/2013-08-13-jeremy-edberg http://www.heavybit.com/library/video/2013-08-13-jeremy-edbe...
"As I mentioned before, you need to watch your limits. You should also put limits on everything. Put a hard number of limit on number of API requests that could be made from any one IP. Make it a thousand a second, something absurdly high. Because you know what? One day, that's going to happen and you're going to be really glad that that limit was there because now you've blocked somebody from completely taking down your service, maybe because they just wrote a bad robot or something, you know? They were trying to scrape your site, they screwed up, they're hitting you 10,000 times a second. At least, it won't take you down. Or, maybe perhaps, you've been linked from the Yahoo front page, which is what took down Reddit the first time when somebody put a Reddit button on a page that was linked from the Yahoo! home page. We basically DDoS-ed ourselves with the Reddit buttons. We quickly figured out how to put that on a CDN, so we didn't have to deal with that problem anymore."
- minimaxir 11y agoThere is a 5,000/hour rate limit. (https://developer.github.com/v3/activity/starring/ https://developer.github.com/v3/activity/starring/) So 120,000 repo Stars a day at maximum utility. You can get a list of all GitHub repos and derive their IDs with some BigQuery shenanigans on the GitHub Archive, then just feed the IDs into a while-loop. Simple to implement, albeit not necessarily ethical. EDIT: 5,000 limit is not endpoint specific. But if you use it all for one endpoint...
- codinghorror 11y agoThat seems like a super high limit.
- minimaxir 11y agoIt's high because the 5,000 limit is globally shared for all non-search queries: https://developer.github.com/v3/rate_limit/ https://developer.github.com/v3/rate_limit/