3 ms·

I build a system that scraped GitHub. Even though GitHub can clearly handle lots of traffic I still rate limited the hell out of it. The only time I've ever see

• mdaniel 3y ago
  > I still rate limited the hell out of it Merely for your consideration, they actually do a great job of indicating in the response how many more requests per "window" the current authentication is allowed, and a header containing the epoch at which time the window will reset: https://docs.github.com/en/rest/overview/resources-in-the-rest-api?apiVersion=2022-11-28#rate-limit-headers https://docs.github.com/en/rest/overview/resources-in-the-re... I would suspect, all things being equal, that politely spaced requests are better than "as fast as computers can go" but I was just trying to point out that one need not bend oppressively over the other direction when the site is kind enough to tell the caller about the boundaries in a machine-readable format
  • that_guy_iain 3y ago
    That's on the API. For what I wanted it was do various HTTP requests to the website and do git clones.