5 ms·
Show HN: New BucketRateLimiter Python package to rate limit requests to APIs
- kozirev8 4y agoI developed a new Python package BucketRateLimiter which allows to rate limit requests to external APIs. Some APIs allow only limited number of requests per second and the package was created to address the issue. The Package is based on Bucket conception and support use in asyncio and multithreading Python apps.
- nousermane 4y agoFor posts promoting own work, please consider adding "Show HN:" in front of submission title.
- kozirev8 4y agothanks!
- 35mm 4y agoDoes this work well with requests_cache?
- kozirev8 4y agoMy package does not rely on any http client libraries, you can rate limit any functions you implement yourself. I provided example-scripts in examples folder, the scripts are ready to use. The scripts illustrates what you can do with the package.
- zanecodes 4y agoHow common is it for HTTP client libraries to respect HTTP 429 Too Many Requests and the Retry-After header? And how many APIs actually implement it?
- AlotOfReading 4y agoIt's uncommon for user facing stuff, but only a little uncommon on the API side for abuse rate limiting. I know curl supports it. This kind of code is useful anywhere you have message buses with finite resources, which is a lot broader than just HTTP. It lets you separate the how of data processing and transformation from the when. I keep a token bucket header in my personal toolbox for embedded work because it's just such a common and simple building block.
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- ydlr 4y agoIn my experience, servers often don't send 429 responses. Just last week I encountered an API that sent a 200 response with a body that read "Rate limit exceeded."
- NovemberWhiskey 4y agoIt's always struck me as pretty much useless; unless the service is uniquely identifying each downstream and somehow backing them off individually (which is not usually the case, you're usually backing off a consuming identity like an account or an address) then Retry-After is more or less meaning-free for an individual client. In a practical sense, a 429 is just a 503. Back off and try again if you have time, otherwise fail. It's the passive-aggressive "I'm not providing service to your well-formed request, but it's you and not me" response code.
- js2 4y agoFor whatever reason, this seems like a popular thing to implement. There are several on pypi already so it would useful to know why you wrote a new one. Something that seems missing is a pluggable backend for when you need to coordinate rate-limiting across machines. This library allows for a variety of backends: https://pyrate-limiter.readthedocs.io/en/latest/ https://pyrate-limiter.readthedocs.io/en/latest/
- kozirev8 4y agoI found that some people struggle to work with existent solution and decided to implement another one from scratch. The question in SO gave me the idea to write the package. https://stackoverflow.com/questions/73336932/making-rate-limit-requests-to-cursor-paginated-api-using-asyncio/ https://stackoverflow.com/questions/73336932/making-rate-lim...
- cntlzw 4y agoTricky problem to implement. I tried to compute requests per seconds from multiple threads. Problem gets much harder if you reach a certain threshold. I think it was round 100 requests per second in my case. After this memory contention becomes a problem. Then I learned about LongAdder in Java. It was a really interesting topic.