4 ms·
Advanced usage of Python requests: timeouts, retries, hooks
- hmhrex 7y agoThank you for this brief reference. I'm always going back to my old projects to remind myself how I did things with requests. I didn't realize some of things existed.
- dhxt 7y agoYou're welcome. I thought I'd aggregate all the useful tricks I'd found while working on my last project. It's a collection of snippets I had found in the docs and in Github issues. The post contains the most important snippets, but you might find the advanced section of the requests docs [1] useful and the requests_toolbelt [2]. [1] https://requests.readthedocs.io/en/master/user/advanced/ https://requests.readthedocs.io/en/master/user/advanced/ [2] https://toolbelt.readthedocs.io/en/latest/ https://toolbelt.readthedocs.io/en/latest/
- hmhrex 7y agoI will definitely be diving into the advanced section in the Requests docs. Thanks!
- sciurus 7y agoI wish that requests had a default timeout instead of defaulting to waiting forever. It's very easy for devs to forget to think about timeouts, but inevitably that ends in a cascading production outage.
- dhxt 7y agoI'm not sure if you noticed, but the post contains a section on how to set default timeouts. https://findwork.dev/blog/advanced-usage-python-requests-timeouts-retries-hooks/#setting-default-timeouts https://findwork.dev/blog/advanced-usage-python-requests-tim...
- sciurus 7y agoRight. I'm saying the library should set a default timeout by default, which you could then change if needed. Of course, whatever it chose would inevitably be wrong for many scenarios, it would be less wrong than defaulting to waiting forever.
- dhxt 7y agoI absolutely agree. According to this comment [1] they're planning to introduce it in v3. [1] https://github.com/psf/requests/issues/3070#issuecomment-328083039 https://github.com/psf/requests/issues/3070#issuecomment-328...
- inetknght 7y agoI just recently dealt with a timeout issue in requests: I was setting the timeout to 1 but it took 10 seconds for the request to fail. It turns out that the timeout does not apply to the name resolver and the name lookup was timing out. So the default system-level timeout was 10 seconds.
- ddeokbokki 7y agoThat sentry responses decorator is absolutely beautiful
- jakeogh 7y agoIf I enable logging of retries: logging.basicConfig(level=logging.DEBUG) I dont see any retries on response 200: ConnectionError(ReadTimeoutError("HTTPSConnectionPool(host='domain.com', port=443): Read timed out.")) Is there a way to also retry on ChunkedEncodingError?
- da12 7y agoFantastic reference. Thanks for sharing!