3 ms·

I wonder where the time is going. I suspect that something like atop would show that the CPU is idle for much of the time, which suggests that the problem is n

by anamax 5y ago

I wonder where the time is going. I suspect that something like atop would show that the CPU is idle for much of the time, which suggests that the problem is not Python or even the GIL. If that's true, there's something ineffective about the way that the requests are being made/handled. AFAIK, the author did not try "make a bunch of requests before waiting for answers". If many of the requests are going to the same server, blasting a bunch of requests down a single socket before waiting for a response should be faster, if only because there's only one ssl handshake, even if the remote server doesn't pipeline request processing. (I suspect that "too many threads" didn't work because that reduced connection reuse.) There's a fairly simple monkey patch to http\httpsconnection that enables request pipelining. It might also be useful to use something like epoll to wait for the sockets.

mpeg 5y ago
The author very much did try to do that, doing requests with async APIs is often the most performant way to do them because under the hood they use IO async primitives like epoll/kqueue/iocp depending on the environment they run on. If you check the code in the article, there is an innocent looking call to decode the response as JSON; that's most likely where the CPU time is going, most JSON decoders are just not very fast. I remember back in the day having to write my own decoder for a JSON API because I was getting capped on around 250req/s on a windows machine (using c# and iocp which is pretty fast). I think I almost doubled performance by taking some decoding shortcuts when the JSON structure was known ahead of time. TL;DR: network I/O is fast, json is a really slow format to parse.
- anamax 5y ago
  Json decoding will show up as CPU usage. Epoll to decide which socket to read is just one thing - efficiently using sockets also matters, hence my suggestion of communication pipelining.

by anamax 5y ago

mpeg 5y ago

The author very much did try to do that, doing requests with async APIs is often the most performant way to do them because under the hood they use IO async primitives like epoll/kqueue/iocp depending on the environment they run on. If you check the code in the article, there is an innocent looking call to decode the response as JSON; that's most likely where the CPU time is going, most JSON decoders are just not very fast. I remember back in the day having to write my own decoder for a JSON API because I was getting capped on around 250req/s on a windows machine (using c# and iocp which is pretty fast). I think I almost doubled performance by taking some decoding shortcuts when the JSON structure was known ahead of time. TL;DR: network I/O is fast, json is a really slow format to parse.

anamax 5y ago
Json decoding will show up as CPU usage. Epoll to decide which socket to read is just one thing - efficiently using sockets also matters, hence my suggestion of communication pipelining.