3 ms·

I didn't look too closely, but I'm wondering if this is Python's GIL. So instead of nginx -> multiple ~independent Python processes, each async handler is fight

I didn't look too closely, but I'm wondering if this is Python's GIL. So instead of nginx -> multiple ~independent Python processes, each async handler is fighting for the lock, even if running on a different core. So read as 700 queries / core vs 700 queries / server. If so, in a slightly tweaked Python server setup, that'd be 3K-12K/s per server. For narrower benchmarking, keeping sequential, and doing container 1 pinned to core 1 <-> container 2 pinned to core 2 might tell an even clearer picture. I did enjoy the work overall, incl Graviton comparison. Likewise, OS X's painfully slow Docker impl has driven me to Windows w/ WSL2 -> Ubuntu + nvidia-docker, which has been night and day, so not as surprised that those numbers are weird.

miketheman 5y ago
Glad you liked it! In my example, I'm using an unmodified gunicorn runner to load the uvloop worker. So I'm still only using a single worker process. Once I start tweaking the `--workers` count, I get a much higher queries per second. And you're correct - this is a narrow benchmark, not designed to test total TPS or saturate resources.
- lmeyerov 5y ago
  Great, suspected something like that :) Interesting wrt workers -- how does TCP vs sockets start looking in a multiple worker + multicore scenario, esp. wrt peak QPS? That's more confusing for schedulers :) Also, FWIW, the GIL thing might even be true in the case of single core <> single vs multiple workers. Docker supports pinning (`cpuset`?), so should be pretty doable.. We actually have a fairly similar production setup, so up to GPUs confusing things, have been curious on how deep to look in upcoming scaling work. As a fun extra wrinkle, we also used to have nginx as a poor man's api gateway: `request -> [ nginx_container -[tcp]-> app_container1 -[tcp]-> nginx_container -[tcp]-> app_container2 -[tcp]-> ... ]`, but had enough quality issues that we removed the internal nginx indirections.
  - miketheman 5y ago
    Wow, that's a lot of indirection! I've had the same setup, but only two requests deep. I think that if there's added value of each nginx tier (if each service has static assets, for example), then there may be some benefit, but tracing through all the layers gets harder. Glad to hear you removed the internal layers!

miketheman 5y ago

Glad you liked it! In my example, I'm using an unmodified gunicorn runner to load the uvloop worker. So I'm still only using a single worker process. Once I start tweaking the `--workers` count, I get a much higher queries per second. And you're correct - this is a narrow benchmark, not designed to test total TPS or saturate resources.

lmeyerov 5y ago
Great, suspected something like that :) Interesting wrt workers -- how does TCP vs sockets start looking in a multiple worker + multicore scenario, esp. wrt peak QPS? That's more confusing for schedulers :) Also, FWIW, the GIL thing might even be true in the case of single core <> single vs multiple workers. Docker supports pinning (`cpuset`?), so should be pretty doable.. We actually have a fairly similar production setup, so up to GPUs confusing things, have been curious on how deep to look in upcoming scaling work. As a fun extra wrinkle, we also used to have nginx as a poor man's api gateway: `request -> [ nginx_container -[tcp]-> app_container1 -[tcp]-> nginx_container -[tcp]-> app_container2 -[tcp]-> ... ]`, but had enough quality issues that we removed the internal nginx indirections.
- miketheman 5y ago
  Wow, that's a lot of indirection! I've had the same setup, but only two requests deep. I think that if there's added value of each nginx tier (if each service has static assets, for example), then there may be some benefit, but tracing through all the layers gets harder. Glad to hear you removed the internal layers!