3 ms·
Every time I read an article like this, all I can think is, this is what Elixir and Erlang are built for. I love Python and use it on a daily basis, but develop
by wacowz 10y ago
Every time I read an article like this, all I can think is, this is what Elixir and Erlang are built for. I love Python and use it on a daily basis, but developing concurrent systems in Python is beginning to look like a questionable choice, while there exists tools better suited for this.
Celery works great, and the integration with django is superb. However, it's consistently been a memory hog for us. Hell, we had to write a shell script to regularly check for memory use by celery and take necessary precautionary steps.
- brianwawok 10y agoWhat level of concurrency is required though? 99.9% of web request only needs concurrency at the user level, which standard python + fork will do just nicely. Making a realtime system to crunch some massive numbers in 50 micros? Then you might want some C or Java concurrency. But much of the world is not coded at that level.
- wacowz 10y agoIt's often the case that some work needs to be done outside of the request/response lifecycle, eg external api calls, data aggregation, sending notifications etc. Most of it is essentially IO bound. We happen to have lots of such tasks and therefore use celery, but there's only so much celery can do, given it's concurrency level is bound by the number of cores available since it uses multiprocessing. An actor based approach or sth like Go's goroutines would be a better fit.
- bitwize 10y agoDeveloping concurrent systems in Python on Unix is a questionable choice. On Windows it's a doddle, because Windows supports true async I/O. See: http://pyparallel.org http://pyparallel.org
- wacowz 10y agoThe numbers for pyparallel are quite impressive.