4 ms·
He's saying run 32 individual processes, not 32 threads within one process. Python's global interpreter lock will knobble you if you're using threads.
by jstanley 10y ago
He's saying run 32 individual processes, not 32 threads within one process. Python's global interpreter lock will knobble you if you're using threads.
- darpa_escapee 10y agoCelery has a worker pool of separate Python processes that jobs can be offloaded to. It side steps the GIL because it doesn't use threading.
- brianwawok 10y agoWell two ways. Either launch 32 or 64 copies of the process using multiprocessing, or 1000 threads with geventlet. I have never found a box I couldn't peg :) I guess if you had 1 gb ram and 16 cores it would be challenging in python.. but a few gb of ram and we are on.
- retrogradeorbit 10y agoWe are using processes in celery, not threads. Yet we still can't get to full utilisation. On an eight CPU machine we get around 650% - 750% CPU utilisation. It varies, it goes up and down in that range. We never get to 800%. The JVM version is just constantly 800%.