5 ms·
I feel like the #1 downside of Python for the last few years is that you cannot take advantage of multiple cores of a CPU easily. Especially when you think it i
by ZuLuuuuuu 6y ago
I feel like the #1 downside of Python for the last few years is that you cannot take advantage of multiple cores of a CPU easily. Especially when you think it is heavily used in data analysis. We use Python for data analysis as well, and for 95% of operations we are doing, numpy is fast enough that we don't have any complaints. But sometimes, we do wish to be able to take advantage of all the cores in our CPUs, especially now that we can easily get an 8 core CPU for a reasonable price.
There is multiprocessing module but you cannot share memory between processes. I guess the best options are either writing a C extension or using Numba. Writing C extensions require either distributing binary packages or C compiler to be present on the target computer which is not always ideal. So is using Numba the best solution currently? I tried it a bit in the past but the errors I got was a bit hard to interpret compared to regular Python errors.
I wish the multithreading module had support for native threads. Is there any PEPs trying to bring easy multi-core support for Python?
- andi999 6y agoWhy not use the multiprocessing module? (if you can partition your problem space). I find memory sharing much more difficult to reliably implement an algorithm, and speed gain might be lower through cache conflicts.
- mhh__ 6y ago> cache conflicts This can be an absolute sinkhole for performance, but if you have a large dataset you shouldn't be sharing lines that much one would hope.
- andi999 6y agoThen you shd be able to do process parallelization.
- BlackFly 6y agoA real hard requirement is going to be resource pools such as session linked connection pools with associated data. But there are just plenty of places where you don't want to be forced into a message passing architecture for a service and shared memory may be a better fit.
- mhh__ 6y agoIf you are genuinely hitting a wall like that surely it's time to move on from python? Or at least think about it
- ZuLuuuuuu 6y agoThe thing is that the current ecosystem (numpy + scipy + pandas + PyCharm) fits our company structure perfectly (very few software engineers, mostly test engineers who are not very proficient coders). And we already have tens of thousands lines of code. So changing the whole ecosystem just because 5% of the problems are slow is too big of a jump for us. For now, it is easier for us to write a bit of C code for places where we absolutely need speed. But it would have been so convenient if we could just have native threads, so that we don't even need that 5% C code.
- mhh__ 6y agoDo you know what you are actually stalled on? i.e. Memory or Instructions, if it's the latter you can probably eek out some more performance just by fiddling with compiler flags (you'd be surprised simultaneously how clever and how utterly braindead some compilers are if you play with them on Compiler Explorer for a bit) Also, if you are running on Intel you might have some luck with Intel's profiling tools as, although I've never used them for python they are by far and away the best things in this area, they do support Python. I don't know what details they give you in from Python code directly, but if you've used perf to profile code at a low level before, vTune is like a Tom & Jerry shotgun in comparison.
- 6y ago
- szubie 6y agoRecent versions of Python (3.8+) have introduced a SharedMemory class for sharing memory between different processes (see https://docs.python.org/3/library/multiprocessing.shared_memory.html https://docs.python.org/3/library/multiprocessing.shared_mem...). The implementation may still be a bit buggy though, so use with caution! https://bugs.python.org/issue38119 https://bugs.python.org/issue38119
- ZuLuuuuuu 6y agoThank you, I'll definitely check it out!
- anentropic 6y agoThere are also some libraries built on top of it that might be useful https://github.com/dillonalaird/shared_numpy https://github.com/dillonalaird/shared_numpy
- ekianjo 6y ago> Python for the last few years is that you cannot take advantage of multiple cores of a CPU easily. Same thing in the R ecosystem, it's possible to use multiple cores with the parallel package but there are caveats too.
- curiousgal 6y agoI haven't run into any issues with that package honestly (mclapply ftw!). Other than not being able to use it on Windows of course.
- gbrown 6y agoIt’s process based, so communication overhead is heavy and the infrastructure is minimal - if you’re distributing jobs which are individually large it works fine, but it can’t really be used to replace multithreading in a shared memory model.
- clircle 6y agoI do as much data work as possible in data.table, which parallelizes things under the hood, and doesn't seem to require me to configure anything.
- killingtime74 6y agoTry Dask
- zozbot234 6y ago> I feel like the #1 downside of Python for the last few years is that you cannot take advantage of multiple cores of a CPU easily. Also a big downside with JavaScript. Of course both Python and JS are high-level interpreted languages where high-performance use cases aren't the foremost priority.
- azangru 6y ago> Also a big downside with JavaScript. Can't Node run multiple processes (or multiple workers, not sure of the proper terminology) on multiple cores? As discussed in this thread on StackOverflow: https://stackoverflow.com/questions/61893497/node-js-on-multi-core-processors https://stackoverflow.com/questions/61893497/node-js-on-mult...
- willseth 6y ago> Of course both Python and JS are high-level interpreted languages where high-performance use cases aren't the foremost priority. Modern JS engines are all primarily JIT-based, generally only interpreting code when it's faster than waiting on the JIT. JS JITs are quite good now, and in many cases will produce optimized C-equivalent compiled code from fairly naive JS.
- pletnes 6y agoNumpy operations release the GIL (usually at least) so you can use a threadpool and, indeed, share memory. Just try it and you may be pleasantly surprised. Dask is great if you’re processing large amounts of data, and it recommends and supports threads for this reason.
- ZuLuuuuuu 6y agoI didn't know that. So let's say I have 4 regular Python threads calling the same function, and in this function, let's say I call numpy.add on the same array (but different parts of the array), then will it actually use different cores for these 4 different threads? I will try it out, if it works, then that's actually great and would be super easy.
- pletnes 6y agoGreat, then my comment was worth it! Not sure there’s anything to prevent you from foot-shooting if you’re operating on the same array. I’d expect that this is only safe to do if, as you say, arrays are not overlapping. Let me know what you find.
- captain_price7 6y agoThe answer is No, isn't it?
- ZuLuuuuuu 5y agoI wrote a small code to test it. Unfortunately it didn't work, but I don't know if the problem is me, maybe I made a mistake...
- jeremiecoullon 6y agoI've been using JAX (https://jax.readthedocs.io/en/latest/ https://jax.readthedocs.io/en/latest/) for scientific computing in general (in particular MCMC algorithms), as it's really fast. Even on a CPU you get massive speedups compared to numpy (can be up to 2 or 3 orders of magnitude faster in some cases). The main selling point of the library is automatic differential and compilation to XLA, but I've been using it even when I don't need gradients, as it's really fast (due to compilation). I also really like the random number generator as it's very good for reproducibility. I've played around with Julia in the past and really liked it, but in terms speed Jax has pretty much solved that problem for me
- PartiallyTyped 6y agoHave you tried numba+numpy? In my experience, it is much faster than Jax and can compile to cuda. It's not caveat free, but it also removes the hustle of labeling arrays as donated in Jax. You may find this interesting https://github.com/scikit-hep/iminuit/blob/develop/tutorial/automatic_differentiation.ipynb https://github.com/scikit-hep/iminuit/blob/develop/tutorial/...
- cfcf14 6y agoHave you been successful in implementing non-trivial computational code in numba/numpy? I've always found it starts to really break for anything which isn't really trivial, and the errors are mostly non-prescriptive and highly verbose.
- craftinator 6y agoI had the same experience about 2 years ago. Maybe it's changed since then? It was nice when you had a pure math function to write, but otherwise seemed to be unreliable, especially in multithreaded and multiprocess situations.
- PartiallyTyped 6y agoI still have some issues in nojit mode, but jit is most of the time fine. It's still a bit iffy with lists, but most of the time I can use numpy arrays. TBF I am mainly using it for mostly pure path functions.
- CogitoCogito 6y ago> We use Python for data analysis as well, and for 95% of operations we are doing, numpy is fast enough that we don't have any complaints. But sometimes, we do wish to be able to take advantage of all the cores in our CPUs, especially now that we can easily get an 8 core CPU for a reasonable price. I understand the sentiment, but in this case I wonder if it is less a python issue and more an issue with numpy. I don't see any reason why numpy couldn't execute the last line of the following multi-threaded: >>> import numpy as np >>> a = np.arange(1000000000) >>> b = 2 \* a The same thing applies to matrix multiplication. I'm guessing numpy has chosen not to for good reason though. edit: Actually it seems numpy is multi-threaded under certain circumstances: https://stackoverflow.com/questions/16617973/why-isnt-numpy-mean-multithreaded https://stackoverflow.com/questions/16617973/why-isnt-numpy-...
- leephillips 6y agoNumpy will perform a parallelized calculation when using one of the underlying BLAS routines, if you have the appropriate libraries installed on your machine. So to take advantage of your cores, try to write the calculation as a call to a linear algebra function. Loops or expressions with arithmetic operators as in your example will run on one core. Or, just use Julia.
- N1H1L 6y agoTry dask Distribute your data and run everything as dask.delayed and then compute only at the end. Also check out legate.numpy from Nvidia which promises to be a drop in numpy replacement that will use all your CPU cores without any tweaks on your part. https://github.com/nv-legate/legate.numpy https://github.com/nv-legate/legate.numpy
- yummypaint 6y agoWhat I do is write single-threaded programs that process only a section of data, and manage them with HTCondor. It's some extra overhead and complexity but this way you can easily scale to hundreds of machines without changing anything. You also get mature queuing and job management tools that work independently of your program. If your application is easy to paralellize and you might need more than one machine i highly reccommend this route.