3 ms·
Yes, it seems that there are some types of simulations that could be structured as numpy operations steered by a little Python code. But can this kind of code b
by leephillips 13y ago
Yes, it seems that there are some types of simulations that could be structured as numpy operations steered by a little Python code. But can this kind of code be run effectively on large multi-processor machines?
- yummyfajitas 13y agoArray operations tend to be highly parallelizable by nature. Numpy operations can certainly be parallelized, and even distributed. Take a look at blaze. https://github.com/ContinuumIO/blaze https://github.com/ContinuumIO/blaze
- leephillips 13y agoThat looks interesting. I know that numpy calculations should be straightforwardly parallelizable; my question, out of curiosity, was whether they were in practice.
- cdavid 13y agoNote also that a common bottleneck for array processing is memory bandwidth. Multithreading something that is memory bound will not get you much speed. There are tons of optimisation, new representations that can be experimented with for arrays. While NumPy is already reasonably fast, I am convinced you can get much faster by expanding it (within or outside it). String/Object arrays nowhere near as useful as they could be as well.
- MichaelSalib 13y agonumexpr (a tool used to optimize performance of numpy code) has support for parallelizing operations. See http://code.google.com/p/numexpr/wiki/MultiThreadVM http://code.google.com/p/numexpr/wiki/MultiThreadVM