4 ms·

this is such a good idea! the @synchronized decorator to collect the parallelized task at the end of a parent call is very very smart & simple.

nomel 10y ago
It also has a potential for slowing down your code if used willy nilly, of course. Multiprocessing gets pretty useless for anything outside of independent CPU bound tasks with little IPC and simple data types that can be stuck into shared memory. If you're using multiprocessing pools so often that you think you need a decorator to clean up your code, then wow, I'd like to see what you're up to. ;)
- awinter-py 10y ago
  you guessed right that this isn't directly useful to my life. all my parallelism is already taken care of by a framework. that said I really like the idea of using inner and outer function calls as a hook for spawning and collecting promises. it doesn't only have to be a join on a CPU-bound worker pool; this feels like a cleaner way to abstract IO waits than the yield statements I saw in an early prototype of tulip. In general, this feels like a clean way to compose any library logic that involves an event loop or execution plan rather than just a function call.
- m_mueller 10y ago
  > If you're using multiprocessing pools so often that you think you need a decorator to clean up your code, then wow, I'd like to see what you're up to. ;) .. and I think once you do that in Python you should probably use NumPy. I've been thinking that there should be a way to just program kernels in Modern Fortran because it's the easiest to interact with NumPy data structures (NumPy is nothing but glue code around a collection of very efficient Fortran numeric code). f2py [1] is doing that basically, but I've never had the chance to setup a project like this. [1] http://docs.scipy.org/doc/numpy-1.10.1/user/c-info.python-as-glue.html http://docs.scipy.org/doc/numpy-1.10.1/user/c-info.python-as...

nomel 10y ago

It also has a potential for slowing down your code if used willy nilly, of course. Multiprocessing gets pretty useless for anything outside of independent CPU bound tasks with little IPC and simple data types that can be stuck into shared memory. If you're using multiprocessing pools so often that you think you need a decorator to clean up your code, then wow, I'd like to see what you're up to. ;)

awinter-py 10y ago
you guessed right that this isn't directly useful to my life. all my parallelism is already taken care of by a framework. that said I really like the idea of using inner and outer function calls as a hook for spawning and collecting promises. it doesn't only have to be a join on a CPU-bound worker pool; this feels like a cleaner way to abstract IO waits than the yield statements I saw in an early prototype of tulip. In general, this feels like a clean way to compose any library logic that involves an event loop or execution plan rather than just a function call.
m_mueller 10y ago
> If you're using multiprocessing pools so often that you think you need a decorator to clean up your code, then wow, I'd like to see what you're up to. ;) .. and I think once you do that in Python you should probably use NumPy. I've been thinking that there should be a way to just program kernels in Modern Fortran because it's the easiest to interact with NumPy data structures (NumPy is nothing but glue code around a collection of very efficient Fortran numeric code). f2py [1] is doing that basically, but I've never had the chance to setup a project like this. [1] http://docs.scipy.org/doc/numpy-1.10.1/user/c-info.python-as-glue.html http://docs.scipy.org/doc/numpy-1.10.1/user/c-info.python-as...