4 ms·

A couple of things offhand: - It has Python 2 and 3 support - It's a wrapper for the Python built-in "multiprocessing" library - It spreads out work over all

A couple of things offhand: - It has Python 2 and 3 support - It's a wrapper for the Python built-in "multiprocessing" library - It spreads out work over all cores (so the abstraction hides the ability to control the pool) Seems like a great way to get your feet wet with multiprocessing in Python, but it likely has limited use in production...although certain infrastructures like resource limited containers might be able to accommodate it.

rcthompson 10y ago
Is there no way to specify the number of cores to use?
- gshulegaard 10y ago
  You could certainly limit the process pool in the multiprocessing library itself: https://docs.python.org/2/library/multiprocessing.html https://docs.python.org/2/library/multiprocessing.html ... but it doesn't seem like this decorator abstraction accounts for it. > That's it, two lines of changes is all we need in order to parallelize this program. Now this program will make use of all the cores on the machine it's running on, allowing it to run significantly faster. > As an overview, DECO is mainly just a smart wrapper for Python's multiprocessing.pool. When @concurrent is applied to a function it replaces it with calls to pool.apply_async. Additionally when arguments are passed to pool.apply_async, DECO replaces any index mutable objects with proxies, allowing it to detect and synchronize mutations of these objects. The results of these calls can then be obtained by calling wait() on the concurrent function, invoking a synchronization event. I haven't really dug around the source code, but it sounds like not really.
- jonesetc 10y ago
  Looks like there is: https://github.com/alex-sherman/deco/blob/cee63391bf4c6d66ee40628fb6ab61e62d3bc236/deco/conc.py#L108-L117 https://github.com/alex-sherman/deco/blob/cee63391bf4c6d66ee...
  - gshulegaard 10y ago
    Interesting, so when they claim it automatically scales out to all cores what they mean is it defaults to 3 unless overridden.
    - lqdc13 10y ago
      Should really be multiprocessing.cpu_count() / 2 or something like that.
      - xapata 10y ago
        I found out recently that some tasks do better if there are slightly more processes than cores.
  - deleted 10y ago
    [deleted]
- tomkinstinch 10y ago
  If you check out the test file[1] in the repository for deco, it looks like you can specify the number of cores by setting the "processes" attribute of the concurrent-decorated function [object]. 1. https://github.com/alex-sherman/deco/blob/master/conc_test.py#L27 https://github.com/alex-sherman/deco/blob/master/conc_test.p... https://github.com/alex-sherman/deco/blob/cee63391bf4c6d66ee40628fb6ab61e62d3bc236/deco/conc.py#L112-L117 https://github.com/alex-sherman/deco/blob/cee63391bf4c6d66ee...
- vector57 10y ago
  There is in the latest version. Using like @concurrent(...) passes all arguments directly to Pool(...)
tomkinstinch 10y ago
It's not Py3 compatible yet, but from the commit log it seems like they're working on it (we could all help?).
- gshulegaard 10y ago
  I wonder what is keeping them from Python 3 support. I'll take a look, although I am currently stuck on Python 2.7 for my current project.
  - tomkinstinch 10y ago
    See this issue comment: https://github.com/alex-sherman/deco/issues/2#issuecomment-217776369 https://github.com/alex-sherman/deco/issues/2#issuecomment-2...
- vector57 10y ago
  I would love some help on this project! I have managed to get Python 3 support working in the 0.3 release for at least a few examples, and any further help with bug reporting would be very much appreciated.
zjaffee 10y ago
Does anyone know how this library addresses the Global Interpreter Lock (GIL) issue, as multiprocessing really isn't that great/is worse than running on a single thread in many versions of python.
- pdonis 10y ago
  > the Global Interpreter Lock (GIL) issue This applies to programs that run Python bytecode in multiple threads in the same process. Multiprocessing forks multiple processes, so there is no GIL issue.

rcthompson 10y ago

Is there no way to specify the number of cores to use?

gshulegaard 10y ago
You could certainly limit the process pool in the multiprocessing library itself: https://docs.python.org/2/library/multiprocessing.html https://docs.python.org/2/library/multiprocessing.html ... but it doesn't seem like this decorator abstraction accounts for it. > That's it, two lines of changes is all we need in order to parallelize this program. Now this program will make use of all the cores on the machine it's running on, allowing it to run significantly faster. > As an overview, DECO is mainly just a smart wrapper for Python's multiprocessing.pool. When @concurrent is applied to a function it replaces it with calls to pool.apply_async. Additionally when arguments are passed to pool.apply_async, DECO replaces any index mutable objects with proxies, allowing it to detect and synchronize mutations of these objects. The results of these calls can then be obtained by calling wait() on the concurrent function, invoking a synchronization event. I haven't really dug around the source code, but it sounds like not really.
jonesetc 10y ago
Looks like there is: https://github.com/alex-sherman/deco/blob/cee63391bf4c6d66ee40628fb6ab61e62d3bc236/deco/conc.py#L108-L117 https://github.com/alex-sherman/deco/blob/cee63391bf4c6d66ee...
- gshulegaard 10y ago
  Interesting, so when they claim it automatically scales out to all cores what they mean is it defaults to 3 unless overridden.
  - lqdc13 10y ago
    Should really be multiprocessing.cpu_count() / 2 or something like that.
    - xapata 10y ago
      I found out recently that some tasks do better if there are slightly more processes than cores.
- deleted 10y ago
  [deleted]
tomkinstinch 10y ago
If you check out the test file[1] in the repository for deco, it looks like you can specify the number of cores by setting the "processes" attribute of the concurrent-decorated function [object]. 1. https://github.com/alex-sherman/deco/blob/master/conc_test.py#L27 https://github.com/alex-sherman/deco/blob/master/conc_test.p... https://github.com/alex-sherman/deco/blob/cee63391bf4c6d66ee40628fb6ab61e62d3bc236/deco/conc.py#L112-L117 https://github.com/alex-sherman/deco/blob/cee63391bf4c6d66ee...
vector57 10y ago
There is in the latest version. Using like @concurrent(...) passes all arguments directly to Pool(...)

tomkinstinch 10y ago

It's not Py3 compatible yet, but from the commit log it seems like they're working on it (we could all help?).

gshulegaard 10y ago
I wonder what is keeping them from Python 3 support. I'll take a look, although I am currently stuck on Python 2.7 for my current project.
- tomkinstinch 10y ago
  See this issue comment: https://github.com/alex-sherman/deco/issues/2#issuecomment-217776369 https://github.com/alex-sherman/deco/issues/2#issuecomment-2...
vector57 10y ago
I would love some help on this project! I have managed to get Python 3 support working in the 0.3 release for at least a few examples, and any further help with bug reporting would be very much appreciated.

zjaffee 10y ago

Does anyone know how this library addresses the Global Interpreter Lock (GIL) issue, as multiprocessing really isn't that great/is worse than running on a single thread in many versions of python.

pdonis 10y ago
> the Global Interpreter Lock (GIL) issue This applies to programs that run Python bytecode in multiple threads in the same process. Multiprocessing forks multiple processes, so there is no GIL issue.