3 ms·

Is there no way to specify the number of cores to use?

gshulegaard 10y ago
You could certainly limit the process pool in the multiprocessing library itself: https://docs.python.org/2/library/multiprocessing.html https://docs.python.org/2/library/multiprocessing.html ... but it doesn't seem like this decorator abstraction accounts for it. > That's it, two lines of changes is all we need in order to parallelize this program. Now this program will make use of all the cores on the machine it's running on, allowing it to run significantly faster. > As an overview, DECO is mainly just a smart wrapper for Python's multiprocessing.pool. When @concurrent is applied to a function it replaces it with calls to pool.apply_async. Additionally when arguments are passed to pool.apply_async, DECO replaces any index mutable objects with proxies, allowing it to detect and synchronize mutations of these objects. The results of these calls can then be obtained by calling wait() on the concurrent function, invoking a synchronization event. I haven't really dug around the source code, but it sounds like not really.
jonesetc 10y ago
Looks like there is: https://github.com/alex-sherman/deco/blob/cee63391bf4c6d66ee40628fb6ab61e62d3bc236/deco/conc.py#L108-L117 https://github.com/alex-sherman/deco/blob/cee63391bf4c6d66ee...
- gshulegaard 10y ago
  Interesting, so when they claim it automatically scales out to all cores what they mean is it defaults to 3 unless overridden.
  - lqdc13 10y ago
    Should really be multiprocessing.cpu_count() / 2 or something like that.
    - xapata 10y ago
      I found out recently that some tasks do better if there are slightly more processes than cores.
- deleted 10y ago
  [deleted]
tomkinstinch 10y ago
If you check out the test file[1] in the repository for deco, it looks like you can specify the number of cores by setting the "processes" attribute of the concurrent-decorated function [object]. 1. https://github.com/alex-sherman/deco/blob/master/conc_test.py#L27 https://github.com/alex-sherman/deco/blob/master/conc_test.p... https://github.com/alex-sherman/deco/blob/cee63391bf4c6d66ee40628fb6ab61e62d3bc236/deco/conc.py#L112-L117 https://github.com/alex-sherman/deco/blob/cee63391bf4c6d66ee...
vector57 10y ago
There is in the latest version. Using like @concurrent(...) passes all arguments directly to Pool(...)

You could certainly limit the process pool in the multiprocessing library itself: https://docs.python.org/2/library/multiprocessing.html https://docs.python.org/2/library/multiprocessing.html ... but it doesn't seem like this decorator abstraction accounts for it. > That's it, two lines of changes is all we need in order to parallelize this program. Now this program will make use of all the cores on the machine it's running on, allowing it to run significantly faster. > As an overview, DECO is mainly just a smart wrapper for Python's multiprocessing.pool. When @concurrent is applied to a function it replaces it with calls to pool.apply_async. Additionally when arguments are passed to pool.apply_async, DECO replaces any index mutable objects with proxies, allowing it to detect and synchronize mutations of these objects. The results of these calls can then be obtained by calling wait() on the concurrent function, invoking a synchronization event. I haven't really dug around the source code, but it sounds like not really.

jonesetc 10y ago

Looks like there is: https://github.com/alex-sherman/deco/blob/cee63391bf4c6d66ee40628fb6ab61e62d3bc236/deco/conc.py#L108-L117 https://github.com/alex-sherman/deco/blob/cee63391bf4c6d66ee...

gshulegaard 10y ago
Interesting, so when they claim it automatically scales out to all cores what they mean is it defaults to 3 unless overridden.
- lqdc13 10y ago
  Should really be multiprocessing.cpu_count() / 2 or something like that.
  - xapata 10y ago
    I found out recently that some tasks do better if there are slightly more processes than cores.
deleted 10y ago
[deleted]

tomkinstinch 10y ago

If you check out the test file[1] in the repository for deco, it looks like you can specify the number of cores by setting the "processes" attribute of the concurrent-decorated function [object]. 1. https://github.com/alex-sherman/deco/blob/master/conc_test.py#L27 https://github.com/alex-sherman/deco/blob/master/conc_test.p... https://github.com/alex-sherman/deco/blob/cee63391bf4c6d66ee40628fb6ab61e62d3bc236/deco/conc.py#L112-L117 https://github.com/alex-sherman/deco/blob/cee63391bf4c6d66ee...

vector57 10y ago

There is in the latest version. Using like @concurrent(...) passes all arguments directly to Pool(...)