7 ms·

CharmPy – A high-level parallel and distributed programming framework

69 points by lainon 8y ago

leetrout 8y ago
That seems like a very poor choice in name given the existence of PyCharm...
- abakus 8y ago
  On the other hand, it follows the Numpy, Scipy's convention of using py as suffix instead of prefix.
- davidmr 8y ago
  I'm not sure what better to call it. Charm++ (its core) was around ruining the lives of chemistry grad students and their sysadmins long, long before pycharm was a twinkle in its creators' eyes. I'm sure jetbrains won't be coming for it any time soon.
  - deadbeef404 8y ago
    Charmer? CharmIt? Charming? CharmHPD? Charmplus? Superintendentchalmers? The problem here isn't that PyCharm came second or that Jetbrains will have an issue. The problem here is that it's so many conversations will go > Have you used CharmPy? > You mean PyCharm? > No, CharmPy! > Are we talking about the same thing? > No > Oh, that's just confusing, then.
    - davidmr 8y ago
      I dunno, maybe. They're so completely different that it seems extremely unlikely to me that anyone will confuse the two in a conversation with any context whatsoever. The conversation you made up would almost certainly start with a discussion about scientific parallel programming. Any discussion I've ever had about charm++ (too many, I'm afraid) wouldn't have been confused for a discussion about an IDE. Either way, it's done.
ianbertolacci 8y ago
Very cool. However, this documentation is incomplete. It needs the full Python API listing.
- juanjgalvez 8y ago
  We'll update the documentation with the API in the next couple of days. Eventually there will also be a more comprehensive manual explaining every feature.
devxpy 8y ago
Can you please lay out the differences between this and Dask? https://dask.pydata.org/en/latest/ https://dask.pydata.org/en/latest/ I work on a parallel programming framework for python myself. Not geared towards performance, but the ease of use. http://zproc.readthedocs.io/en/latest/ http://zproc.readthedocs.io/en/latest/
- eslaught 8y ago
  As far as I know, Dask is at its core a tasking model (i.e. tasks have input and outputs, and run automatically when inputs are ready in a dataflow-like model). Charm++ (on which CharmPy is based) is an actor model. Think Erlang for HPC. You've got a set of objects that are all nominally running concurrently, and objects can send messages to one another. Personally, I prefer the task-based model (but of course I'm biased since I work on one myself). In a proper task-based model, you can't have races or deadlocks, everything looks to a first approximation to be sequential. In actor models there's pretty much no way to hide the conncurrency, and all the traditional pitfalls of parallel programming are exposed to the user.
- juanjgalvez 8y ago
  There are quite a few differences between them. Disclaimer: I work on CharmPy, and I'm not an expert on Dask, so my comments might be biased and not entirely accurate with respect to Dask. Obvious difference between the two is programming style. CharmPy (its current core API) is based on asynchronous method execution between distributed objects. Being objects they can have state and data which allows for a lot of flexibility. In Dask, you express a workflow as a series of dependent tasks (which as far as I know are stateless so it's more like functional programming) and dask schedules it for you. The scheduling is centralized (done in only one place, so it's like a master-worker pattern) even if you use the "distributed" scheduler (which is needed for multi-node runs). With CharmPy you can have truly distributed applications. Another thing I observed with the dask model is that, since everything needs to be translated into a task graph before execution, there seems to be poor support for mutable distributed numpy arrays. A mutation operation like modifying a single element of a distributed array is not allowed as far as I know (I have tried), and other mutation operations that are supported actually generate a completely different task graph as a result, with the overhead this entails. In charmpy, this restriction does not exist since you can just invoke a method on the object that holds the data that you want to modify, and do it in-place. In terms of performance, our initial tests have shown huge performance difference, with CharmPy being up to 200x faster (this is comparing with dask distributed scheduler for a very simple BSP-style program). Of course, difference will vary by workload, but one thing to note is that Dask is pure-python, while CharmPy's core runs in C/C++. The current level of task granularity that we can comfortably support is a few hundred microseconds, and we expect to improve it further. In contrast, the Dask documentation for the distributed scheduler explicitly warns against using small task granularity, recommending tasks larger than 100 ms duration. And something like Jug recommends tasks longer than 20 seconds. We are planning on adding other APIs on top of the core charmpy API, to accommodate other programming styles. For example, offer better support for the functional parallel programming style (there is a small example of parallel map in the codebase using charmpy), or task scheduling.
woodson 8y ago
Related but somewhat off-topic question: Is there an easy to use parallel processing framework for python that also works well on Windows (anaconda)? I keep having issues with lost processes and other more or less random crashes, no matter whether I use joblib or Dask, etc. All I really need is a parallel for or apply.
- juanjgalvez 8y ago
  You can install charmpy on Windows with pip if that works for you. Launching multiple processes is straightforward (you can check the documentation at charmpy.readthedocs.io). We don't offer an API yet in charmpy to explicitly do things like parallel apply (but will soon). You can however look at `examples/parallel-map/par-map.py` in the source code which shows a simple example of how to do it with the current API and might be what you are looking for.
wedn3sday 8y ago
This seems pretty cool, but I'm left with one question. In the example from the repo it states, "The following computes Pi in parallel, using any number of machines and processors." However, after reading through all the docs, I see no reference whatsoever to any multi-machine support, only multi-processing on a single machine. Can this span over a cluster? Although they make the claim that it will "scale to hundreds of thousands of cores" (which I want to believe, I would love to use this) without the API documentation to show me how to do this, their fancy library doesnt do much good.
- juanjgalvez 8y ago
  Hi. Yes, applications can span multiple nodes (e.g. in clusters and supercomputers), and is one of the main use cases of charm++/charmpy. The fact that you don't see anything in the API or examples is that application code is basically transparent to the amount of processes that are launched. What determines the number of processes used is the launcher (e.g. charmrun, or something like aprun or ibrun on other systems). During initialization, the charmpy runtime will figure out internally how many charmpy processes are active in the job. With charmrun, you can launch multiple processes in one host, but also across multiple hosts (by ssh'ing into each one and spawning the processes). This is done automatically by charmrun assuming you specify a list of hosts (called nodelist, see http://charm.cs.illinois.edu/manuals/html/charm++/C.html http://charm.cs.illinois.edu/manuals/html/charm++/C.html). Again, the application code is not affected by this. Similarly, on other systems you can launch charmpy applications with the system job launcher (e.g. aprun, sbatch, ibrun…). We have done so for example on Cray supercomputers. It is simple enough but we have to update the documentation to at least show an example of this.
  - wedn3sday 8y ago
    Thanks for the response! This is great, makes it pretty much a better version of mpi. Could you add this to the documentation, or if its already there maybe a link on the main doc page about running on a cluster?
targafarian 8y ago
from charmpy import * Please, for the love of God, import names explicitly or use e.g. `import charmpy as cp` and subsequently `cp.foo` so that reading example code we get a better sense of your API without having to guess which names were possibly overwritten.
- juanjgalvez 8y ago
  You must be referring to the example in the README. That is the only example in the source code or docs that uses `import *` as far as I'm aware. But yeah, I agree. It's fixed now.
  - targafarian 8y ago
    Much credit to you for changing that in the example. Will give it a try to see what the advantages/disadvantages are next time I need do do more complex tasks in parallel (I generally like Dask, but I would certainly need to try this out to know where each is better or worse--or if it's just a matter of taste).
Caminha 8y ago
Is there a way to interop that with MPI based tools (such as Trilinos or PyTrilinos)? I would really love to be able to write some of my unstructured mesh based simulations on something like that.
- juanjgalvez 8y ago
  For CharmPy this is not currently supported, but Charm++ can interoperate with MPI libraries, so with luck it wouldn't require much effort to get it working. I will open an issue on github to track this task.

69 points by lainon 8y ago

leetrout 8y ago

That seems like a very poor choice in name given the existence of PyCharm...

abakus 8y ago
On the other hand, it follows the Numpy, Scipy's convention of using py as suffix instead of prefix.
davidmr 8y ago
I'm not sure what better to call it. Charm++ (its core) was around ruining the lives of chemistry grad students and their sysadmins long, long before pycharm was a twinkle in its creators' eyes. I'm sure jetbrains won't be coming for it any time soon.
- deadbeef404 8y ago
  Charmer? CharmIt? Charming? CharmHPD? Charmplus? Superintendentchalmers? The problem here isn't that PyCharm came second or that Jetbrains will have an issue. The problem here is that it's so many conversations will go > Have you used CharmPy? > You mean PyCharm? > No, CharmPy! > Are we talking about the same thing? > No > Oh, that's just confusing, then.
  - davidmr 8y ago
    I dunno, maybe. They're so completely different that it seems extremely unlikely to me that anyone will confuse the two in a conversation with any context whatsoever. The conversation you made up would almost certainly start with a discussion about scientific parallel programming. Any discussion I've ever had about charm++ (too many, I'm afraid) wouldn't have been confused for a discussion about an IDE. Either way, it's done.

ianbertolacci 8y ago

Very cool. However, this documentation is incomplete. It needs the full Python API listing.

juanjgalvez 8y ago
We'll update the documentation with the API in the next couple of days. Eventually there will also be a more comprehensive manual explaining every feature.

devxpy 8y ago

Can you please lay out the differences between this and Dask? https://dask.pydata.org/en/latest/ https://dask.pydata.org/en/latest/ I work on a parallel programming framework for python myself. Not geared towards performance, but the ease of use. http://zproc.readthedocs.io/en/latest/ http://zproc.readthedocs.io/en/latest/

eslaught 8y ago
As far as I know, Dask is at its core a tasking model (i.e. tasks have input and outputs, and run automatically when inputs are ready in a dataflow-like model). Charm++ (on which CharmPy is based) is an actor model. Think Erlang for HPC. You've got a set of objects that are all nominally running concurrently, and objects can send messages to one another. Personally, I prefer the task-based model (but of course I'm biased since I work on one myself). In a proper task-based model, you can't have races or deadlocks, everything looks to a first approximation to be sequential. In actor models there's pretty much no way to hide the conncurrency, and all the traditional pitfalls of parallel programming are exposed to the user.
juanjgalvez 8y ago
There are quite a few differences between them. Disclaimer: I work on CharmPy, and I'm not an expert on Dask, so my comments might be biased and not entirely accurate with respect to Dask. Obvious difference between the two is programming style. CharmPy (its current core API) is based on asynchronous method execution between distributed objects. Being objects they can have state and data which allows for a lot of flexibility. In Dask, you express a workflow as a series of dependent tasks (which as far as I know are stateless so it's more like functional programming) and dask schedules it for you. The scheduling is centralized (done in only one place, so it's like a master-worker pattern) even if you use the "distributed" scheduler (which is needed for multi-node runs). With CharmPy you can have truly distributed applications. Another thing I observed with the dask model is that, since everything needs to be translated into a task graph before execution, there seems to be poor support for mutable distributed numpy arrays. A mutation operation like modifying a single element of a distributed array is not allowed as far as I know (I have tried), and other mutation operations that are supported actually generate a completely different task graph as a result, with the overhead this entails. In charmpy, this restriction does not exist since you can just invoke a method on the object that holds the data that you want to modify, and do it in-place. In terms of performance, our initial tests have shown huge performance difference, with CharmPy being up to 200x faster (this is comparing with dask distributed scheduler for a very simple BSP-style program). Of course, difference will vary by workload, but one thing to note is that Dask is pure-python, while CharmPy's core runs in C/C++. The current level of task granularity that we can comfortably support is a few hundred microseconds, and we expect to improve it further. In contrast, the Dask documentation for the distributed scheduler explicitly warns against using small task granularity, recommending tasks larger than 100 ms duration. And something like Jug recommends tasks longer than 20 seconds. We are planning on adding other APIs on top of the core charmpy API, to accommodate other programming styles. For example, offer better support for the functional parallel programming style (there is a small example of parallel map in the codebase using charmpy), or task scheduling.

woodson 8y ago

Related but somewhat off-topic question: Is there an easy to use parallel processing framework for python that also works well on Windows (anaconda)? I keep having issues with lost processes and other more or less random crashes, no matter whether I use joblib or Dask, etc. All I really need is a parallel for or apply.

juanjgalvez 8y ago
You can install charmpy on Windows with pip if that works for you. Launching multiple processes is straightforward (you can check the documentation at charmpy.readthedocs.io). We don't offer an API yet in charmpy to explicitly do things like parallel apply (but will soon). You can however look at `examples/parallel-map/par-map.py` in the source code which shows a simple example of how to do it with the current API and might be what you are looking for.

wedn3sday 8y ago

This seems pretty cool, but I'm left with one question. In the example from the repo it states, "The following computes Pi in parallel, using any number of machines and processors." However, after reading through all the docs, I see no reference whatsoever to any multi-machine support, only multi-processing on a single machine. Can this span over a cluster? Although they make the claim that it will "scale to hundreds of thousands of cores" (which I want to believe, I would love to use this) without the API documentation to show me how to do this, their fancy library doesnt do much good.

juanjgalvez 8y ago
Hi. Yes, applications can span multiple nodes (e.g. in clusters and supercomputers), and is one of the main use cases of charm++/charmpy. The fact that you don't see anything in the API or examples is that application code is basically transparent to the amount of processes that are launched. What determines the number of processes used is the launcher (e.g. charmrun, or something like aprun or ibrun on other systems). During initialization, the charmpy runtime will figure out internally how many charmpy processes are active in the job. With charmrun, you can launch multiple processes in one host, but also across multiple hosts (by ssh'ing into each one and spawning the processes). This is done automatically by charmrun assuming you specify a list of hosts (called nodelist, see http://charm.cs.illinois.edu/manuals/html/charm++/C.html http://charm.cs.illinois.edu/manuals/html/charm++/C.html). Again, the application code is not affected by this. Similarly, on other systems you can launch charmpy applications with the system job launcher (e.g. aprun, sbatch, ibrun…). We have done so for example on Cray supercomputers. It is simple enough but we have to update the documentation to at least show an example of this.
- wedn3sday 8y ago
  Thanks for the response! This is great, makes it pretty much a better version of mpi. Could you add this to the documentation, or if its already there maybe a link on the main doc page about running on a cluster?

targafarian 8y ago

from charmpy import * Please, for the love of God, import names explicitly or use e.g. `import charmpy as cp` and subsequently `cp.foo` so that reading example code we get a better sense of your API without having to guess which names were possibly overwritten.

juanjgalvez 8y ago
You must be referring to the example in the README. That is the only example in the source code or docs that uses `import *` as far as I'm aware. But yeah, I agree. It's fixed now.
- targafarian 8y ago
  Much credit to you for changing that in the example. Will give it a try to see what the advantages/disadvantages are next time I need do do more complex tasks in parallel (I generally like Dask, but I would certainly need to try this out to know where each is better or worse--or if it's just a matter of taste).

Caminha 8y ago

Is there a way to interop that with MPI based tools (such as Trilinos or PyTrilinos)? I would really love to be able to write some of my unstructured mesh based simulations on something like that.

juanjgalvez 8y ago
For CharmPy this is not currently supported, but Charm++ can interoperate with MPI libraries, so with luck it wouldn't require much effort to get it working. I will open an issue on github to track this task.