4 ms·
Worth mentioning that the cluster tools in IPython implement basically the same system as he ends up with, plus lots of other functionality and it is using ZMQ
by gh02t 11y ago
Worth mentioning that the cluster tools in IPython implement basically the same system as he ends up with, plus lots of other functionality and it is using ZMQ as the backend as well. If you've got a small to medium sized task (I'd say less than 500 nodes is a good number) to run on your cluster ipcluster is pretty great. Even when I have a proper queuing system like SLURM or PBS in place, I still often use ipcluster on top.
http://ipython.org/ipython-doc/dev/parallel/ http://ipython.org/ipython-doc/dev/parallel/
- nickpsecurity 11y agoThanks for the link. That might come in handy if I do any more clustering.
- andreabedini 11y agoI'm curious on how you use ipcluster on top of pbs
- gh02t 11y agoIf you mean the practicalities of how to use it, they describe it a bit here http://ipython.org/ipython-doc/dev/parallel/parallel_process.html#using-ipcluster-in-pbs-mode http://ipython.org/ipython-doc/dev/parallel/parallel_process... . If you mean how have I used it specifically, I've used it to do a variety of things. Ranging from recreating what was described in the original post to using it as an ad hoc MPI replacement. I find it's handy for managing jobs that have some coupling (and so aren't easily suited to running via the features of the batch system), but aren't tightly coupled enough to warrant the suffering of having to write MPI-based code.