8 ms·

Easy cluster parallelization with ZeroMQ

64 points by mdup 11y ago

gh02t 11y ago
Worth mentioning that the cluster tools in IPython implement basically the same system as he ends up with, plus lots of other functionality and it is using ZMQ as the backend as well. If you've got a small to medium sized task (I'd say less than 500 nodes is a good number) to run on your cluster ipcluster is pretty great. Even when I have a proper queuing system like SLURM or PBS in place, I still often use ipcluster on top. http://ipython.org/ipython-doc/dev/parallel/ http://ipython.org/ipython-doc/dev/parallel/
- nickpsecurity 11y ago
  Thanks for the link. That might come in handy if I do any more clustering.
- andreabedini 11y ago
  I'm curious on how you use ipcluster on top of pbs
  - gh02t 11y ago
    If you mean the practicalities of how to use it, they describe it a bit here http://ipython.org/ipython-doc/dev/parallel/parallel_process.html#using-ipcluster-in-pbs-mode http://ipython.org/ipython-doc/dev/parallel/parallel_process... . If you mean how have I used it specifically, I've used it to do a variety of things. Ranging from recreating what was described in the original post to using it as an ad hoc MPI replacement. I find it's handy for managing jobs that have some coupling (and so aren't easily suited to running via the features of the batch system), but aren't tightly coupled enough to warrant the suffering of having to write MPI-based code.
Merkur 11y ago
I had fun reading - thx. I found the solution of "requesting workers" interesting. My first reflex to the round robin argument was - a better load balancer but the requesting worker made that totally not necessary! I wounder what negative sites sutch architectures may have?
nickpsecurity 11y ago
Nice article. I hope to see a benchmark of it on a less parallel problem with low-latency network hardware. Maybe a comparison of that against a vanilla MPI or parallel framework. We'll get to see what its real overhead is.
- gh02t 11y ago
  It's a bit hard to compare to MPI, which is considerably lower level and is oriented towards tightly coupled compute clusters. MPI is more about passing around data structures and not so much about dispatch and execution of independent tasks, which is more of a distributed computing type endeavor. You can build something like this on top of MPI (and I did once as an experiment), but it is tough to compare them directly since they are designed on different assumptions.
  - nickpsecurity 11y ago
    Fair point. I've been out of cluster computing for years so what do you think would be comparable?
    - dalke 11y ago
      Somewhat tangential, but you may be interested in this essay on "HPC is dying and MPI is killing it" http://www.dursi.ca/hpc-is-dying-and-mpi-is-killing-it/ http://www.dursi.ca/hpc-is-dying-and-mpi-is-killing-it/ . There were many comments here on HN, at https://news.ycombinator.com/item?id=9335441 https://news.ycombinator.com/item?id=9335441 . The essay also has a couple of followups. It really is an excellent essay. The author suggests GASNet, Charm++, and several other things that would be comparable to MPI.
      - nickpsecurity 11y ago
        That's an excellent essay. I agree with about every point. I worked MPI over ten years ago. It seemed like an assembler language for cluster computing: something for raw efficiency or to hold us over until we get a real language. I was playing with Cilk and other parallel tools then. Much better but little adoption or optimization. The examples given in Spark and Chapel show how much better it can be. Such methods deserve much more investment. The bad thing is that the DOE labs are clinging so hard to their existing tools when they're in the best position to invest in new ones via huge grants they have. They built supercomputing before anyone heard of the cloud. Their wisdom combined with today's datacenter engineers could result in anything from great leaps in efficiency to something revolutionary. Yet, they act elitist and value backward compatibility over everything else. Starting to look like those mainframe programmers squeezing every ounce of productivity they can out of COBOL. I think the change will have to come from outside. A group that works both sides with funding to do new projects. They either (a) improve MPI usage for new workloads or (b) get the alternatives up to MPI performance for HPC workloads. Then, they open the tool up to all kinds of new projects in both HPC and cloud-centered research (esp cloud compute clusters). Maybe the two sides will accidentally start sharing with each other when contributing to the same projects. Not much hope past that.
avargas 11y ago
ZeroMQ is amazing. A few years ago I built a prototype project for a client that basically was fail2ban but scalable. It monitored nginx logs and broadcasted some information, and workers did the rest. Most of the data was in memory and passed around, and communication was done via ZeroMQ. It was done this way so that we could split the heavy-load components off the server and into workers, and allow the server to simply do it's job: tail nginx logs and act upon ban requests from workers. It was amazing, sadly I never completed it and deployed it on a production environment but from initial tests, it outperformed fail2ban by a lot.
FullyFunctional 11y ago
Great writeup. Perhaps slightly less clean, you could cut the messages in half by having the available report the result as well (empty in the initial case). Is the worker idling between replying and receiving new work? In other words, does ZMQ enable overlapping, say always having two outstanding requests per worker?

64 points by mdup 11y ago

gh02t 11y ago

Worth mentioning that the cluster tools in IPython implement basically the same system as he ends up with, plus lots of other functionality and it is using ZMQ as the backend as well. If you've got a small to medium sized task (I'd say less than 500 nodes is a good number) to run on your cluster ipcluster is pretty great. Even when I have a proper queuing system like SLURM or PBS in place, I still often use ipcluster on top. http://ipython.org/ipython-doc/dev/parallel/ http://ipython.org/ipython-doc/dev/parallel/

nickpsecurity 11y ago
Thanks for the link. That might come in handy if I do any more clustering.
andreabedini 11y ago
I'm curious on how you use ipcluster on top of pbs
- gh02t 11y ago
  If you mean the practicalities of how to use it, they describe it a bit here http://ipython.org/ipython-doc/dev/parallel/parallel_process.html#using-ipcluster-in-pbs-mode http://ipython.org/ipython-doc/dev/parallel/parallel_process... . If you mean how have I used it specifically, I've used it to do a variety of things. Ranging from recreating what was described in the original post to using it as an ad hoc MPI replacement. I find it's handy for managing jobs that have some coupling (and so aren't easily suited to running via the features of the batch system), but aren't tightly coupled enough to warrant the suffering of having to write MPI-based code.

Merkur 11y ago

I had fun reading - thx. I found the solution of "requesting workers" interesting. My first reflex to the round robin argument was - a better load balancer but the requesting worker made that totally not necessary! I wounder what negative sites sutch architectures may have?

nickpsecurity 11y ago

Nice article. I hope to see a benchmark of it on a less parallel problem with low-latency network hardware. Maybe a comparison of that against a vanilla MPI or parallel framework. We'll get to see what its real overhead is.

gh02t 11y ago
It's a bit hard to compare to MPI, which is considerably lower level and is oriented towards tightly coupled compute clusters. MPI is more about passing around data structures and not so much about dispatch and execution of independent tasks, which is more of a distributed computing type endeavor. You can build something like this on top of MPI (and I did once as an experiment), but it is tough to compare them directly since they are designed on different assumptions.
- nickpsecurity 11y ago
  Fair point. I've been out of cluster computing for years so what do you think would be comparable?
  - dalke 11y ago
    Somewhat tangential, but you may be interested in this essay on "HPC is dying and MPI is killing it" http://www.dursi.ca/hpc-is-dying-and-mpi-is-killing-it/ http://www.dursi.ca/hpc-is-dying-and-mpi-is-killing-it/ . There were many comments here on HN, at https://news.ycombinator.com/item?id=9335441 https://news.ycombinator.com/item?id=9335441 . The essay also has a couple of followups. It really is an excellent essay. The author suggests GASNet, Charm++, and several other things that would be comparable to MPI.
    - nickpsecurity 11y ago
      That's an excellent essay. I agree with about every point. I worked MPI over ten years ago. It seemed like an assembler language for cluster computing: something for raw efficiency or to hold us over until we get a real language. I was playing with Cilk and other parallel tools then. Much better but little adoption or optimization. The examples given in Spark and Chapel show how much better it can be. Such methods deserve much more investment. The bad thing is that the DOE labs are clinging so hard to their existing tools when they're in the best position to invest in new ones via huge grants they have. They built supercomputing before anyone heard of the cloud. Their wisdom combined with today's datacenter engineers could result in anything from great leaps in efficiency to something revolutionary. Yet, they act elitist and value backward compatibility over everything else. Starting to look like those mainframe programmers squeezing every ounce of productivity they can out of COBOL. I think the change will have to come from outside. A group that works both sides with funding to do new projects. They either (a) improve MPI usage for new workloads or (b) get the alternatives up to MPI performance for HPC workloads. Then, they open the tool up to all kinds of new projects in both HPC and cloud-centered research (esp cloud compute clusters). Maybe the two sides will accidentally start sharing with each other when contributing to the same projects. Not much hope past that.

avargas 11y ago

ZeroMQ is amazing. A few years ago I built a prototype project for a client that basically was fail2ban but scalable. It monitored nginx logs and broadcasted some information, and workers did the rest. Most of the data was in memory and passed around, and communication was done via ZeroMQ. It was done this way so that we could split the heavy-load components off the server and into workers, and allow the server to simply do it's job: tail nginx logs and act upon ban requests from workers. It was amazing, sadly I never completed it and deployed it on a production environment but from initial tests, it outperformed fail2ban by a lot.

FullyFunctional 11y ago

Great writeup. Perhaps slightly less clean, you could cut the messages in half by having the available report the result as well (empty in the initial case). Is the worker idling between replying and receiving new work? In other words, does ZMQ enable overlapping, say always having two outstanding requests per worker?