5 ms·

Distributed Systems with ZeroMQ

65 points by jpro 14y ago

KenCochrane 14y ago
If you like this, you should check out ZeroRPC, it handles a lot of the boilerplate code you would need to write by hand. Links: - http://zerorpc.dotcloud.com http://zerorpc.dotcloud.com - https://github.com/dotcloud/zerorpc-python https://github.com/dotcloud/zerorpc-python
jandrewrogers 14y ago
To make a distinction that the article does not (but should), ZeroMQ is a network transport abstraction layer. It is not useful for many distributed systems despite the title for the same reason. For non-trivial distributed systems, behaviors that are correct for network transport use cases that are built into ZeroMQ are pathological for distributed system use cases. In the specific case of ZeroMQ, there is a deep underlying assumption that logical queues are fundamentally independent things. As a corollary, scheduling what work is done on which queues is of no consequence as long as the contract of the individual queues is upheld. For any non-trivial distributed system, the above assumption is not true. Correct and scalable scheduling of operations is a function of the current status of all logical queues visible to a process. The processing priority of one queue is dependent on the current status of all other queues, which can change from operation to operation. Distributed systems are cooperatively scheduled and much of the self-balancing behavior of good distributed system designs come from this adaptive scheduling behavior. Unfortunately, systems like ZeroMQ intentionally hide and encapsulate all of the properties of the network transport that would be used to inform the scheduling of operations over a set of logical queues. And if you assume logical queues are fundamentally independent from a scheduling perspective then that is a good design. ZeroMQ is a good for moving bits over a network but it is often not a correct choice for distributed systems.
- rdtsc 14y ago
  All good points. Never thought of it that way. Are you thinking of RabbitMQ and distributed queues? Is it about all the distributed processes seeing a consistent queue state, or are you talking in general about having a quick feedback 'loading' value for each worker (as each grabs an item of the the queue, they are busy and will be less likely to grab one, so other less busy ones can take over).?
- snprbob86 14y ago
  > For non-trivial distributed systems, behaviors that are correct for network transport use cases that are built into ZeroMQ are pathological for distributed system use cases. I'm not sure what you mean. Please provide one or more concrete examples. > ZeroMQ is a good for moving bits over a network but it is often not a correct choice for distributed systems. It seems to me that distributed systems are defined by moving bits over a network. You're right that there is some confusion regarding what ZeroMQ actually us. This is in no small part due to it's unfortunate name: ZeroMQ isn't a queue. It is implemented with queues, but that's because it sends and receives discrete messages instead of bytes, like it's underlying protocols. The underlying transport protocols use "buffers" to store bytes on their way from your application to their destination. When you have a buffer of messages, that's called a queue! ZeroMQ is a network protocol for messaging. If you need custom scheduling, load balancing, etc, you can send and receive control messages on a secondary channel. The ZeroMQ Guide [1] is extremely enlightening in this regard. It's worth reading even if you never use ZeroMQ because all of the underlying principals apply to any network transport. [1] http://zguide.zeromq.org/page:all http://zguide.zeromq.org/page:all
  - scott_s 14y ago
    It seems to me that distributed systems are defined by moving bits over a network. Sorta. People say "distributed system" to mean a single, logical system - typically an application, or maybe some middleware that executes applications - that runs on many machines. So, yes, a distributed system will have bits moving over the network. But the point that jandrewrogers is making is that in a distributed system, the network connections are often dependent on each other. That is, sending data on one connection will impact the sending of data on another connection. When someone says "distributed systems," you're probably thinking of a client application talking to a server, and maybe that server talking to some backend. That's not what people mean when they use the term. They mean, in the large case, thousands of processes running across hundreds of hosts, and all of those processes may communicate with each other. In the client-server-backend case, it's okay to assume that each of those connections are independent, and to reason about them as such. In the thousand-to-thousand case in a distributed system, it's often not okay to assume that each of those connections is independent.
soravux 14y ago
We are currently using ZeroMQ in our distributed task framework in Python, SCOOP (http://scoop.googlecode.com http://scoop.googlecode.com). ZeroMQ was chosen as the communication library because it simply works and isn't bloated. No need to implement the state-machines for common patterns in our sockets, ZMQ does it and fast. It doesn't replace a standard socket, though, it only add a layer of functionalities over it. While using it, we found some minor negative point such as delays needed by the socket upon shutdown, which require sleeps between unit tests, or the random port connector that is not random... But overall, ZMQ is a tool that saved us much developing time and should not be overlooked by distribution systems.
nathancahill 14y ago
Required reading for anyone learning Python

65 points by jpro 14y ago

If you like this, you should check out ZeroRPC, it handles a lot of the boilerplate code you would need to write by hand. Links: - http://zerorpc.dotcloud.com http://zerorpc.dotcloud.com - https://github.com/dotcloud/zerorpc-python https://github.com/dotcloud/zerorpc-python

jandrewrogers 14y ago

To make a distinction that the article does not (but should), ZeroMQ is a network transport abstraction layer. It is not useful for many distributed systems despite the title for the same reason. For non-trivial distributed systems, behaviors that are correct for network transport use cases that are built into ZeroMQ are pathological for distributed system use cases. In the specific case of ZeroMQ, there is a deep underlying assumption that logical queues are fundamentally independent things. As a corollary, scheduling what work is done on which queues is of no consequence as long as the contract of the individual queues is upheld. For any non-trivial distributed system, the above assumption is not true. Correct and scalable scheduling of operations is a function of the current status of all logical queues visible to a process. The processing priority of one queue is dependent on the current status of all other queues, which can change from operation to operation. Distributed systems are cooperatively scheduled and much of the self-balancing behavior of good distributed system designs come from this adaptive scheduling behavior. Unfortunately, systems like ZeroMQ intentionally hide and encapsulate all of the properties of the network transport that would be used to inform the scheduling of operations over a set of logical queues. And if you assume logical queues are fundamentally independent from a scheduling perspective then that is a good design. ZeroMQ is a good for moving bits over a network but it is often not a correct choice for distributed systems.

rdtsc 14y ago
All good points. Never thought of it that way. Are you thinking of RabbitMQ and distributed queues? Is it about all the distributed processes seeing a consistent queue state, or are you talking in general about having a quick feedback 'loading' value for each worker (as each grabs an item of the the queue, they are busy and will be less likely to grab one, so other less busy ones can take over).?
snprbob86 14y ago
> For non-trivial distributed systems, behaviors that are correct for network transport use cases that are built into ZeroMQ are pathological for distributed system use cases. I'm not sure what you mean. Please provide one or more concrete examples. > ZeroMQ is a good for moving bits over a network but it is often not a correct choice for distributed systems. It seems to me that distributed systems are defined by moving bits over a network. You're right that there is some confusion regarding what ZeroMQ actually us. This is in no small part due to it's unfortunate name: ZeroMQ isn't a queue. It is implemented with queues, but that's because it sends and receives discrete messages instead of bytes, like it's underlying protocols. The underlying transport protocols use "buffers" to store bytes on their way from your application to their destination. When you have a buffer of messages, that's called a queue! ZeroMQ is a network protocol for messaging. If you need custom scheduling, load balancing, etc, you can send and receive control messages on a secondary channel. The ZeroMQ Guide [1] is extremely enlightening in this regard. It's worth reading even if you never use ZeroMQ because all of the underlying principals apply to any network transport. [1] http://zguide.zeromq.org/page:all http://zguide.zeromq.org/page:all
- scott_s 14y ago
  It seems to me that distributed systems are defined by moving bits over a network. Sorta. People say "distributed system" to mean a single, logical system - typically an application, or maybe some middleware that executes applications - that runs on many machines. So, yes, a distributed system will have bits moving over the network. But the point that jandrewrogers is making is that in a distributed system, the network connections are often dependent on each other. That is, sending data on one connection will impact the sending of data on another connection. When someone says "distributed systems," you're probably thinking of a client application talking to a server, and maybe that server talking to some backend. That's not what people mean when they use the term. They mean, in the large case, thousands of processes running across hundreds of hosts, and all of those processes may communicate with each other. In the client-server-backend case, it's okay to assume that each of those connections are independent, and to reason about them as such. In the thousand-to-thousand case in a distributed system, it's often not okay to assume that each of those connections is independent.

soravux 14y ago

We are currently using ZeroMQ in our distributed task framework in Python, SCOOP (http://scoop.googlecode.com http://scoop.googlecode.com). ZeroMQ was chosen as the communication library because it simply works and isn't bloated. No need to implement the state-machines for common patterns in our sockets, ZMQ does it and fast. It doesn't replace a standard socket, though, it only add a layer of functionalities over it. While using it, we found some minor negative point such as delays needed by the socket upon shutdown, which require sleeps between unit tests, or the random port connector that is not random... But overall, ZMQ is a tool that saved us much developing time and should not be overlooked by distribution systems.

nathancahill 14y ago

Required reading for anyone learning Python