4 ms·

I've worked on more than one thousand+ machine distributed systems. These systems all use unix sockets extensively. Unix sockets are pretty much independent fro

I've worked on more than one thousand+ machine distributed systems. These systems all use unix sockets extensively. Unix sockets are pretty much independent from each other from the developer's perspective. Scheduling and coordination is not a concern of the socket layer of the stack. In most production distributed systems, there is an agreed upon common RPC protocol and a variety of ad-hoc messaging protocols. Often, this is HTTP for RPC coupled with a variety of job worker systems, maybe an "enterprise" message broker, and line-oriented protocol or two, and a few other custom services. ZeroMQ isn't trying to replace unix sockets. Nor is it trying to be a panacea for distributed systems. It's trying to standardize a set of sound topology patterns with robust implementations over a common protocol. It's a damn useful tool for distributed systems. If you're system has everyone all chatting at once and saturating your links, that's not ZeroMQ's problem any more than it's unix sockets' problem.

scott_s 14y ago
If you think "enterprise" when someone says "distributed system," then you are not talking about the same kind of thing that jandrewrogers and I are talking about. We're talking about high performance computing (HPC), where the communication patterns among nodes in a cluster can be complex and dominate performance. I disagree with his strong statement that if something like ZMQ works for you, it's not a distributed system, but otherwise, everything he has said is spot on.
jandrewrogers 14y ago
Any non-trivial distributed system will exhibit pathological feedback loops due to random events throughout the system synchronizing on each other and interacting in strange ways. It is a very common design problem for parallel systems. It manifests as suboptimal throughput in the best cases and total meltdown in the worst. The "cost" of a message operation is not fixed, and pathological operations are much more expensive by definition, so always selecting the least expensive operation that can be run at any point in time from the perspective of the node maximizes the aggregate throughput of the system. Of course, it also has the side effect of applying negative feedback to pathological positive feedback loops. This is actually a mathematics consequence, so there are limited workarounds. Nor is it intuitive. Making parts of the system slower, with no other changes, can increase the aggregate throughput of the distributed system. For non-trivial distributed systems, you are unlikely to produce a robustly optimal design by accident.

scott_s 14y ago

If you think "enterprise" when someone says "distributed system," then you are not talking about the same kind of thing that jandrewrogers and I are talking about. We're talking about high performance computing (HPC), where the communication patterns among nodes in a cluster can be complex and dominate performance. I disagree with his strong statement that if something like ZMQ works for you, it's not a distributed system, but otherwise, everything he has said is spot on.

jandrewrogers 14y ago

Any non-trivial distributed system will exhibit pathological feedback loops due to random events throughout the system synchronizing on each other and interacting in strange ways. It is a very common design problem for parallel systems. It manifests as suboptimal throughput in the best cases and total meltdown in the worst. The "cost" of a message operation is not fixed, and pathological operations are much more expensive by definition, so always selecting the least expensive operation that can be run at any point in time from the perspective of the node maximizes the aggregate throughput of the system. Of course, it also has the side effect of applying negative feedback to pathological positive feedback loops. This is actually a mathematics consequence, so there are limited workarounds. Nor is it intuitive. Making parts of the system slower, with no other changes, can increase the aggregate throughput of the distributed system. For non-trivial distributed systems, you are unlikely to produce a robustly optimal design by accident.