4 ms·

Back in 2014, I was rebuilding the core of an event processing engine. At the time, the decision was between Apache Kafka and rolling our own. After investiga

Back in 2014, I was rebuilding the core of an event processing engine. At the time, the decision was between Apache Kafka and rolling our own. After investigating Zookeeper, we decided to roll our own and chose ZeroMQ as the messaging layer as our on-prem customers probably didn't want to own and manage Zookeeper. ZeroMQ was absolutely solid and stable; incredibly trouble free and the only issues we ran into were when IT teams didn't open the ports we documented in the configuration procedure. (The resultant architecture actually looks a lot like Flink) In any case, ZeroMQ is a fantastic piece of technology that I feel like I don't see out in the wild quite enough. It's dead simple and incredibly stable from my experience.

vxxzy 3y ago
If you look close at the neuralink demo you’ll notice they use zeromq. I suspect the flexibility and the reduced latency makes a difference.
perbu 3y ago
It's also quite fast. I remember we had a similar use-case. This was for collecting live statistics from a sizable Varnish Cache cluster. We wrote a in memory database to store the data. It's been chugging away for about 10 years now and last I heard the zmq traffic alone was about 3Gbps with zero issues.
- formerly_proven 3y ago
  > It's also quite fast. IIRC it does all I/O on separate background threads which means every operation actually goes through an inter-thread queue. Which can be good and bad.
- ksimukka 3y ago
  Aside from some scaling issues, it still is a great solution for having real-time insight into the performance of Varnish and the cache. We even went as far as writing a Golang version of the VCS server to better handle some of the scaling challenges. I don’t remember the exact library that we are using to call into ZeroMQ, but the CGO overhead was minimal.
kabes 3y ago
In a company I work for they decided to do the same, around the same time. I believe it was a wrong call. Over time requirements have grown and we ended up bolting all kinds of kafka features on top of the zeromq thing, but of course much crappier. And in the meantime kafka doesn't require zookeeper anymore and is the de-facto standard
- CharlieDigital 3y ago
  Of course, ZeroMQ and Kafka are two very different tools that serve different purposes and one needs to understand the tradeoffs. For us, delivering an on-prem commercial off the shelf solution, it was untenable to expect the customer IT team to operate a separate, relatively huge piece of tech (remember, this is 2014). Maybe the heuristics would be different today with K8s and advancement of Kafka. But ZeroMQ as an in-process, distributed messaging layer is dead simple. If your use case requires anything else on top of that, it's on the team to design the right solution like resiliency, statefulness, etc. For a high throughput, distributed compute focused use case, I think ZMQ is still a great choice for coordinating execution. But Kafka and other options on the market now are great choices for higher order abstractions.
  - pc86 3y ago
    I worked on a similar greenfield project around the same time and we looked at RabbitMQ and Kafka, eventually going the RabbitMQ route. We were also developing an on-prem COTS product, and zookeeper played a big role in our decision to go with RMQ, not to mention at the time CloudAMQP had a very generous free tier (not so much anymore, but it's still okay-bordering-decent). No one install would ever hit the scale where Kafka makes sense so I still think it was a good call pushing 10 years later.
- castlecrasher2 3y ago
  A company I worked for had the same problem. Messages were being dropped, and either no one on backend knew how or wanted to investigate. I was on the data team and we just had to deal with it.
throwaway9870 3y ago
The problem is that it started out as any but stable and reliable. It asserted on received data, which in a network application is a super newbie mistake. When I looked at it, the pub/sub socket would hang forever if the other end interrupted the connection. So the zeromq guide which said "look how easy it is" was only true if you ignored errors. If you are writing network code and ignore errors, well, good luck. That was a long time ago (~10yrs) so if it is better now, good for them. Also, both founders have left the project. One passed from cancer, the other didn't like what he built and started over in C. Not that they can't be replaced, but transitions can be hard and take time.
- chomp 3y ago
  > When I looked at it, the pub/sub socket would hang forever if the other end interrupted the connection. You just gave me openstack+zeromq flashbacks
- jvanderbot 3y ago
  Yes - nanomsg / nng are quite good. I think / believe they'll be part and parcel for some space missions soon enough.
- victor106 3y ago
  > It asserted on received data, which in a network application is a super newbie mistake What do mature network applications do instead?
  - praptak 3y ago
    TCP asserts on received data, so I guess GP really meant reliable message queues. They generally assert after commit to disk.
    - remexre 3y ago
      What does it mean to assert in this context? I assume not the same thing as C's assert macro; is it the EE sense of "pulling a pin high / low"?
flyinghamster 3y ago
If you have played around with GNU Radio, ZeroMQ is baked-in for communications with outside applications. I've played with it a bit, and found it to be very much fire-and-forget once you have it set up properly.
- talent_deprived 3y ago
  That's cool, I've played with GNU Radio some, wasn't aware it used ZeroMQ, actually, I've used several *MQ's but not ZeroMQ, guess I should look into it more.
dist-epoch 3y ago
NATS is today's ZeroMQ, even easier and more reliable.
- CharlieDigital 3y ago
  Is it? ZeroMQ runs in-process as a library. NATS looks like it requires a server.
  - dist-epoch 3y ago
    Fair. Having used both (from Python) I found NATS much better. ZeroMQ in particular caused a lot of problems around the HWM (High Water Mark) limit for me or around process starting order (who creates what channel). Having a separate server helps with the last problem.
    - CharlieDigital 3y ago
      ZMQ for me fills a very specific use case of needing high-throughput, _in-process_ distributed messaging. I think once the _in-process_ constraint is lifted -- you can install or rely on an external messaging server -- then the field becomes much wider in terms of solutions to pick from. BTW, the way we solved for a similar HWM issue is that we decoupled the ingestion of events from the distribution of said events (with ZMQ). So one process was ingesting events and would send it to a coordinator process that would then send events to processing nodes. The coordinator would reply with it's current queue size and the ingest process would do a back-off once it reached a threshold. This allowed us to stop/start any node in the cluster in any order.
  - cmrdporcupine 3y ago
    Exactly, zmq is far more flexible in deployment options because of this. We (at work) use ZMQ + our own special sauces to run vehicle automation messaging stuff. Running some enterprise-ish broker in that environment just isn't on the menu.
  - traceroute66 3y ago
    No. It doesn't. Sure, to get the most out of NATS you should run it as a server (which frankly isn't difficult as its a single Go binary). But you can embed it if you so wish. Indeed if you look at the NATS source code[1], that is exactly how NATS do their testing. They use an embedded NATS server in their test routines. [1]https://github.com/nats-io/nats-server/tree/main/test https://github.com/nats-io/nats-server/tree/main/test
cmrdporcupine 3y ago
The problem with ZeroMQ is that it is highly opinionated about threading and concurrency in a way that doesn't necessarily mesh well with other components of your application stack that have their own opinions http://wiki.zeromq.org/blog:multithreading-magic http://wiki.zeromq.org/blog:multithreading-magic There's nothing wrong with the opinions presented there -- but it really wants to own your whole threading / serving stack, and isn't really compatible with e.g. a Rust async tokio binary where threads can "move around" etc.
- wallstprog 3y ago
  Well, the original non-thread-safe sockets (e.g., ZMQ_SUB, etc.) were that way because they support multi-part messages. And yes, the fact that they are not thread-safe is a bit of a stumbling-block for ZeroMQ newbies (including me, back in the day). However, it's not that hard to work around their non-thread-safe nature with a single dispatch thread (e.g., see <https://github.com/nyfix/OZ/blob/master/src/transport.c#L457 https://github.com/nyfix/OZ/blob/master/src/transport.c#L457>), and that approach has other potential benefits. In any case, newer socket types (e.g., ZMQ_CLIENT) have since been defined that are thread-safe, but necessarily don't support multi-part messages. (They tend to be different in other ways as well -- e.g., ZMQ_RADIO/ZMQ_DISH are "sort of" replacements for the original ZMQ_PUB/ZMQ_SUB sockets, but have other constraints as well).
deleted 3y ago
[deleted]
giovannibonetti 3y ago
I wonder if this is one of those problems Erlang and the Beam VM solve out of the box. Native clustering, message passing from one node to the other, no need for a separate dependency... That's the same design, isn't it? Disclaimer: I don't work with any of that tech right now, but I'm looking forward to working with Elixir, which is in the same ecosystem.
wallstprog 3y ago
At my day job we run > 50 million messages per day through ZeroMQ, and have been quite pleased with its performance and stability (https://github.com/nyfix/OZ/tree/master#who-uses-it https://github.com/nyfix/OZ/tree/master#who-uses-it). But the truth is that ZeroMQ has a steep learning curve, and some rather sharp edges that can leave nasty cuts if one is not careful. If you're interested in becoming familiar with ZeroMQ, you might want to check out our implementation: <https://github.com/nyfix/OZ/ https://github.com/nyfix/OZ/>.

vxxzy 3y ago

If you look close at the neuralink demo you’ll notice they use zeromq. I suspect the flexibility and the reduced latency makes a difference.

perbu 3y ago

It's also quite fast. I remember we had a similar use-case. This was for collecting live statistics from a sizable Varnish Cache cluster. We wrote a in memory database to store the data. It's been chugging away for about 10 years now and last I heard the zmq traffic alone was about 3Gbps with zero issues.

formerly_proven 3y ago
> It's also quite fast. IIRC it does all I/O on separate background threads which means every operation actually goes through an inter-thread queue. Which can be good and bad.
ksimukka 3y ago
Aside from some scaling issues, it still is a great solution for having real-time insight into the performance of Varnish and the cache. We even went as far as writing a Golang version of the VCS server to better handle some of the scaling challenges. I don’t remember the exact library that we are using to call into ZeroMQ, but the CGO overhead was minimal.

kabes 3y ago

In a company I work for they decided to do the same, around the same time. I believe it was a wrong call. Over time requirements have grown and we ended up bolting all kinds of kafka features on top of the zeromq thing, but of course much crappier. And in the meantime kafka doesn't require zookeeper anymore and is the de-facto standard

CharlieDigital 3y ago
Of course, ZeroMQ and Kafka are two very different tools that serve different purposes and one needs to understand the tradeoffs. For us, delivering an on-prem commercial off the shelf solution, it was untenable to expect the customer IT team to operate a separate, relatively huge piece of tech (remember, this is 2014). Maybe the heuristics would be different today with K8s and advancement of Kafka. But ZeroMQ as an in-process, distributed messaging layer is dead simple. If your use case requires anything else on top of that, it's on the team to design the right solution like resiliency, statefulness, etc. For a high throughput, distributed compute focused use case, I think ZMQ is still a great choice for coordinating execution. But Kafka and other options on the market now are great choices for higher order abstractions.
- pc86 3y ago
  I worked on a similar greenfield project around the same time and we looked at RabbitMQ and Kafka, eventually going the RabbitMQ route. We were also developing an on-prem COTS product, and zookeeper played a big role in our decision to go with RMQ, not to mention at the time CloudAMQP had a very generous free tier (not so much anymore, but it's still okay-bordering-decent). No one install would ever hit the scale where Kafka makes sense so I still think it was a good call pushing 10 years later.
castlecrasher2 3y ago
A company I worked for had the same problem. Messages were being dropped, and either no one on backend knew how or wanted to investigate. I was on the data team and we just had to deal with it.

throwaway9870 3y ago

The problem is that it started out as any but stable and reliable. It asserted on received data, which in a network application is a super newbie mistake. When I looked at it, the pub/sub socket would hang forever if the other end interrupted the connection. So the zeromq guide which said "look how easy it is" was only true if you ignored errors. If you are writing network code and ignore errors, well, good luck. That was a long time ago (~10yrs) so if it is better now, good for them. Also, both founders have left the project. One passed from cancer, the other didn't like what he built and started over in C. Not that they can't be replaced, but transitions can be hard and take time.

chomp 3y ago
> When I looked at it, the pub/sub socket would hang forever if the other end interrupted the connection. You just gave me openstack+zeromq flashbacks
jvanderbot 3y ago
Yes - nanomsg / nng are quite good. I think / believe they'll be part and parcel for some space missions soon enough.
victor106 3y ago
> It asserted on received data, which in a network application is a super newbie mistake What do mature network applications do instead?
- praptak 3y ago
  TCP asserts on received data, so I guess GP really meant reliable message queues. They generally assert after commit to disk.
  - remexre 3y ago
    What does it mean to assert in this context? I assume not the same thing as C's assert macro; is it the EE sense of "pulling a pin high / low"?

flyinghamster 3y ago

If you have played around with GNU Radio, ZeroMQ is baked-in for communications with outside applications. I've played with it a bit, and found it to be very much fire-and-forget once you have it set up properly.

talent_deprived 3y ago
That's cool, I've played with GNU Radio some, wasn't aware it used ZeroMQ, actually, I've used several *MQ's but not ZeroMQ, guess I should look into it more.

dist-epoch 3y ago

NATS is today's ZeroMQ, even easier and more reliable.

CharlieDigital 3y ago
Is it? ZeroMQ runs in-process as a library. NATS looks like it requires a server.
- dist-epoch 3y ago
  Fair. Having used both (from Python) I found NATS much better. ZeroMQ in particular caused a lot of problems around the HWM (High Water Mark) limit for me or around process starting order (who creates what channel). Having a separate server helps with the last problem.
  - CharlieDigital 3y ago
    ZMQ for me fills a very specific use case of needing high-throughput, _in-process_ distributed messaging. I think once the _in-process_ constraint is lifted -- you can install or rely on an external messaging server -- then the field becomes much wider in terms of solutions to pick from. BTW, the way we solved for a similar HWM issue is that we decoupled the ingestion of events from the distribution of said events (with ZMQ). So one process was ingesting events and would send it to a coordinator process that would then send events to processing nodes. The coordinator would reply with it's current queue size and the ingest process would do a back-off once it reached a threshold. This allowed us to stop/start any node in the cluster in any order.
- cmrdporcupine 3y ago
  Exactly, zmq is far more flexible in deployment options because of this. We (at work) use ZMQ + our own special sauces to run vehicle automation messaging stuff. Running some enterprise-ish broker in that environment just isn't on the menu.
- traceroute66 3y ago
  No. It doesn't. Sure, to get the most out of NATS you should run it as a server (which frankly isn't difficult as its a single Go binary). But you can embed it if you so wish. Indeed if you look at the NATS source code[1], that is exactly how NATS do their testing. They use an embedded NATS server in their test routines. [1]https://github.com/nats-io/nats-server/tree/main/test https://github.com/nats-io/nats-server/tree/main/test

cmrdporcupine 3y ago

The problem with ZeroMQ is that it is highly opinionated about threading and concurrency in a way that doesn't necessarily mesh well with other components of your application stack that have their own opinions http://wiki.zeromq.org/blog:multithreading-magic http://wiki.zeromq.org/blog:multithreading-magic There's nothing wrong with the opinions presented there -- but it really wants to own your whole threading / serving stack, and isn't really compatible with e.g. a Rust async tokio binary where threads can "move around" etc.

wallstprog 3y ago
Well, the original non-thread-safe sockets (e.g., ZMQ_SUB, etc.) were that way because they support multi-part messages. And yes, the fact that they are not thread-safe is a bit of a stumbling-block for ZeroMQ newbies (including me, back in the day). However, it's not that hard to work around their non-thread-safe nature with a single dispatch thread (e.g., see <https://github.com/nyfix/OZ/blob/master/src/transport.c#L457 https://github.com/nyfix/OZ/blob/master/src/transport.c#L457>), and that approach has other potential benefits. In any case, newer socket types (e.g., ZMQ_CLIENT) have since been defined that are thread-safe, but necessarily don't support multi-part messages. (They tend to be different in other ways as well -- e.g., ZMQ_RADIO/ZMQ_DISH are "sort of" replacements for the original ZMQ_PUB/ZMQ_SUB sockets, but have other constraints as well).

deleted 3y ago

[deleted]

giovannibonetti 3y ago

I wonder if this is one of those problems Erlang and the Beam VM solve out of the box. Native clustering, message passing from one node to the other, no need for a separate dependency... That's the same design, isn't it? Disclaimer: I don't work with any of that tech right now, but I'm looking forward to working with Elixir, which is in the same ecosystem.

wallstprog 3y ago

At my day job we run > 50 million messages per day through ZeroMQ, and have been quite pleased with its performance and stability (https://github.com/nyfix/OZ/tree/master#who-uses-it https://github.com/nyfix/OZ/tree/master#who-uses-it). But the truth is that ZeroMQ has a steep learning curve, and some rather sharp edges that can leave nasty cuts if one is not careful. If you're interested in becoming familiar with ZeroMQ, you might want to check out our implementation: <https://github.com/nyfix/OZ/ https://github.com/nyfix/OZ/>.