3 ms·

"ZMQ still relies on a message passing model, which is orthogonal to the threaded shared memory model all our hardware actually presents to us." So what? There

"ZMQ still relies on a message passing model, which is orthogonal to the threaded shared memory model all our hardware actually presents to us." So what? There are all sorts of successful language and computing models which are light years removed from threaded shared memory. Message passing without sharing state, except for some lock-free primitives, is bloody brilliant for very scalable concurrency - as Erlang has shown us for coming up on 30 years now. "The advantage is more being able to go from threads to processes, then to the network, easily." That's the raison d'etre for ZMQ to be sure. But very pleasurable multithreading is a massive advantage too. If it scales out to nodes as well, so much the better. "That advantage is just as useless to most applications as threading is to achieve concurrency (rather than parallelism)." Threading using ZMQ message passing is a superb way to achieve concurrency (given that this is the real world and we can't all rewrite our apps from scratch in Go or Clojure). I'm a threading beginner and I've got video decoding, graphics handling and networking for a real time app all in separate threads going great guns without falling over or messing each other around - on a very low spec machine too I might add. On a four core workstation, all threads get a whole core to themselves no problem. Try it. You'll like it :)

nly 13y ago
Any locked (or lock-free) pointer queue with the right ownership semantics will give you what ZMQ does within a threaded program. A coroutine library will get you even more flexibility if you just want to setup pipelines. If you want greater scalability, there are lots of non-blocking I/O event libraries (libuv/libevent/boost asio). None of these solutions involve moving to a message passing model. I'm not discrediting that operating between threads is a handy addition to ZMQ, but using it for communication between threads seems like a lot of work (moving to that message passing model) if you never intend to leave one process. I do happen to like messaging, but when your data becomes complex it's a right dog...you're now in the world of protocol design.
- camperman 13y ago
  "Any locked (or lock-free) pointer queue with the right ownership semantics will give you what ZMQ does within a threaded program." OK, now I think you're trolling, perhaps by cutting and pasting from random technical manuals. What you said there makes no sense at all. Could you paste some example code please? "A coroutine library will get you even more flexibility if you just want to setup pipelines." Now I know you're trolling. Coroutines are completely useless for real concurrency like decoding audio and video in the background simultaneously - because only one coroutine can be executing at the same time. "If you want greater scalability, there are lots of non-blocking I/O event libraries (libuv/libevent/boost asio). None of these solutions involve moving to a message passing model." What if I don't really care about I/O? What if I have a small program that needs to use multiple threads on multiple cores to efficiently search a large problem space while doing video decoding in the background and having a responsive UI? I must just link to Boost? Er, no thanks. "I'm not discrediting that operating between threads is a handy addition to ZMQ, but using it for communication between threads seems like a lot of work (moving to that message passing model) if you never intend to leave one process." Compared to getting synchronization right, optimizing critical sections, debugging random failures under load, it's trivial. "I do happen to like messaging, but when your data becomes complex it's a right dog...you're now in the world of protocol design." I'm going to assume that you're still in school or not actually a programmer. ZeroMQ is used by AT&T, Cisco, EA, Los Alamos Labs, NASA, Weta Digital, Zynga, Spotify, Samsung Electronics, Microsoft, and CERN (I grabbed this list from its front page). CERN just published a report in which it evaluated ZeroMQ for a custom middleware solutions running 4000 server processes communicating with 80 000 devices for a total of 2 000 000 I/O points that could be queried. It aced the requirements. Erlang has used message passing for decades in the telecoms industry which has enormously complex data and unique realtime, distributed requirements. Protocol design is easy. Main thread creates child. Child loads video and reports its ready. Main thread sends START. Child thread receives START and starts. Child thread reports DONE when done. Was that so hard?

nly 13y ago

Any locked (or lock-free) pointer queue with the right ownership semantics will give you what ZMQ does within a threaded program. A coroutine library will get you even more flexibility if you just want to setup pipelines. If you want greater scalability, there are lots of non-blocking I/O event libraries (libuv/libevent/boost asio). None of these solutions involve moving to a message passing model. I'm not discrediting that operating between threads is a handy addition to ZMQ, but using it for communication between threads seems like a lot of work (moving to that message passing model) if you never intend to leave one process. I do happen to like messaging, but when your data becomes complex it's a right dog...you're now in the world of protocol design.

camperman 13y ago
"Any locked (or lock-free) pointer queue with the right ownership semantics will give you what ZMQ does within a threaded program." OK, now I think you're trolling, perhaps by cutting and pasting from random technical manuals. What you said there makes no sense at all. Could you paste some example code please? "A coroutine library will get you even more flexibility if you just want to setup pipelines." Now I know you're trolling. Coroutines are completely useless for real concurrency like decoding audio and video in the background simultaneously - because only one coroutine can be executing at the same time. "If you want greater scalability, there are lots of non-blocking I/O event libraries (libuv/libevent/boost asio). None of these solutions involve moving to a message passing model." What if I don't really care about I/O? What if I have a small program that needs to use multiple threads on multiple cores to efficiently search a large problem space while doing video decoding in the background and having a responsive UI? I must just link to Boost? Er, no thanks. "I'm not discrediting that operating between threads is a handy addition to ZMQ, but using it for communication between threads seems like a lot of work (moving to that message passing model) if you never intend to leave one process." Compared to getting synchronization right, optimizing critical sections, debugging random failures under load, it's trivial. "I do happen to like messaging, but when your data becomes complex it's a right dog...you're now in the world of protocol design." I'm going to assume that you're still in school or not actually a programmer. ZeroMQ is used by AT&T, Cisco, EA, Los Alamos Labs, NASA, Weta Digital, Zynga, Spotify, Samsung Electronics, Microsoft, and CERN (I grabbed this list from its front page). CERN just published a report in which it evaluated ZeroMQ for a custom middleware solutions running 4000 server processes communicating with 80 000 devices for a total of 2 000 000 I/O points that could be queried. It aced the requirements. Erlang has used message passing for decades in the telecoms industry which has enormously complex data and unique realtime, distributed requirements. Protocol design is easy. Main thread creates child. Child loads video and reports its ready. Main thread sends START. Child thread receives START and starts. Child thread reports DONE when done. Was that so hard?