4 ms·

I am not a n00b programmer by any means but I've never needed to use threads until recently. Non-trivial threading - like decoding video in the background and c

I am not a n00b programmer by any means but I've never needed to use threads until recently. Non-trivial threading - like decoding video in the background and coordinating with audio kind of non-trivial - is hard. Difficult to reason about, unbelievably frustrating to debug, errors seem random, races, deadlocks - you name it, I struggled with it. That all went away when I discovered Zero MQ (http://zeromq.org/intro:read-the-manual http://zeromq.org/intro:read-the-manual). The manual claims: "If you've spent years learning tricks to make your MT code work at all, let alone rapidly, with locks and semaphores and critical sections, you will be disgusted when you realize it was all for nothing." It's not exaggerating. ZMQ's multithreading is marvelous. No mutexes, no locks, no semaphores, no critical sections, no load problems - just perfectly scalable threads that work with any language on any platform. In fact it's so cool, that if you don't have a problem that needs multithreading, go out and find one.

kasey_junk 13y ago
I've used Zero MQ a lot and really like it. It can make writing concurrency applications much simpler. That said, it is important to note that the library itself is using synchronization primitives. Understanding when and how ZMQ does this is very important for optimizing throughput.
- camperman 13y ago
  Yeah, but as an introduction to multithreading it's awesome. You will still learn how to create threads, how to communicate with them, how to synchronise them and how to avoid sharing state if you use Zero MQ. None of that is abstracted. But once you've got the principles down, then you can dig deeper or do it by hand with pthreads or your OS's library if you feel the need. The main pain with learning threading is you have to wrap your head around the principles of it - which are fairly foreign - while at the same time battling with the very strange and frustrating errors because it's so easy to get wrong. ZMQ teaches you the principles without the errors after which you can go master the low level bits if you wish.
nly 13y ago
ZMQ still relies on a message passing model, which is orthogonal to the threaded shared memory model all our hardware actually presents to us. The advantage is more being able to go from threads to processes, then to the network, easily. That advantage is just as useless to most applications as threading is to achieve concurrency (rather than parallelism).
- camperman 13y ago
  "ZMQ still relies on a message passing model, which is orthogonal to the threaded shared memory model all our hardware actually presents to us." So what? There are all sorts of successful language and computing models which are light years removed from threaded shared memory. Message passing without sharing state, except for some lock-free primitives, is bloody brilliant for very scalable concurrency - as Erlang has shown us for coming up on 30 years now. "The advantage is more being able to go from threads to processes, then to the network, easily." That's the raison d'etre for ZMQ to be sure. But very pleasurable multithreading is a massive advantage too. If it scales out to nodes as well, so much the better. "That advantage is just as useless to most applications as threading is to achieve concurrency (rather than parallelism)." Threading using ZMQ message passing is a superb way to achieve concurrency (given that this is the real world and we can't all rewrite our apps from scratch in Go or Clojure). I'm a threading beginner and I've got video decoding, graphics handling and networking for a real time app all in separate threads going great guns without falling over or messing each other around - on a very low spec machine too I might add. On a four core workstation, all threads get a whole core to themselves no problem. Try it. You'll like it :)
  - nly 13y ago
    Any locked (or lock-free) pointer queue with the right ownership semantics will give you what ZMQ does within a threaded program. A coroutine library will get you even more flexibility if you just want to setup pipelines. If you want greater scalability, there are lots of non-blocking I/O event libraries (libuv/libevent/boost asio). None of these solutions involve moving to a message passing model. I'm not discrediting that operating between threads is a handy addition to ZMQ, but using it for communication between threads seems like a lot of work (moving to that message passing model) if you never intend to leave one process. I do happen to like messaging, but when your data becomes complex it's a right dog...you're now in the world of protocol design.

kasey_junk 13y ago

I've used Zero MQ a lot and really like it. It can make writing concurrency applications much simpler. That said, it is important to note that the library itself is using synchronization primitives. Understanding when and how ZMQ does this is very important for optimizing throughput.

camperman 13y ago
Yeah, but as an introduction to multithreading it's awesome. You will still learn how to create threads, how to communicate with them, how to synchronise them and how to avoid sharing state if you use Zero MQ. None of that is abstracted. But once you've got the principles down, then you can dig deeper or do it by hand with pthreads or your OS's library if you feel the need. The main pain with learning threading is you have to wrap your head around the principles of it - which are fairly foreign - while at the same time battling with the very strange and frustrating errors because it's so easy to get wrong. ZMQ teaches you the principles without the errors after which you can go master the low level bits if you wish.

nly 13y ago

ZMQ still relies on a message passing model, which is orthogonal to the threaded shared memory model all our hardware actually presents to us. The advantage is more being able to go from threads to processes, then to the network, easily. That advantage is just as useless to most applications as threading is to achieve concurrency (rather than parallelism).

camperman 13y ago
"ZMQ still relies on a message passing model, which is orthogonal to the threaded shared memory model all our hardware actually presents to us." So what? There are all sorts of successful language and computing models which are light years removed from threaded shared memory. Message passing without sharing state, except for some lock-free primitives, is bloody brilliant for very scalable concurrency - as Erlang has shown us for coming up on 30 years now. "The advantage is more being able to go from threads to processes, then to the network, easily." That's the raison d'etre for ZMQ to be sure. But very pleasurable multithreading is a massive advantage too. If it scales out to nodes as well, so much the better. "That advantage is just as useless to most applications as threading is to achieve concurrency (rather than parallelism)." Threading using ZMQ message passing is a superb way to achieve concurrency (given that this is the real world and we can't all rewrite our apps from scratch in Go or Clojure). I'm a threading beginner and I've got video decoding, graphics handling and networking for a real time app all in separate threads going great guns without falling over or messing each other around - on a very low spec machine too I might add. On a four core workstation, all threads get a whole core to themselves no problem. Try it. You'll like it :)
- nly 13y ago
  Any locked (or lock-free) pointer queue with the right ownership semantics will give you what ZMQ does within a threaded program. A coroutine library will get you even more flexibility if you just want to setup pipelines. If you want greater scalability, there are lots of non-blocking I/O event libraries (libuv/libevent/boost asio). None of these solutions involve moving to a message passing model. I'm not discrediting that operating between threads is a handy addition to ZMQ, but using it for communication between threads seems like a lot of work (moving to that message passing model) if you never intend to leave one process. I do happen to like messaging, but when your data becomes complex it's a right dog...you're now in the world of protocol design.