6 ms·

Multithreading Magic - Why Everything You Thought You Knew about Concurrency...

66 points by adulau 16y ago

po 16y ago
I like ZeroMQ but this slideshare doesn't deliver any substance at all. For an overview, skip it and check out the white paper (if you can call it that) cowritten by the same guy (Pieter Hintjens) instead: http://www.zeromq.org/whitepapers:multithreading-magic http://www.zeromq.org/whitepapers:multithreading-magic
- thijsterlouw 16y ago
  These slides were part of a "lightning talk" on ZeroMQ (video here: http://streaming.fosdem.org:8000/ferrer/0mq.ogv http://streaming.fosdem.org:8000/ferrer/0mq.ogv ). So I think Pieter just wanted to make the audience curious and not dive into the details. I do agree that it might be better to use more slides for patterns and use cases and fewer generic/actor-slides. Also the argument that it's not taught in university was a bit weak.
  - phintjens 16y ago
    There's a copy of the video here: http://www.youtube.com/watch?v=CCBYzKfmQ4U http://www.youtube.com/watch?v=CCBYzKfmQ4U The FOSDEM streaming server seems to be timing out right now.
- nikster 16y ago
  Dunno, made perfect sense to me. It's just that many (most?) programmers, in my own experience, don't fully appreciate the insanity that is traditional "shared data" concurrency. What I see is that they program something buggy, but where the bugs are rare enough to make the program work most times.
  - jshen 16y ago
    For many types of problems, shared state concurrency is not the nightmare it's often made out to be. For example, if your problem can fit into the consumer producer model then it's really easy to make that work on the jvm today.
- tbrownaw 16y ago
  The most widely used languages, C and C++, do not offer any support for concurrency. Programmers roll their own by using threading APIs. Languages that do support concurrency, such as Java, Python, .NET, and Ruby, do it in a brute-force manner. Depending on the implementation - there are over a dozen Ruby interpreters, for example - they may offer "green threading" or true multithreading. Neither approach scales, due to reliance on locks. It is left to exotic languages like Erlang to do it right. You know how I wrote my first concurrent programs? Create threads with pthread_create(3), and pass ownership of objects by sending pointers over connections created with pipe(3). Which sounds an awful lot like this cure-all actor model (which I hadn't heard of yet) that requires languages almost nobody's ever heard of. Of all the approaches taken to multithreading, only one is known to work properly. That means, it scales to any number of cores, ... clone(2) and socket(7) ... avoids all locks,... Your threads/actors still block waiting for messages. I've seen an example of someone using this to write a mutex in Erlang. ... costs little more than conventional single-threaded programming,... Probably depends how much you have to twist your code to fit this functional+actor model. ... is easy to learn,... If you discount having to learn a whole new language and turn your code inside-out. ... and does not crash in strange ways. Instead of crashing, it hangs because you accidentally wrote a mutex.
  - toddh 16y ago
    I've built quite a few actor systems in C++ for real-time platforms. You use a threading library like pthreads, which works fine in C++, use thread safe libraries, create an Actor object with a queue that encapsulates a thread, and then have your async messaging system route and drop messages in the right actor queue. As long as application writers can manage their impulses to share state between the threads it works excellently. The only problem is usually the shared state inherent in the memory allocator, so some sort of thread based memory allocation scheme works best. You also get to choose your response latency via priority, which is something you don't get with Erlang.
    - defen 16y ago
      Keep in mind that the original point of erlang was "reliable software", not "high performance" (quotes because those are fuzzy terms) - there are many classes of errors that simply cannot happen in erlang that happen all the time in real-world C++ programming. You also get distributed computation for free via a well-tested VM.
nc 16y ago
ZeroMQ is fantastic for low latency communication between processes, but probably not the best choice for building 'actor' based systems. The slides don't really provide much basis for the argument. Erlang & Scala (for a more object-oriented feel) are both excellent for building actor based systems.
newobj 16y ago
Whaa?? "Concurrency is hard... Actors... ZeroMQ... End scene!" This is far from enlightening.
- phintjens 16y ago
  :-) If you hope for enlightenment from watching a 10-minute talk, you're going to be disappointed. I do like your summary of the presentation... "Concurrency is hard... Actors... ZeroMQ... End scene!". Nice. Enlightenment requires that you change in some way. For me, like many people who have taken the step of downloading and learning 0MQ, the feeling of "wow, this is too easy, where's the catch?" comes only after a few days writing code, and then your brain makes a slight adjustment, and it's all obvious, and that is a small but real enlightenment.
slavak 16y ago
What I got out of this presentation: "You thought the only way to do concurrency was by using synchronization primitives? Surprise! You can also do it by message-passing between processes!" What I think of this: Say whaaaaaaaaaaat? Message-passing as a concurrency model? That's crazy talk! Brilliant crazy talk! I wish this was such common knowledge that it was being taught in undergraduate "Parallel Programming" courses everywhere. Oh wait... IT TOTALLY IS!
- nikster 16y ago
  Pfft - I've seen way too many code bases with random threading code. Speaking of Java here - what's needed is that it's simply not possible to write new Thread(foo).start(); - the message passing model has to be enforced. I mean, read the Java documentation! What they say about threading is this: "Something slow? Just throw it in a background thread!". It's inane.
- dkarl 16y ago
  The carryover from computer science education to practice is so weak that you have to teach and persuade all over again. People leave school, get a job, and spend five years only using knowledge from their sophomore "Data Structures in C" class while struggling to catch up on all the critical stuff that their CS program taught them nothing about (SQL, source control, release management, etc.) Having misunderstood the purpose of undergraduate education, they believe this experience discredits everything they were taught in school, so they hold you to a higher threshold of proof if you're pitching ideas they heard in upper-level CS courses than if you're pitching ideas they've never heard before.
- KirinDave 16y ago
  Is that really true? I meet very few people, even fresh out of good schools, who have any practical knowledge of the Actor model. Many have heard of it, but most don't know that about even basic concepts surrounding it like Needham's Duality.
  - slavak 16y ago
    While my CDP class didn't explicitly discuss the Actor Model, message-passing as a concurrency model was studied and we did implement algorithms using it. How basic is the Actor Model for implementing concurrency using message-passing, besides the obvious advantage of formally defining a common language of terms and proven theorems for what you're doing?
tommi 16y ago
I wish Pieter had a bit less sensational title than "Multithreading Magic - Why Everything You Thought You Knew about Concurrency is Bogus, if not actually Totally Insane" considering that the slides contain only common knowledge.
antirez 16y ago
For what it is worth this is the model Redis, and Redis cluster, are beating on, and why we are not going multi threaded.
- lylejohnson 16y ago
  You meant "betting on", right? Not trying to be that guy, just wanted to make sure I understood your comment.
  - antirez 16y ago
    Yes, thanks and sorry for the error. Unfortunately it is no longer possible to edit the comment (but I agree with the edit timeout FWIW).
xtacy 16y ago
CSP (Communicating Sequential Processes) also seem similar: http://en.wikipedia.org/wiki/Communicating_sequential_processes http://en.wikipedia.org/wiki/Communicating_sequential_proces... There's a patch for Python that does CSP! http://code.google.com/p/python-csp/ http://code.google.com/p/python-csp/
- arethuza 16y ago
  Once upon a time there were even dedicated CPUs that had their own programming language based on CSP: http://en.wikipedia.org/wiki/Transputer http://en.wikipedia.org/wiki/Transputer http://en.wikipedia.org/wiki/Occam_%28programming_language%29 http://en.wikipedia.org/wiki/Occam_%28programming_language%2...
phintjens 16y ago
It's great to get feedback on the talk. Impossible to even introduce 0MQ properly in 10 minutes, so this presentation was really just to drive discussion. I'd apologise for the polemical title but that's just how I talk. To some of the comments here... - Parallel programming may be taught in CompSci courses, for sure, and it's a popular use for 0MQ, but (a) it's not applied to mainstream application development, and (b) it is not designed to scale to networks of any size. - The Actor model is not key to building a successful message-based concurrent application, but it's a good example, and helps people understand that there are alternatives to shared-state concurrency. - It's IMO useless to ask people to learn Erlang or Scala to get message based concurrency. People use Java, C, C#, PHP, Python, and probably still COBOL somewhere. So the challenge is how to give this mainstream a toolset that lets them build large parallel applications. Perhaps next year at FOSDEM we can do a devroom and take the time to see a real application evolve from a simple stand-alone process into a real distributed one. 0MQ is very hard to grasp as a theory, one needs to actually use it for a few days before the beauty a cheap, universal, fast, intelligent, easy to use, and asynchronous queuing messaging fabric hits home. Cheers!
- nuxi 16y ago
  I've glanced at the 0mq sources a bit, and I'm curious about the error handling. For example, zmq::tcp_socket_t::read() handles following errnos from recv(): EAGAIN, EWOULDBLOCK, EINTR, ECONNRESET and EPIPE. If recv() fails for any other reason (e.g. ENOBUFS, ETIMEDOUT) and returns -1, the following is performed: errno_assert (nbytes != -1); which translates to #define errno_assert(x) \ do {\ if (unlikely (!(x))) {\ perror (NULL);\ fprintf (stderr, "%s (%s:%d)\n", #x, __FILE__, __LINE__);\ abort ();\ }\ } while (false) The first thing I noticed is the use of perror() and fprintf() for error reporting. Any daemon that closed its stdin/out/err and not had them redirected to /dev/null will most likely output errors to other file descriptors (sockets, other open files etc.). Is this by design or something that was missed? Secondly, the use of abort() in an API is probably also not the best choice. Say I'm writing a (robust) database server and an error happens on one connection - the error will bring the whole server down, which kind of defeats the purpose. Is this also by design? [Edit: formatting]
vilda 16y ago
Message passing is not a golden bullet. It is a synchronization primitive for distributed systems. Basically on the same level as mutexes, semaphores, and monitors. Badly designed you are in the same concurrency hell.
- phintjens 16y ago
  Badly designed, anything can become hell. Crazy has no limits. The point of the talk was that message passing is fundamentally not at the same level as mutexes, semaphores, and monitors. Shared state does not scale linearly, no matter how well you use it, whereas message passing does, given cheap, fast, universal messaging.
  - HenryR 16y ago
    A message queue is shared state. It may be more convenient shared state, and since it only needs to be got right once it might be more bulletproof shared state, but you have multiple threads that are serialising on access to a queue no matter how it is implemented.
    - phintjens 16y ago
      You're right that a message queue is shared state and has to be gotten right but that can be done (and ZeroMQ does it) without locks. It's thus invisible to application developers. You're wrong to say that multiple threads serialize on a queue no matter how it's implemented. ZeroMQ lets threads write full speed to a queue while another thread reads full speed from the queue, without wait states.
farmer_ted 16y ago
Whenever I need an Actor, I go with Nathan Fillion.
xentronium 16y ago
Naming Magic: Why Every Other Link Including This One Is A Bait
philjackson 16y ago
...the bar man said "we don't serve event loops here" Node.js walks into a bar...
slavak 16y ago
Having read up a bit more on ZeroMQ, I have to ask: How is this, if at all, superior to, say, MPI[1]? [1]http://en.wikipedia.org/wiki/Message_Passing_Interface http://en.wikipedia.org/wiki/Message_Passing_Interface
- phintjens 16y ago
  Significantly simpler, faster, and more general. MPI is like one of the patterns ZeroMQ provides (the pipeline pattern) with a lot on top. That makes it great for parallel programming, but not so great for (e.g.) data distribution (pubsub) and other forms of message queuing.

66 points by adulau 16y ago

po 16y ago

I like ZeroMQ but this slideshare doesn't deliver any substance at all. For an overview, skip it and check out the white paper (if you can call it that) cowritten by the same guy (Pieter Hintjens) instead: http://www.zeromq.org/whitepapers:multithreading-magic http://www.zeromq.org/whitepapers:multithreading-magic

thijsterlouw 16y ago
These slides were part of a "lightning talk" on ZeroMQ (video here: http://streaming.fosdem.org:8000/ferrer/0mq.ogv http://streaming.fosdem.org:8000/ferrer/0mq.ogv ). So I think Pieter just wanted to make the audience curious and not dive into the details. I do agree that it might be better to use more slides for patterns and use cases and fewer generic/actor-slides. Also the argument that it's not taught in university was a bit weak.
- phintjens 16y ago
  There's a copy of the video here: http://www.youtube.com/watch?v=CCBYzKfmQ4U http://www.youtube.com/watch?v=CCBYzKfmQ4U The FOSDEM streaming server seems to be timing out right now.
nikster 16y ago
Dunno, made perfect sense to me. It's just that many (most?) programmers, in my own experience, don't fully appreciate the insanity that is traditional "shared data" concurrency. What I see is that they program something buggy, but where the bugs are rare enough to make the program work most times.
- jshen 16y ago
  For many types of problems, shared state concurrency is not the nightmare it's often made out to be. For example, if your problem can fit into the consumer producer model then it's really easy to make that work on the jvm today.
tbrownaw 16y ago
The most widely used languages, C and C++, do not offer any support for concurrency. Programmers roll their own by using threading APIs. Languages that do support concurrency, such as Java, Python, .NET, and Ruby, do it in a brute-force manner. Depending on the implementation - there are over a dozen Ruby interpreters, for example - they may offer "green threading" or true multithreading. Neither approach scales, due to reliance on locks. It is left to exotic languages like Erlang to do it right. You know how I wrote my first concurrent programs? Create threads with pthread_create(3), and pass ownership of objects by sending pointers over connections created with pipe(3). Which sounds an awful lot like this cure-all actor model (which I hadn't heard of yet) that requires languages almost nobody's ever heard of. Of all the approaches taken to multithreading, only one is known to work properly. That means, it scales to any number of cores, ... clone(2) and socket(7) ... avoids all locks,... Your threads/actors still block waiting for messages. I've seen an example of someone using this to write a mutex in Erlang. ... costs little more than conventional single-threaded programming,... Probably depends how much you have to twist your code to fit this functional+actor model. ... is easy to learn,... If you discount having to learn a whole new language and turn your code inside-out. ... and does not crash in strange ways. Instead of crashing, it hangs because you accidentally wrote a mutex.
- toddh 16y ago
  I've built quite a few actor systems in C++ for real-time platforms. You use a threading library like pthreads, which works fine in C++, use thread safe libraries, create an Actor object with a queue that encapsulates a thread, and then have your async messaging system route and drop messages in the right actor queue. As long as application writers can manage their impulses to share state between the threads it works excellently. The only problem is usually the shared state inherent in the memory allocator, so some sort of thread based memory allocation scheme works best. You also get to choose your response latency via priority, which is something you don't get with Erlang.
  - defen 16y ago
    Keep in mind that the original point of erlang was "reliable software", not "high performance" (quotes because those are fuzzy terms) - there are many classes of errors that simply cannot happen in erlang that happen all the time in real-world C++ programming. You also get distributed computation for free via a well-tested VM.

nc 16y ago

ZeroMQ is fantastic for low latency communication between processes, but probably not the best choice for building 'actor' based systems. The slides don't really provide much basis for the argument. Erlang & Scala (for a more object-oriented feel) are both excellent for building actor based systems.

newobj 16y ago

Whaa?? "Concurrency is hard... Actors... ZeroMQ... End scene!" This is far from enlightening.

phintjens 16y ago
:-) If you hope for enlightenment from watching a 10-minute talk, you're going to be disappointed. I do like your summary of the presentation... "Concurrency is hard... Actors... ZeroMQ... End scene!". Nice. Enlightenment requires that you change in some way. For me, like many people who have taken the step of downloading and learning 0MQ, the feeling of "wow, this is too easy, where's the catch?" comes only after a few days writing code, and then your brain makes a slight adjustment, and it's all obvious, and that is a small but real enlightenment.

slavak 16y ago

What I got out of this presentation: "You thought the only way to do concurrency was by using synchronization primitives? Surprise! You can also do it by message-passing between processes!" What I think of this: Say whaaaaaaaaaaat? Message-passing as a concurrency model? That's crazy talk! Brilliant crazy talk! I wish this was such common knowledge that it was being taught in undergraduate "Parallel Programming" courses everywhere. Oh wait... IT TOTALLY IS!

nikster 16y ago
Pfft - I've seen way too many code bases with random threading code. Speaking of Java here - what's needed is that it's simply not possible to write new Thread(foo).start(); - the message passing model has to be enforced. I mean, read the Java documentation! What they say about threading is this: "Something slow? Just throw it in a background thread!". It's inane.
dkarl 16y ago
The carryover from computer science education to practice is so weak that you have to teach and persuade all over again. People leave school, get a job, and spend five years only using knowledge from their sophomore "Data Structures in C" class while struggling to catch up on all the critical stuff that their CS program taught them nothing about (SQL, source control, release management, etc.) Having misunderstood the purpose of undergraduate education, they believe this experience discredits everything they were taught in school, so they hold you to a higher threshold of proof if you're pitching ideas they heard in upper-level CS courses than if you're pitching ideas they've never heard before.
KirinDave 16y ago
Is that really true? I meet very few people, even fresh out of good schools, who have any practical knowledge of the Actor model. Many have heard of it, but most don't know that about even basic concepts surrounding it like Needham's Duality.
- slavak 16y ago
  While my CDP class didn't explicitly discuss the Actor Model, message-passing as a concurrency model was studied and we did implement algorithms using it. How basic is the Actor Model for implementing concurrency using message-passing, besides the obvious advantage of formally defining a common language of terms and proven theorems for what you're doing?

tommi 16y ago

I wish Pieter had a bit less sensational title than "Multithreading Magic - Why Everything You Thought You Knew about Concurrency is Bogus, if not actually Totally Insane" considering that the slides contain only common knowledge.

antirez 16y ago

For what it is worth this is the model Redis, and Redis cluster, are beating on, and why we are not going multi threaded.

lylejohnson 16y ago
You meant "betting on", right? Not trying to be that guy, just wanted to make sure I understood your comment.
- antirez 16y ago
  Yes, thanks and sorry for the error. Unfortunately it is no longer possible to edit the comment (but I agree with the edit timeout FWIW).

xtacy 16y ago

CSP (Communicating Sequential Processes) also seem similar: http://en.wikipedia.org/wiki/Communicating_sequential_processes http://en.wikipedia.org/wiki/Communicating_sequential_proces... There's a patch for Python that does CSP! http://code.google.com/p/python-csp/ http://code.google.com/p/python-csp/

arethuza 16y ago
Once upon a time there were even dedicated CPUs that had their own programming language based on CSP: http://en.wikipedia.org/wiki/Transputer http://en.wikipedia.org/wiki/Transputer http://en.wikipedia.org/wiki/Occam_%28programming_language%29 http://en.wikipedia.org/wiki/Occam_%28programming_language%2...

phintjens 16y ago

It's great to get feedback on the talk. Impossible to even introduce 0MQ properly in 10 minutes, so this presentation was really just to drive discussion. I'd apologise for the polemical title but that's just how I talk. To some of the comments here... - Parallel programming may be taught in CompSci courses, for sure, and it's a popular use for 0MQ, but (a) it's not applied to mainstream application development, and (b) it is not designed to scale to networks of any size. - The Actor model is not key to building a successful message-based concurrent application, but it's a good example, and helps people understand that there are alternatives to shared-state concurrency. - It's IMO useless to ask people to learn Erlang or Scala to get message based concurrency. People use Java, C, C#, PHP, Python, and probably still COBOL somewhere. So the challenge is how to give this mainstream a toolset that lets them build large parallel applications. Perhaps next year at FOSDEM we can do a devroom and take the time to see a real application evolve from a simple stand-alone process into a real distributed one. 0MQ is very hard to grasp as a theory, one needs to actually use it for a few days before the beauty a cheap, universal, fast, intelligent, easy to use, and asynchronous queuing messaging fabric hits home. Cheers!

nuxi 16y ago
I've glanced at the 0mq sources a bit, and I'm curious about the error handling. For example, zmq::tcp_socket_t::read() handles following errnos from recv(): EAGAIN, EWOULDBLOCK, EINTR, ECONNRESET and EPIPE. If recv() fails for any other reason (e.g. ENOBUFS, ETIMEDOUT) and returns -1, the following is performed: errno_assert (nbytes != -1); which translates to #define errno_assert(x) \ do {\ if (unlikely (!(x))) {\ perror (NULL);\ fprintf (stderr, "%s (%s:%d)\n", #x, __FILE__, __LINE__);\ abort ();\ }\ } while (false) The first thing I noticed is the use of perror() and fprintf() for error reporting. Any daemon that closed its stdin/out/err and not had them redirected to /dev/null will most likely output errors to other file descriptors (sockets, other open files etc.). Is this by design or something that was missed? Secondly, the use of abort() in an API is probably also not the best choice. Say I'm writing a (robust) database server and an error happens on one connection - the error will bring the whole server down, which kind of defeats the purpose. Is this also by design? [Edit: formatting]

vilda 16y ago

Message passing is not a golden bullet. It is a synchronization primitive for distributed systems. Basically on the same level as mutexes, semaphores, and monitors. Badly designed you are in the same concurrency hell.

phintjens 16y ago
Badly designed, anything can become hell. Crazy has no limits. The point of the talk was that message passing is fundamentally not at the same level as mutexes, semaphores, and monitors. Shared state does not scale linearly, no matter how well you use it, whereas message passing does, given cheap, fast, universal messaging.
- HenryR 16y ago
  A message queue is shared state. It may be more convenient shared state, and since it only needs to be got right once it might be more bulletproof shared state, but you have multiple threads that are serialising on access to a queue no matter how it is implemented.
  - phintjens 16y ago
    You're right that a message queue is shared state and has to be gotten right but that can be done (and ZeroMQ does it) without locks. It's thus invisible to application developers. You're wrong to say that multiple threads serialize on a queue no matter how it's implemented. ZeroMQ lets threads write full speed to a queue while another thread reads full speed from the queue, without wait states.

farmer_ted 16y ago

Whenever I need an Actor, I go with Nathan Fillion.

xentronium 16y ago

Naming Magic: Why Every Other Link Including This One Is A Bait

philjackson 16y ago

...the bar man said "we don't serve event loops here" Node.js walks into a bar...

slavak 16y ago

Having read up a bit more on ZeroMQ, I have to ask: How is this, if at all, superior to, say, MPI[1]? [1]http://en.wikipedia.org/wiki/Message_Passing_Interface http://en.wikipedia.org/wiki/Message_Passing_Interface

phintjens 16y ago
Significantly simpler, faster, and more general. MPI is like one of the patterns ZeroMQ provides (the pipeline pattern) with a lot on top. That makes it great for parallel programming, but not so great for (e.g.) data distribution (pubsub) and other forms of message queuing.