8 ms·

Understanding Real-World Concurrency Bugs in Go [pdf]

165 points by oldgun 8y ago

jstewartmobile 8y ago
The terminology could use some refinement. Not sure "message passing" is the right phrase to describe Go channels. Most encounters I've had with "message passing"--whether it was AJAX, Win32 PostMessage, or grade school--have been non-blocking. Go channels block. edit: Here's a pretty good write-up from 2016 on why channels are an anti-pattern: https://www.jtolio.com/2016/03/go-channels-are-bad-and-you-should-feel-bad/ https://www.jtolio.com/2016/03/go-channels-are-bad-and-you-s...
- zzzcpan 8y ago
  It's fine. It's still message passing and it's obvious they refer to Go flavored CSP style message passing whenever they talk about it, not anything else.
  - eternalban 8y ago
    (I've been using Go since it was released and like the language.) Go tried CSP in the beginning but in CSP the 'sequential process' preempts only on blocking (IO) or until finished. Also message passing is just that. You can't reach out and modify a message after you have posted it, IRL. This means messages are 'safe' to use concurrently since they are copies. That, in conjuction with strict CSP ssemantics (processes always run to completion with no preemptive cooperation) gets you 'safe concurrency' at the price of performance and at scale 'reasoning' about what is happening. So, the practical decisions were (a) honors system message passing via references, (b) preemption at IO and elsewhere, and (c) introduction of mutex, etc. to fall back to [more performant] 'light weight threads' and 'locks' paradigm.
- api 8y ago
  I don't see why blocking is an issue given that goroutines are light weight threads. They are really cheap and can be created as needed, eliminating much of the need for async patterns. It's one of the nicest things about Go given that async is just manually implemented lightweight threading. I would say though that go channels have not turned out to be as useful as I thought they would be. I dont find myself using them that much. But maybe it's just my style or the type of stuff I am writing.
  - ec109685 8y ago
    Lightweight threads are different than async models. With the latter work continues until io or a manual yield happens. With threads, your workload can be preempted outside your control.
  - jstewartmobile 8y ago
    Blocking is a pretty big deal. With something like pipes or sockets, there are timeouts and saner handling of breakages. With Go channels, one false move, and you've got a panic or an infinite wait. I like Go, just not a big fan of channels.
  - pcwalton 8y ago
    > It's one of the nicest things about Go given that async is just manually implemented lightweight threading. No, async/await is typically stackless, while goroutines have stacks. That's a significant difference.
  - littlestymaar 8y ago
    Blocking implies you can get deadlocks if you're not cautious, whereas with async you just can't.
zzzcpan 8y ago
This is pretty good and deserves more attention. Github makes bug studies so much more approachable. Hopefully the study will make people more cautious about Go's concurrency model, which is as error prone as shared memory multithreading, if not more.
- weberc2 8y ago
  What concurrency model do you prefer? Seems like it’s easy enough to opt into a share-nothing model in Go, and this is generally what I do unless I have performance concerns (and as a consequence, I very rarely see concurrency bugs). I’m also not sure how much static analysis a la Rust could help the problem without accepting Rust levels of productivity.
  - ricketycricket 8y ago
    I wouldn't say I'm an expert in this area, but I do find Erlang/Elixir's actor model quite compelling.
    - tormeh 8y ago
      The big advantage of actors are that they are compatible with network loss. Erlang and Akka actors are network transparent. Local-only actors are missing the point, IMO.
      - weberc2 8y ago
        I've not used an actor model--is there some router component that resolves the process to which a message needs to be sent? Is every send() operation a tuple of `(address, message)`? If so, presumably the router component decides whether there is a local actor or whether it needs to go out over the network?
        macintux 8y ago
        Erlang runs atop its own VM that includes scheduler, routing services, etc. There are (at least) a couple of different ways to identify the recipient of a message: process ID (unique identifier for another actor) or Erlang's name service. The VM does indeed know whether the recipient is local; the sender typically neither knows nor cares, although the information is available if useful.
tapirl 8y ago
is there a html version?
c61 8y ago
The two key takeaways for me are: (1) "Contrary to the common belief that message passing is less error-prone, more blocking bugs in our studied Go applications are caused by wrong message passing than by wrong shared memory protection." (2) "Shared memory synchronization operations are used more often than message passing, ..." So basically, (1) people have more trouble writing correct multithreaded routines using message passing, and (2) in large production applications people tend to fall back to using shared-memory primitives rather than message-passing anyway. It seems the intention of the Go language designers was to make a language that would be simple and easy for programmers to pick up. Given that most programmers are already accustomed to multithreaded programming using shared memory and that this is conceptually simpler, I think the language designers made a mistake by throwing channels, a relatively new and experimental approach (yes, I know CSP is from 1978, but I'm talking about widely-adopted industrial languages), into the language. I think it was the single biggest mistake in the design of Go.
- tedunangst 8y ago
  Channels are appealing, but they're trickier than they seem. Especially when you're selecting on two channels waiting for one event to happen first but then both happen at the same time. Eventually you end up with special code in both cases to check whether the othe thing happened and try to stop it, and then what if you're late, ... The code is easy to write, but hard to make correct.
- deleted 8y ago
  [deleted]
- naikrovek 8y ago
  > most programmers are already accustomed to multithreaded programming using shared memory Aren't most programmers JavaScript programmers accustomed to single threaded applications passing variables around? I've been programming for 20 years, as a career, and I've never once used shared memory in the way that I think when I think "shared memory". Maybe some languages I've used do things internally with shared memory, I don't know. My point here is to be aware that the people you work with and know aren't necessarily representative of "most programmers" even though it feels that way to each of us. We expose ourselves to what we know more than what we don't know, and that influences how we each see the world.
crimsonalucard 8y ago
"Surprisingly, our study shows that it is as easy to make concurrency bugs with message passing as with shared memory, sometimes even more. For example, around 58% of blocking bugs are caused by message passing. In addition to the violation of Go’s channel usage rules (e.g., waiting on a channel that no one sends data to or close), many concurrency bugs are caused by the mixed usage of message passing and other new semantics and new libraries in Go, which can easily be overlooked but hard to detect."
- jstewartmobile 8y ago
  On the one hand, they're absolutely right. On the other hand, statistical methods don't give quality it's due. I'd rather fight a blocking bug than a data-updated-via-race-condition bug.
crimsonalucard 8y ago
As much as I don't like javascript, there is a whole swathe of concurrency bugs that exists in golang and not in nodejs.
- Thaxll 8y ago
  Makes sense since Nodejs has no parallelism features.
  - beatgammit 8y ago
    And that's a large reason why I use Go: scaling to multiple cores works out of the box if you build it with goroutines.
  - pcwalton 8y ago
    Go doesn't really have parallelism features either. A language designed for parallelism needs to have data-parallel features like parallel iterators, good support for SIMD, and concurrent data structures, neither of which Go has. (In fact, due to lack of generics, you can't even build your own generic concurrent data structures!)
    - Thaxll 8y ago
      Your comment makes no sense, goroutines are multiplexed on each cores thus it has parallelism. https://en.wikipedia.org/wiki/Parallel_computing https://en.wikipedia.org/wiki/Parallel_computing "Parallel computing is a type of computation in which many calculations or the execution of processes are carried out simultaneously." Do you enjoy trolling and dismissing Go in general? Reading your comments sounds like some envy toward Go popularity. When you're not complaining about concurrency features it's about Go GC.
      - pcwalton 8y ago
        And Node has workers and can thus achieve parallelism as well. What matters isn't whether the language can achieve parallelism: it's whether the language is designed for it. Go is designed for concurrency, not parallelism. It's Rob Pike's view [1] that concurrency enables parallelism "for free". I think that's oversimplified and at odds with the way people who write CPU-bound code actually use parallelism features, but that's how Go was designed: it omits traditional parallelism features in favor of concurrency features. [1]: https://blog.golang.org/concurrency-is-not-parallelism https://blog.golang.org/concurrency-is-not-parallelism
mirceal 8y ago
confirms some of my suspicions to claims on how sexy and easy go is when it comes to concurrency. if anything it will give you a false a sense of security but god help you in case things go south and you’ve abused these features.
- mirceal 8y ago
  reminded me of: https://twitter.com/pasiphae_goals/status/923835427995508741?s=20 https://twitter.com/pasiphae_goals/status/923835427995508741...
Animats 8y ago
Well, yes. Go's shared memory model is essentially the same as that of C and C++. You have locks, and you have data objects, and the language has no idea which lock covers what data. That's the big thing Rust got right. Go's message passing model is somewhat error prone. It lets you pass references. So you can create shared data through the channels. It's easy to do this accidentally by passing slices, which are references to the underlying array, across a channel. More generally, Go channels are one-way. Since you often want an answer back, you have to create some two-way mechanism from the raw one-way channels. It seems to be common to do this at the raw channel level, rather than using some "object" like encapsulation. Creating a mechanism which shuts down cleanly is somewhat tricky, too.

165 points by oldgun 8y ago

The terminology could use some refinement. Not sure "message passing" is the right phrase to describe Go channels. Most encounters I've had with "message passing"--whether it was AJAX, Win32 PostMessage, or grade school--have been non-blocking. Go channels block. edit: Here's a pretty good write-up from 2016 on why channels are an anti-pattern: https://www.jtolio.com/2016/03/go-channels-are-bad-and-you-should-feel-bad/ https://www.jtolio.com/2016/03/go-channels-are-bad-and-you-s...

zzzcpan 8y ago
It's fine. It's still message passing and it's obvious they refer to Go flavored CSP style message passing whenever they talk about it, not anything else.
- eternalban 8y ago
  (I've been using Go since it was released and like the language.) Go tried CSP in the beginning but in CSP the 'sequential process' preempts only on blocking (IO) or until finished. Also message passing is just that. You can't reach out and modify a message after you have posted it, IRL. This means messages are 'safe' to use concurrently since they are copies. That, in conjuction with strict CSP ssemantics (processes always run to completion with no preemptive cooperation) gets you 'safe concurrency' at the price of performance and at scale 'reasoning' about what is happening. So, the practical decisions were (a) honors system message passing via references, (b) preemption at IO and elsewhere, and (c) introduction of mutex, etc. to fall back to [more performant] 'light weight threads' and 'locks' paradigm.
api 8y ago
I don't see why blocking is an issue given that goroutines are light weight threads. They are really cheap and can be created as needed, eliminating much of the need for async patterns. It's one of the nicest things about Go given that async is just manually implemented lightweight threading. I would say though that go channels have not turned out to be as useful as I thought they would be. I dont find myself using them that much. But maybe it's just my style or the type of stuff I am writing.
- ec109685 8y ago
  Lightweight threads are different than async models. With the latter work continues until io or a manual yield happens. With threads, your workload can be preempted outside your control.
- jstewartmobile 8y ago
  Blocking is a pretty big deal. With something like pipes or sockets, there are timeouts and saner handling of breakages. With Go channels, one false move, and you've got a panic or an infinite wait. I like Go, just not a big fan of channels.
- pcwalton 8y ago
  > It's one of the nicest things about Go given that async is just manually implemented lightweight threading. No, async/await is typically stackless, while goroutines have stacks. That's a significant difference.
- littlestymaar 8y ago
  Blocking implies you can get deadlocks if you're not cautious, whereas with async you just can't.

zzzcpan 8y ago

This is pretty good and deserves more attention. Github makes bug studies so much more approachable. Hopefully the study will make people more cautious about Go's concurrency model, which is as error prone as shared memory multithreading, if not more.

weberc2 8y ago
What concurrency model do you prefer? Seems like it’s easy enough to opt into a share-nothing model in Go, and this is generally what I do unless I have performance concerns (and as a consequence, I very rarely see concurrency bugs). I’m also not sure how much static analysis a la Rust could help the problem without accepting Rust levels of productivity.
- ricketycricket 8y ago
  I wouldn't say I'm an expert in this area, but I do find Erlang/Elixir's actor model quite compelling.
  - tormeh 8y ago
    The big advantage of actors are that they are compatible with network loss. Erlang and Akka actors are network transparent. Local-only actors are missing the point, IMO.
    - weberc2 8y ago
      I've not used an actor model--is there some router component that resolves the process to which a message needs to be sent? Is every send() operation a tuple of `(address, message)`? If so, presumably the router component decides whether there is a local actor or whether it needs to go out over the network?
      - macintux 8y ago
        Erlang runs atop its own VM that includes scheduler, routing services, etc. There are (at least) a couple of different ways to identify the recipient of a message: process ID (unique identifier for another actor) or Erlang's name service. The VM does indeed know whether the recipient is local; the sender typically neither knows nor cares, although the information is available if useful.

tapirl 8y ago

is there a html version?

c61 8y ago

The two key takeaways for me are: (1) "Contrary to the common belief that message passing is less error-prone, more blocking bugs in our studied Go applications are caused by wrong message passing than by wrong shared memory protection." (2) "Shared memory synchronization operations are used more often than message passing, ..." So basically, (1) people have more trouble writing correct multithreaded routines using message passing, and (2) in large production applications people tend to fall back to using shared-memory primitives rather than message-passing anyway. It seems the intention of the Go language designers was to make a language that would be simple and easy for programmers to pick up. Given that most programmers are already accustomed to multithreaded programming using shared memory and that this is conceptually simpler, I think the language designers made a mistake by throwing channels, a relatively new and experimental approach (yes, I know CSP is from 1978, but I'm talking about widely-adopted industrial languages), into the language. I think it was the single biggest mistake in the design of Go.

tedunangst 8y ago
Channels are appealing, but they're trickier than they seem. Especially when you're selecting on two channels waiting for one event to happen first but then both happen at the same time. Eventually you end up with special code in both cases to check whether the othe thing happened and try to stop it, and then what if you're late, ... The code is easy to write, but hard to make correct.
deleted 8y ago
[deleted]
naikrovek 8y ago
> most programmers are already accustomed to multithreaded programming using shared memory Aren't most programmers JavaScript programmers accustomed to single threaded applications passing variables around? I've been programming for 20 years, as a career, and I've never once used shared memory in the way that I think when I think "shared memory". Maybe some languages I've used do things internally with shared memory, I don't know. My point here is to be aware that the people you work with and know aren't necessarily representative of "most programmers" even though it feels that way to each of us. We expose ourselves to what we know more than what we don't know, and that influences how we each see the world.

crimsonalucard 8y ago

"Surprisingly, our study shows that it is as easy to make concurrency bugs with message passing as with shared memory, sometimes even more. For example, around 58% of blocking bugs are caused by message passing. In addition to the violation of Go’s channel usage rules (e.g., waiting on a channel that no one sends data to or close), many concurrency bugs are caused by the mixed usage of message passing and other new semantics and new libraries in Go, which can easily be overlooked but hard to detect."

jstewartmobile 8y ago
On the one hand, they're absolutely right. On the other hand, statistical methods don't give quality it's due. I'd rather fight a blocking bug than a data-updated-via-race-condition bug.

crimsonalucard 8y ago

As much as I don't like javascript, there is a whole swathe of concurrency bugs that exists in golang and not in nodejs.

Thaxll 8y ago
Makes sense since Nodejs has no parallelism features.
- beatgammit 8y ago
  And that's a large reason why I use Go: scaling to multiple cores works out of the box if you build it with goroutines.
- pcwalton 8y ago
  Go doesn't really have parallelism features either. A language designed for parallelism needs to have data-parallel features like parallel iterators, good support for SIMD, and concurrent data structures, neither of which Go has. (In fact, due to lack of generics, you can't even build your own generic concurrent data structures!)
  - Thaxll 8y ago
    Your comment makes no sense, goroutines are multiplexed on each cores thus it has parallelism. https://en.wikipedia.org/wiki/Parallel_computing https://en.wikipedia.org/wiki/Parallel_computing "Parallel computing is a type of computation in which many calculations or the execution of processes are carried out simultaneously." Do you enjoy trolling and dismissing Go in general? Reading your comments sounds like some envy toward Go popularity. When you're not complaining about concurrency features it's about Go GC.
    - pcwalton 8y ago
      And Node has workers and can thus achieve parallelism as well. What matters isn't whether the language can achieve parallelism: it's whether the language is designed for it. Go is designed for concurrency, not parallelism. It's Rob Pike's view [1] that concurrency enables parallelism "for free". I think that's oversimplified and at odds with the way people who write CPU-bound code actually use parallelism features, but that's how Go was designed: it omits traditional parallelism features in favor of concurrency features. [1]: https://blog.golang.org/concurrency-is-not-parallelism https://blog.golang.org/concurrency-is-not-parallelism

mirceal 8y ago

confirms some of my suspicions to claims on how sexy and easy go is when it comes to concurrency. if anything it will give you a false a sense of security but god help you in case things go south and you’ve abused these features.

mirceal 8y ago
reminded me of: https://twitter.com/pasiphae_goals/status/923835427995508741?s=20 https://twitter.com/pasiphae_goals/status/923835427995508741...

Animats 8y ago

Well, yes. Go's shared memory model is essentially the same as that of C and C++. You have locks, and you have data objects, and the language has no idea which lock covers what data. That's the big thing Rust got right. Go's message passing model is somewhat error prone. It lets you pass references. So you can create shared data through the channels. It's easy to do this accidentally by passing slices, which are references to the underlying array, across a channel. More generally, Go channels are one-way. Since you often want an answer back, you have to create some two-way mechanism from the raw one-way channels. It seems to be common to do this at the raw channel level, rather than using some "object" like encapsulation. Creating a mechanism which shuts down cleanly is somewhat tricky, too.