4 ms·

It wasn't really tricky concurrency. Somebody just made the mistake of sharing a pointer across goroutines. It was quite indirect. Boils down to a function take

It wasn't really tricky concurrency. Somebody just made the mistake of sharing a pointer across goroutines. It was quite indirect. Boils down to a function takeing a param and holds onto it. `go` is used at some point closing over this pointer. And now we have a data race in the waiting.

jchw 1y ago
Aside from the type system bypass described in the article though, this is basically no different from the status quo for virtually all languages with free threading that aren't Rust. I argue that while everyone is fallible, experienced programmers usually don't make this sort of mistake directly, because they are usually well aware that sharing pointers over a closure on another thread is a recipe for disaster. Instead, I think that most of these issues are actually involving tricky circumstances that result in this happening by accident, like accidentally re-using an err variable from the enclosing function of a closure. These sorts of bugs are really bad, because they happen in spite of everyone knowing not to inappropriately share data across goroutines, and they are easy to sneak by in code reviews unnoticed. They don't intuitively look like concurrency bugs, and sometimes they're as little as one single : away from being correct. (Which I agree is a bad place for a language design to be.) Thankfully though, people don't just throw their hands up there; a good amount of work has gone into figuring out the kinds of mistakes that often lead to Go concurrency bugs in the real world and writing static analysis tools that can help prevent them. That work, combined with Go's builtin tools and standard library, and the memory safety of individual isolated goroutines, makes most production Go concurrency bugs fairly boring even compared to C concurrency bugs, even though they theoretically have the same basic problem where you can freely share mutable data unsafely across concurrent threads. So yes, it is still possible to write trivial, obvious concurrency bugs. The language won't stop you. However I've used Go across almost every job I've had since like 2016 and it has been rare to come across a concurrency bug this trivial. I hope I would catch flagrantly shared mutable state across threads during code review.
- bombela 1y ago
  Yes experienced programmers won't make the obvious mistake... until they do because the distance from the source of the bug to it's manifestation is too great to notice until it fails in production.
  - jchw 1y ago
    Look, this is pointless. I'm not learning anything new when you tell me that it can and will happen. How will it happen and how much will it happen? Hence linking to Uber's case study on the issue. The answer? Not that much. Uber started performing race detection in production over a 6 month period and found 2,000 different race conditions. Ouch, that sounds horrible! But wait, we're talking about 50 million lines of Go code and 2,100 services at the time of that writing. That means they were seeing approximately 1 race condition per 25,000 lines of code and about 1 race condition per service. That actually lines up pretty well with my experiences. Although I haven't had a production outage or serious correctness issue caused by a race condition in Go, I have seen probably about one or two race conditions that made it to production per service. I reckon those codebases were likely somewhere between 10,000 and 25,000 lines of code most likely, so not so far off of the scale. But again it doesn't always lead to a serious production outage, it's just that simple. It could be worse too (could corrupt some data and pollute your production database or something, in the worst case) but usually it's better (wonky behavior but no long-term effects, maybe the service periodically crashes but restarts, leading to some dropped requests but no long term downtime.) Uber has no doubt seen at least some Go data races that have caused actual production outages, but they've seen at least 2,000 Go data races that haven't, otherwise they would've probably been caught before the race detector caught them, Go dumps stacktraces on crash. That has to tell you something about the actual probability of causing a production outage due to a data race. Again, you do you, but I will not be losing sleep over this. It is something to be weary of when working on Go services, but it is manageable.
    - zozbot234 1y ago
      Identifiable "wonky" behavior and periodic crashes seem like a very real issue to me. This wouldn't fly for any mission-critical service, it's something that demands a root cause analysis. Especially since it's hard to be sure after the fact that no data has been corrupted somehow or that security invariants have not been violated due to the "wonky" behavior.

jchw 1y ago

Aside from the type system bypass described in the article though, this is basically no different from the status quo for virtually all languages with free threading that aren't Rust. I argue that while everyone is fallible, experienced programmers usually don't make this sort of mistake directly, because they are usually well aware that sharing pointers over a closure on another thread is a recipe for disaster. Instead, I think that most of these issues are actually involving tricky circumstances that result in this happening by accident, like accidentally re-using an err variable from the enclosing function of a closure. These sorts of bugs are really bad, because they happen in spite of everyone knowing not to inappropriately share data across goroutines, and they are easy to sneak by in code reviews unnoticed. They don't intuitively look like concurrency bugs, and sometimes they're as little as one single : away from being correct. (Which I agree is a bad place for a language design to be.) Thankfully though, people don't just throw their hands up there; a good amount of work has gone into figuring out the kinds of mistakes that often lead to Go concurrency bugs in the real world and writing static analysis tools that can help prevent them. That work, combined with Go's builtin tools and standard library, and the memory safety of individual isolated goroutines, makes most production Go concurrency bugs fairly boring even compared to C concurrency bugs, even though they theoretically have the same basic problem where you can freely share mutable data unsafely across concurrent threads. So yes, it is still possible to write trivial, obvious concurrency bugs. The language won't stop you. However I've used Go across almost every job I've had since like 2016 and it has been rare to come across a concurrency bug this trivial. I hope I would catch flagrantly shared mutable state across threads during code review.

bombela 1y ago
Yes experienced programmers won't make the obvious mistake... until they do because the distance from the source of the bug to it's manifestation is too great to notice until it fails in production.
- jchw 1y ago
  Look, this is pointless. I'm not learning anything new when you tell me that it can and will happen. How will it happen and how much will it happen? Hence linking to Uber's case study on the issue. The answer? Not that much. Uber started performing race detection in production over a 6 month period and found 2,000 different race conditions. Ouch, that sounds horrible! But wait, we're talking about 50 million lines of Go code and 2,100 services at the time of that writing. That means they were seeing approximately 1 race condition per 25,000 lines of code and about 1 race condition per service. That actually lines up pretty well with my experiences. Although I haven't had a production outage or serious correctness issue caused by a race condition in Go, I have seen probably about one or two race conditions that made it to production per service. I reckon those codebases were likely somewhere between 10,000 and 25,000 lines of code most likely, so not so far off of the scale. But again it doesn't always lead to a serious production outage, it's just that simple. It could be worse too (could corrupt some data and pollute your production database or something, in the worst case) but usually it's better (wonky behavior but no long-term effects, maybe the service periodically crashes but restarts, leading to some dropped requests but no long term downtime.) Uber has no doubt seen at least some Go data races that have caused actual production outages, but they've seen at least 2,000 Go data races that haven't, otherwise they would've probably been caught before the race detector caught them, Go dumps stacktraces on crash. That has to tell you something about the actual probability of causing a production outage due to a data race. Again, you do you, but I will not be losing sleep over this. It is something to be weary of when working on Go services, but it is manageable.
  - zozbot234 1y ago
    Identifiable "wonky" behavior and periodic crashes seem like a very real issue to me. This wouldn't fly for any mission-critical service, it's something that demands a root cause analysis. Especially since it's hard to be sure after the fact that no data has been corrupted somehow or that security invariants have not been violated due to the "wonky" behavior.