19 ms·

Data Race Patterns in Go

307 points by statenjason 4y ago

smw1218 4y ago
I wrote a deep analysis/reaction to this post: https://medium.com/@scott_white/concurrency-in-go-is-not-magic-37bb16af4b1a https://medium.com/@scott_white/concurrency-in-go-is-not-mag... tl;dr Go doesn't magically solve data races and blaming the language itself isn't well supported by the examples/data.
thallavajhula 4y ago
>We developed a system to detect data races at Uber using a dynamic data race detection technique. By system do you mean a process or a tool that detects these?
bumper_crop 4y ago
Sorry to say, but these hit close to home for me. A lot of the synchronization paradigms in Go are easy to misuse, but lead the author into thinking it's okay. the WaitGroup one is particularly poignant for me, since the race detector doesn't catch it. I'll add one other data race goof: atomic.Value. Look at the implementation. Unlike pretty much every other language I've seen, atomic.Value isn't really atomic, since the concrete type can't ever change after being set. This stems from fact that interfaces are two words rather than one, and they can't be (hardware) atomically set. To fix it, Go just documents "hey, don't do that", and then panics if you do.
- morelisp 4y ago
  > atomic.Value isn't really atomic, since the concrete type can't ever change after being set. How does this mean it's non-atomic? As far as I know you can still never Load() a partial Store(). (Also, even if it was possible, this would never be a good idea...)
  - bumper_crop 4y ago
    That's why I opened with "Look at the implementation". Go is unable to store the type and the pointer at the same time, so it warps what "atomic" means. Pretty much every other language has atomic mean "one of these will win, one will lose". Go says "one will win, one will panic and destroy the goroutine. In fact, it's even worse than that. If the Store() caller goes to sleep between setting the type and storing the pointer, it causes every Goroutine that calls Load() to block. They can't make forward progress if the store caller hangs.
    - fastest963 4y ago
      This is why all the examples call Store immediately with a zero value of the type.
      - bumper_crop 4y ago
        https://go.dev/play/p/xolc9oPwA0C https://go.dev/play/p/xolc9oPwA0C Interfaces don't have a zero type, which means that we can't have an atomic.Value which stores Shape. Atomic Value would be much easier to reason about if it had store semantics similar to a regular `var foo Shape = ...`. One of the other comment threads talked about generics helping this, so maybe there is hope.
        deleted 4y ago
        [deleted]
        yencabulator 4y ago
        Parent means var bestShape atomic.Value bestShape.Store((*Circle)(nil))
        morelisp 4y ago
        Which will store it as a *Circle, and only allow more *Circles, not Shapes. That part of GP’s claim is correct. It just had nothing to do with atomicity; it means something specific, not just “I like the failure mode.”
- Groxx 4y ago
  The lack of generics has forced all Go concurrency to be intrusive (i.e. implemented by the person using literally any concurrency), and yeah. It's horrifyingly error-prone in my experience. It means everyone needs to be an expert, and lol, everyone is not an expert. Generics might save us from the simple, mechanical flaws. Expect to see `Locker<T>` and `Atomic<T>` types cropping up. And unbounded buffered thread-safe queues backing channels. Etc. I'm very, very much looking forward to it. --- edited to rant more --- I also really wonder where all these "go makes concurrency a first-class concept" claims come from, because I see it quite a few places, and I feel like it's making some very strong implied claims that absolutely do not exist. Go has channels and select. That's neat. But on the other hand it has threads... but no thread handles. It has implicit capturing of closures. It has ambiguous value vs pointer semantics. It (style- and ergonomic-wise) encourages field references, which have no way to enforce mutexes or atomics. It has had crippled lock APIs that effectively force use of channels for... I don't know, philosophical reasons? Go is abnormally dangerous when it comes to concurrency IMO. The race detector does an amazing job helping you discover it, but it's very easy to not use it or not take full advantage of it (i.e. non-parallel tests), and few run their production services with the race detector enabled. Because if they did, it would crash all the time, because there are an absurd amount of races in nearly all of the popular libraries (and in common use of those libraries, because concurrency is not a first-class citizen and you can't tell when it's happening / when it shouldn't happen).
  - hexxagone 4y ago
    Go does not have threads but something like "tasks". The fact that no thread handle is exposed allows for transparently moving these tasks across threads if the scheduler decides so. "go makes concurrency a first-class concept" I think it usually refers to goroutines being built in the language. "Go is abnormally dangerous when it comes to concurrency IMO". Personnally, it has not been my experience with Go concurrency. However I have hit some issues when trying to ocrhestrate tasks via channels and ended up resorting to atomics to do the job.
    - saghm 4y ago
      > Go does not have threads but something like "tasks". The fact that no thread handle is exposed allows for transparently moving these tasks across threads if the scheduler decides so. This doesn't stop there being "task handles" then, though? I think the point GP was making is that something that in most languages would be simple methods on a handle like "wait for this task to finish" or "stop this task" instead need to be done manually in Go with channels (or potentially `Context` in the latter case, although that was a later addition to the standard library). It doesn't really matter whether you call it a thread or a task; either way, it would be nice to get some return value from spawning some background operation and being able to use it to directly interact with it. I agree with GP that it does seem like an odd omission, since I haven't really heard any actual practical explanation for it.
- avgcorrection 4y ago
  This is Atomic* * Just don’t be an idiot. Worse is better.
- nindalf 4y ago
  On atomics in Go, the beta for Go 1.19 was released an hour ago (https://groups.google.com/g/golang-announce/c/SNruPJUSFz0?pli=1 https://groups.google.com/g/golang-announce/c/SNruPJUSFz0?pl...). > The sync/atomic package defines new atomic types Bool, Int32, Int64, Uint32, Uint64, Uintptr, and Pointer. These types hide the underlying values so that all accesses are forced to use the atomic APIs. Pointer also avoids the need to convert to unsafe.Pointer at call sites. Int64 and Uint64 are automatically aligned to 64-bit boundaries in structs and allocated data, even on 32-bit systems. Go 1.19 is expected to release in August.
travisd 4y ago
Worth noting that some of these can be detected statically -- and some are detected by go vet (e.g., passing a sync.Mutex by value). I don't think it detects the wg.Add bug, but that seems relatively straightforward(†) to add a check for. (†famous last words, I know)
- kjksf 4y ago
  staticcheck has a check for wg.Add misuse (https://staticcheck.io/docs/checks https://staticcheck.io/docs/checks, https://staticcheck.io/docs/checks#SA2000 https://staticcheck.io/docs/checks#SA2000)
deleted 4y ago
[deleted]
JamesSwift 4y ago
Really good article, and gave me several ideas to track down some gremlins that have been bugging me in a go codebase recently.
jchw 4y ago
This is pretty cool. 50 million lines of code is quite a large corpus to work off of. I'm surprised by some of them. For example, go vet nominally catches misuses of mutexes, so it's surprising that even a few of those slipped through. I wonder if those situations are a bit more complicated than the example. Obviously, the ideal outcome is that static analysis can help eliminate as many issues as possible, by restricting the language, discouraging bad patterns, or giving the programmer more tools to detect bugs. gVisor, for example, has a really interesting tool called checklocks: https://github.com/google/gvisor/tree/master/tools/checklocks https://github.com/google/gvisor/tree/master/tools/checklock... While it definitely has some caveats, ideas like these should help Go programs achieve a greater degree of robustness. Obviously, this class of error would be effectively prevented by borrow checking, but I suppose if you want programming language tradeoffs more tilted towards robustness, Rust already has a lot of that covered.
- likeabbas 4y ago
  > I suppose if you want programming language tradeoffs more tilted towards robustness, Rust already has a lot of that covered Does anyone not want robustness of their language to cover their mistakes?
  - setr 4y ago
    For free? Of course not. At cost? … depends on what you’re charging me, and how much I’m getting
    - likeabbas 4y ago
      Good point. Right now I kind of see modern programming as two fold: 1) Loosely typed to get you what you want faster, but with some mistakes, and 2) Strongly typed that forces you to try harder, but ultimately better I'm usually happier with the latter. I find I become far more frustrated when I try to write python than I do something like Rust just because I know when I write Python that I will have mistakes I'll have to fix in prod, vs when I write Rust I won't have those mistakes (although it'll take me longer to get something to prod)
metadat 4y ago
> 2. Slices are confusing types that create subtle and hard-to-diagnose data races The "Slices" example is just nasty! Like, this is just damning for Go's promise of "_relatively_ easy and carefree concurrency". Think about it for a second or two, >> The reference to the slice was resized in the middle of an append operation from another async routine. What exactly happens in these cases? How can I trust myself, as a fallible human being, to reason about such cases when I'm trying to efficiently roll up a list of results. :-/ Compared to every other remotely mainstream language, perhaps even C++, these are extremely subtle and sharp.. nigh, razor sharp edges. Yuck. One big takeaway is this harsh realization: Golang guarantees are scant more than what is offered by pure, relatively naïve and unadulterated BASH shell programming. I still will use it, but with newfound fear. As a multi-hundred-kloc-authoring-gopher: I love Go, and this article is killing me inside. Go appears extremely sloppy at the edges of the envelope and language boundaries, moreso than even I had ever realized prior to now. Full-disclosure: I am disgusted by the company that is Uber, but I'm grateful to the talented folks who've cast a light on this cesspool region of Golang. Thank you! p.s. inane aside: I never would've guessed that in 2022, Java would start looking more and more appealing in new ways. Until now I've been more or less "all-in" on Go for years.
- jchw 4y ago
  Slices may just be one of the best and worst parts of Go. They're cumbersome, their behavior sometimes feels 'inexplicable,' and even as an experienced developer you are likely to eventually fallen into one of the traps where your 'obvious' code isn't so obvious. That said... when programming in programming languages without a slice type, I always want to have one. And though it's confusing at times, the design does actually make sense; without a doubt, it's hard to think of how you would improve on the actual underlying design. I really wish that Go's container types were persistent immutable or some-such. It wouldn't solve everything, but it feels to me like if they could've managed to do that, it would've been a lot easier to reason about.
  - masklinn 4y ago
    > And though it's confusing at times, the design does actually make sense; without a doubt, it's hard to think of how you would improve on the actual underlying design. Go slices are absolutely the worst type in Go, because out of laziness they serve as both slices and vectors rather than have a separate, independent, and opaque vector types. This schizophrenia is the source of most if not all their traps and issues. > I really wish that Go's container types were persistent immutable or some-such. That would go against everything Go holds dear, since it's allergic to immutability and provides no support whatsoever for it (aside from simple information hiding).
dubswithus 4y ago
> We developed a system to detect data races at Uber using a dynamic data race detection technique. This system, over a period of six months, detected about 2,000 data races in our Go code base, of which our developers already fixed ~1,100 data races. This isn't open source, correct?
- aaronbwebber 4y ago
  Yes it is, it's part of the standard go toolchain as described in the first blog post in the series: https://eng.uber.com/dynamic-data-race-detection-in-go-code/ https://eng.uber.com/dynamic-data-race-detection-in-go-code/
aaronbwebber 4y ago
At least some of these would be caught by running your tests with race detection on? I haven't read the whole article yet but as soon as I read the loop variable one I was pretty sure I have written code with that exact bug and had it caught by tests... https://go.dev/doc/articles/race_detector https://go.dev/doc/articles/race_detector Edit: at the _end_ of the post, they mention that this is the second of two blog posts talking about this, and in the first post they explain that they caught these by deploying the default race detector and why they haven't been running it as part of CI (tl;dr it's slower and more resource-expensive and they had a large backlog). https://eng.uber.com/dynamic-data-race-detection-in-go-code/ https://eng.uber.com/dynamic-data-race-detection-in-go-code/
Klasiaster 4y ago
My favorite example is the IP address type which is an alias for a slice of bytes (type IP []byte). Thus, it gets passed by reference instead of by value and you easily end up working on the same data even if you didn't plan to. This will just be a logical bug but there are data structures in Go which result in memory corruption and introduce the risk of (remote) code execution vulnerabilities.
- armitron 4y ago
  > Thus, it gets passed by reference instead of by value Everything in Go is passed by value, including slices. Go has no reference types, but a lot of people think it does, and that’s a problem. An example of the much bigger problem of low barrier to entry programming, where lots of folks write code but have no deep understanding of the tools they use. If there’s something that Go proves, it’s that one can’t make a language “idiot proof”. Rust has the same problem in terms of folks creating mess after mess with it, except it’s higher barrier to entry and gets a better caliber of people using it.
  - tialaramex 4y ago
    You can't make a language "idiot proof" but by changing how we think about the problem we can make a huge difference. The trick is, what programs do we even want to exist? There's no need to be able to write all the programs you didn't want. In Rust, such programs get consigned to unsafe, which means that yes, sometimes to do general purpose programming (and especially e.g. in Rust's own stdlib) you must use unsafe Rust. But it already means we can constrain "idiots" (or more reasonably, new programmers) to safe Rust and rule out all those problems that aren't in the reduced domain of safe Rust. You can go much further than Rust. WUFFS isn't a general purpose language at all. While a Rust compiler written entirely in Rust isn't a priority it'll likely happen sooner or later, but a WUFFS compiler written in WUFFS is nonsense, WUFFS doesn't even have strings. WUFFS is for, well, Wrangling Untrusted File Formats Safely, hence the name. Notice not files just the file format. WUFFS has no idea what a file is, no file APIs, since it doesn't know what strings are it couldn't easily name files anyway. But inside its domain WUFFS gets to be 100% safe while also being faster than code you'd actually write in other languages. Take buffer overflow buffer[n]. In a language like C++ direct access isn't bounds checked and so overflows are common when n is too large, too dangerous. OK, in a language like (safe) Rust this access is bounds checked, now the overflow is prevented when n is too large but the bounds check cost CPU cycles, a little slower. WUFFS doesn't do either, in WUFFS that variable n was used to index into buffer therefore n is constrained to be 0 <= n < buffer size. If the compiler can see any way that n might exceed this constraint your program does not compile. As a result at runtime there's no overflow and no bounds checking. A complete idiot's WUFFS GIF decoder might be wrong - it could report spurious decoding errors, it could decode a blue dog as a pink roller skate, render images upside down or even decode JPEG instead of GIF - whatever, but it can't escape the limits of WUFFS itself. It can't go off piste and send your password database to a remote HTTP server or delete all your logs, or send spam emails or run some machine code it found inside the supposed GIF file.
jimmymommawitz 4y ago
hintymad 4y ago
> and contains approximately 2,100 unique Go services (and growing). A side topic: this is really not something to be proud of. There used to be more people than quantity of work in Uber and engineers fought for credits by building bogus decomposed services, and the sheer number of services seems indicate it's still so.
- deleted 4y ago
  [deleted]
- jeremy_wiebe 4y ago
  That number really store out to me too. I’d be very curious how they decide what becomes a separate service. I’d also be curious if the 50m lines of code included generated code.
  - shp0ngle 4y ago
    I did not work in Uber, but a similar company, and... it's a very political thing, usually. Taking an existing service and making it into 2 new microservices is a "thing you did". Suddenly, you have "impact" and can claim the new service as "yours". Everyone wants to be a king of their little kingdom.
    - crabbygrabby 4y ago
      You are me and I am you, I'll always be with you
- jordanbeiber 4y ago
  If they still use cadence/temporal[0] extensively this kind of blurs the concept of technical ”services”. We’ve started to use it (temporal) a bit for general automations, and it’s pretty great. Monorepo with a lot of different activities (“microservices”) makes sense. The activites are orchestrated in workflows (much like DDD “sagas”) and scheduled via temporal. This gives awesome introspection and observability. [0] https://temporal.io/ https://temporal.io/
smasher164 4y ago
This definitely matches my experience using Go at my previous organization. 1. Closures and concurrency really don't mix well. The loop variable capture in particular is very pernicious. There's an open issue to change this behavior in the language: https://github.com/golang/go/issues/20733 https://github.com/golang/go/issues/20733. 2. Yep. I've seen this problem in our codebase. I've grown to just be very deliberate with data that needs to be shared. Put it all in a struct that's passed around by its pointer. 3. This issue is caught fairly easily by the race detector. Using a sync.Map or a lock around a map is pretty easy to communicate with other Go devs. 4. This should be documented better, but the convention around structs that should not be passed around by value is to embed a noCopy field inside. https://github.com/golang/go/issues/8005#issuecomment-190753527 https://github.com/golang/go/issues/8005#issuecomment-190753... This will get caught by go vet, since it'll treat it like a Locker. 5 & 6. Go makes it pretty easy to do ad-hoc concurrency as you see fit. This makes it possible for people to just create channels, waitgroups, and goroutines willy-nilly. It's really important to design upfront how you're gonna do an operation concurrently, especially because there aren't many guardrails. I'd suggest that many newcomers stick with x/sync.ErrGroup (which forces you to use its Go method, and can now set a cap on the # of goroutines), and use a *sync.Mutex inside a struct in 99% of cases. 7. Didn't encounter this that often, but sharing a bunch of state between (sub)tests should already be a red flag. Either there's something global that you initialized at the very beginning (like opening a connection), or that state should be scoped and passed down to that individual test, so it can't really infect everything around it.
- 0xFACEFEED 4y ago
  > 3. This issue is caught fairly easily by the race detector. Using a sync.Map or a lock around a map is pretty easy to communicate with other Go devs. I run my Go development server with the -race flag as a default. If it affects performance I'll turn it off but that's very rare in practice. Unfortunately a lot of applications don't run tests against their HTTP endpoints (like only internal library stuff) which is bad bad bad, but the -race flag at least helps mitigate. To anyone reading who cares: 1) Always run your tests with the -race flag! 2) Always write tests for your HTTP handling code too! 3) Run your dev server with -race for a week and see what happens. This will hard crash your Go program and there is nothing you can do about it. You can't recover(). Go vet will not catch anything. The -race flag will! package main import "time" func main() { m := map[int]int{} go poop(m) go poop(m) time.Sleep(5 * time.Second) } func poop(m map[int]int) { for i := 0; i < 1e10; i++ { m[i] = i } }
fmakunbound 4y ago
> contains approximately 2,100 unique Go services (and growing) How is that possible and what do they do???
- Tomis02 4y ago
  Sadly, people get rewarded for what they built (even if makes everyone's lives harder in the long run) than for exercising restraint and saying "no".
  - erik_seaberg 4y ago
    There’s a perception that big tech pays longtime employees less than they’re willing to offer new candidates, making promotion or job hopping the best ways to earn the current market rate. And if you copy Google’s promo process, you get promo-driven development, because there aren’t enough projects that actually need that level of complexity.
    - eru 4y ago
      And sadly, Google isn't even the worst. At least they seemed to allocate quite a few resources for turning off old systems and decommissioning them. At many other enterprises, old systems never properly die. It's just as hard, or often harder, to migrate off the last few uses of a system compared to launching a new system. But while you can get promoted for launching a great-enough new system in almost any organisation, good luck getting promoted for your heroic efforts in shutting down obsolete systems. (I guess it's technically possible. Just unlikely in most places.)
fellellor 4y ago
https://youtu.be/-bCkha6U70o?t=787 https://youtu.be/-bCkha6U70o?t=787
DominoTree 4y ago
Seems like Rob Pike and co may have failed "The key point here is our programmers... They’re not capable of understanding a brilliant language... So, the language that we give them has to be easy for them to understand"
- deleted 4y ago
  [deleted]
- laerus 4y ago
  Yep, if Google wasn't behind Go the language would have already been history like so many other half baked technologies. It won't be long untill people will talk about Golang like they do about JavaScript.
  - crabbygrabby 4y ago
    JavaScript is nutty but let's be honest, js is one of the most widely used programming languages. For better and for worse. I don't think go lang will die out because it does get some things right. Unfortunately, there's still a bit of things going wrong.
    - Groxx 4y ago
      Honestly, I expect Go to be lumped together with JavaScript in terms of "weird but still in use" languages in the long run. There are so many surprising footguns and unsafe patterns that it really stands out as a risky language to me. But it has Google's (implied) backing and it works well enough to be used, and the performance is very good in general. By this point it can probably survive for quite a long time on momentum alone. Which makes it a moderately-safe-to-use-in-a-business language.
  - icedchai 4y ago
    They said the same thing about JavaScript in the 90's. If Netscape wasn't behind it... Over 25 years later, it's still going strong, even though the entire HTML/CSS/JS model of "app development" is and always was half baked.
ohazi 4y ago
A dig against Rust I sometimes hear is "Oh, data race freedom isn't such a big deal, if you really need it, a garbage collected language like Java will give you that guarantee." So now I'm hearing that Go, a garbage collected language, doesn't guarantee data race freedom? I guess it's garbage collected but not "managed" by a runtime or something? Why go to all that effort to get off of C++ just to stop 30% short? These are C-like concurrency bugs, and you still have to use C-like "if not nil" error handling. Why do people keep adopting this language? Where's the appeal?
- Santosh83 4y ago
  Memory safe, excellent tooling, excellent base std library, no manual memory management, static binaries, trivial cross compilation, trivial concurrent programming etc. These advantages are still not inconsiderable over C, although not C++. Yes, however, it is not as "safe" as Rust. But downside is C++ & Rust are harder to design for upfront.
  - masklinn 4y ago
    > Memory safe [...] trivial concurrent programming As this post demonstrates, data races are trivial and common in Go. Because many of the core types (interface, slice, map) are non-thread-safe multiword structures, they also break memory safety: https://blog.stalkr.net/2015/04/golang-data-races-to-break-memory-safety.html https://blog.stalkr.net/2015/04/golang-data-races-to-break-m...
    - throwaway894345 4y ago
      I’ve been programming in Go for a decade at this point, and I’m sure I’ve probably run into a data race before, but for the life of me, I can’t think of any specific instances. I’m not sure they’re as common as you think. Far more often, I’ll run into race conditions in some service (multiple processes touching some network state concurrently), but this happens as often in Go as in Rust or any other language.
      - 4y ago
rapiz 4y ago
Don't take me seriously: I heard people both saying Golang is a better C/Golang is a worse C
100xdang 4y ago
100xdang 4y ago
deleted 4y ago
[deleted]
baalimago 4y ago
Passing sync.Mutex and sync.WaitGroup as value is especially irritating considering that the context.Context is meant to be passed as such
freyr 4y ago
> Uber has adopted Golang (Go for short) Uber has adopted Go (Golang for long)
eurasiantiger 4y ago
”Our Go monorepo consists of about 50 million lines of code (and growing) and contains approximately 2,100 unique Go services (and growing).” What the hell is that company doing? Try to imagine an ERD or DFD of their day-to-day operations. 2,100 unique services…
gigatexal 4y ago
In the closure example does declaring a new variable and setting its value to the iterative or the thing being passed in, does that mitigate the pass by reference issue?
- yencabulator 4y ago
  Yes. for i := 0; i < 10; i++ { i := i ... } https://go.dev/play/p/P7TunJCL7RS https://go.dev/play/p/P7TunJCL7RS
  - gigatexal 4y ago
    Thank you! The example is brilliant. Thanks.
Synersoft 4y ago
[dead]
jasonhansel 4y ago
I think the root cause of a lot of these data races is that Go has no way of marking variables/fields/pointers/etc. as immutable or constant. This makes it easy to lose track of shared mutable state. It's not just data races--it's also logical races, which are near-impossible to detect or prevent without something like transactional memory.
- eru 4y ago
  Yes, missing immutability is a big part of the problem in Go. Given that Go eschewed generics for the longest time, I can sort of see why they left out immutability markers: To keep your sanity, you'd want some functions to take (and return!) both mutable and immutable data, as the situation requires. But some other functions should only take mutable data or only immutable data. Thus ideally you'd need some kind of 'generic' (im-)mutability handling in your type system. (Rust's borrow checker is basically one way to really deal with this (im-)mutability genericity. For example, a function to do binary search on a sorted array doesn't change the array; thus it could take either a mutable or immutable version. But if the array changes (via another thread) while the function is running, then you might get into trouble.)
ryanschneider 4y ago
What’s up with the totally broken syntax highlighting in this post, at least on iOS? 2100 micro services and not one of them is a valid syntax highlighter for blog posts. Edit: oh I see it highlights red and underlines every keyword. I find that incredibly distracting, so much so I assumed their highlighter was broken, but also just realized they are screenshots.
stevefan1999 4y ago
can you guys put it up as a SonarQube rules?
dwrodri 4y ago
Is it just me, or is Golang's concurrency a very double edged sword? My exposure to goroutines and channels was mind-blowing, but I still really struggle reading through Go code and understanding what's going on in memory. The way that Go "abstracts away" ownership feels more like it's hiding important details rather than unnecessary minutia. Here's a simple question that's stumped me for some time: if multiple go routines are popping values out of a channel, does the channel need a mutex? Why do the "fan-out, fan-in?" examples in the "Pipelines and Cancellation" post on the Go blog not require mutex locks? Link here: https://go.dev/blog/pipelines https://go.dev/blog/pipelines Stuff like that, along with the ambiguity of intializing stuff by value vs using make, the memory semantics of some of the primitives (slices, channels, etc). None of it was like "of course". If something is a reference, I'd rather the language tell me it's a reference. Maybe I'm still too new to the language.
- bombela 4y ago
  I am pretty sure the channels are safe to use with multiple producers and multiple consumers (mpmc). But somehow I cannot easily find any official doc clarifying that. Go doesn't do anything to help you with memory safety around concurrency. And the design of the language is also not helping you avoid logical bugs. After using Rust, all other imperative languages feel like using an angle grinder with a wood saw blade and no guard. Sure you can do really good if you are careful. But thing will go sideways remarkably quickly. And with the constant urgency of shipping for yesterday. It makes sense most programs look like the aftermath of The Boys show.
  - Groxx 4y ago
    They are MPMC except for closing (which panics if anything writes after close), yes.
sharno 4y ago
Go picked the concurrency ideas of Erlang but then ignored the main safeguard that makes Erlang's concurrency fearless: Immutability. And if you feel that Erlang's lack of type safety is an issue, then Gleam has you covered.
- eru 4y ago
  Erlang was fun to use! But even with Erlang, concurrency is hard. Any single process's data is immutable, but if you split a process in twain, the resulting union can behave as if it had mutable state. And let's not forget about ETS (term storage), which is basically a mutable hash table that you often have to use to get anything done. In any case, I agree that Go did _not_ improve on Erlang.
- masklinn 4y ago
  > Go picked the concurrency ideas of Erlang but then ignored the main safeguard that makes Erlang's concurrency fearless: Immutability. Not even immutability, isolation. Though obviously immutability makes things less weird, the real gain in terms of concurrency is that you can’t touch any data other than your own (process’s), and for the most part erlang doesn’t “cheat” either: aside from binaries, terms are actually copied over when sent, each process having its own heap. Sequential erlang could be a procedural language based around mutability and it wouldn’t much alter its reliability guarantees (assuming binaries remain immutable, or become COW).
oconnor663 4y ago
Is there a mistake in Figure 2? It looks like myResults is captured, but the capture is never used?

307 points by statenjason 4y ago

I wrote a deep analysis/reaction to this post: https://medium.com/@scott_white/concurrency-in-go-is-not-magic-37bb16af4b1a https://medium.com/@scott_white/concurrency-in-go-is-not-mag... tl;dr Go doesn't magically solve data races and blaming the language itself isn't well supported by the examples/data.

thallavajhula 4y ago

>We developed a system to detect data races at Uber using a dynamic data race detection technique. By system do you mean a process or a tool that detects these?

bumper_crop 4y ago

Sorry to say, but these hit close to home for me. A lot of the synchronization paradigms in Go are easy to misuse, but lead the author into thinking it's okay. the WaitGroup one is particularly poignant for me, since the race detector doesn't catch it. I'll add one other data race goof: atomic.Value. Look at the implementation. Unlike pretty much every other language I've seen, atomic.Value isn't really atomic, since the concrete type can't ever change after being set. This stems from fact that interfaces are two words rather than one, and they can't be (hardware) atomically set. To fix it, Go just documents "hey, don't do that", and then panics if you do.

morelisp 4y ago
> atomic.Value isn't really atomic, since the concrete type can't ever change after being set. How does this mean it's non-atomic? As far as I know you can still never Load() a partial Store(). (Also, even if it was possible, this would never be a good idea...)
- bumper_crop 4y ago
  That's why I opened with "Look at the implementation". Go is unable to store the type and the pointer at the same time, so it warps what "atomic" means. Pretty much every other language has atomic mean "one of these will win, one will lose". Go says "one will win, one will panic and destroy the goroutine. In fact, it's even worse than that. If the Store() caller goes to sleep between setting the type and storing the pointer, it causes every Goroutine that calls Load() to block. They can't make forward progress if the store caller hangs.
  - fastest963 4y ago
    This is why all the examples call Store immediately with a zero value of the type.
    - bumper_crop 4y ago
      https://go.dev/play/p/xolc9oPwA0C https://go.dev/play/p/xolc9oPwA0C Interfaces don't have a zero type, which means that we can't have an atomic.Value which stores Shape. Atomic Value would be much easier to reason about if it had store semantics similar to a regular `var foo Shape = ...`. One of the other comment threads talked about generics helping this, so maybe there is hope.
      - deleted 4y ago
        [deleted]
      - yencabulator 4y ago
        Parent means var bestShape atomic.Value bestShape.Store((*Circle)(nil))
        morelisp 4y ago
        Which will store it as a *Circle, and only allow more *Circles, not Shapes. That part of GP’s claim is correct. It just had nothing to do with atomicity; it means something specific, not just “I like the failure mode.”
Groxx 4y ago
The lack of generics has forced all Go concurrency to be intrusive (i.e. implemented by the person using literally any concurrency), and yeah. It's horrifyingly error-prone in my experience. It means everyone needs to be an expert, and lol, everyone is not an expert. Generics might save us from the simple, mechanical flaws. Expect to see `Locker<T>` and `Atomic<T>` types cropping up. And unbounded buffered thread-safe queues backing channels. Etc. I'm very, very much looking forward to it. --- edited to rant more --- I also really wonder where all these "go makes concurrency a first-class concept" claims come from, because I see it quite a few places, and I feel like it's making some very strong implied claims that absolutely do not exist. Go has channels and select. That's neat. But on the other hand it has threads... but no thread handles. It has implicit capturing of closures. It has ambiguous value vs pointer semantics. It (style- and ergonomic-wise) encourages field references, which have no way to enforce mutexes or atomics. It has had crippled lock APIs that effectively force use of channels for... I don't know, philosophical reasons? Go is abnormally dangerous when it comes to concurrency IMO. The race detector does an amazing job helping you discover it, but it's very easy to not use it or not take full advantage of it (i.e. non-parallel tests), and few run their production services with the race detector enabled. Because if they did, it would crash all the time, because there are an absurd amount of races in nearly all of the popular libraries (and in common use of those libraries, because concurrency is not a first-class citizen and you can't tell when it's happening / when it shouldn't happen).
- hexxagone 4y ago
  Go does not have threads but something like "tasks". The fact that no thread handle is exposed allows for transparently moving these tasks across threads if the scheduler decides so. "go makes concurrency a first-class concept" I think it usually refers to goroutines being built in the language. "Go is abnormally dangerous when it comes to concurrency IMO". Personnally, it has not been my experience with Go concurrency. However I have hit some issues when trying to ocrhestrate tasks via channels and ended up resorting to atomics to do the job.
  - saghm 4y ago
    > Go does not have threads but something like "tasks". The fact that no thread handle is exposed allows for transparently moving these tasks across threads if the scheduler decides so. This doesn't stop there being "task handles" then, though? I think the point GP was making is that something that in most languages would be simple methods on a handle like "wait for this task to finish" or "stop this task" instead need to be done manually in Go with channels (or potentially `Context` in the latter case, although that was a later addition to the standard library). It doesn't really matter whether you call it a thread or a task; either way, it would be nice to get some return value from spawning some background operation and being able to use it to directly interact with it. I agree with GP that it does seem like an odd omission, since I haven't really heard any actual practical explanation for it.
avgcorrection 4y ago
This is Atomic* * Just don’t be an idiot. Worse is better.
nindalf 4y ago
On atomics in Go, the beta for Go 1.19 was released an hour ago (https://groups.google.com/g/golang-announce/c/SNruPJUSFz0?pli=1 https://groups.google.com/g/golang-announce/c/SNruPJUSFz0?pl...). > The sync/atomic package defines new atomic types Bool, Int32, Int64, Uint32, Uint64, Uintptr, and Pointer. These types hide the underlying values so that all accesses are forced to use the atomic APIs. Pointer also avoids the need to convert to unsafe.Pointer at call sites. Int64 and Uint64 are automatically aligned to 64-bit boundaries in structs and allocated data, even on 32-bit systems. Go 1.19 is expected to release in August.

travisd 4y ago

Worth noting that some of these can be detected statically -- and some are detected by go vet (e.g., passing a sync.Mutex by value). I don't think it detects the wg.Add bug, but that seems relatively straightforward(†) to add a check for. (†famous last words, I know)

kjksf 4y ago
staticcheck has a check for wg.Add misuse (https://staticcheck.io/docs/checks https://staticcheck.io/docs/checks, https://staticcheck.io/docs/checks#SA2000 https://staticcheck.io/docs/checks#SA2000)

deleted 4y ago

[deleted]

JamesSwift 4y ago

Really good article, and gave me several ideas to track down some gremlins that have been bugging me in a go codebase recently.

jchw 4y ago

This is pretty cool. 50 million lines of code is quite a large corpus to work off of. I'm surprised by some of them. For example, go vet nominally catches misuses of mutexes, so it's surprising that even a few of those slipped through. I wonder if those situations are a bit more complicated than the example. Obviously, the ideal outcome is that static analysis can help eliminate as many issues as possible, by restricting the language, discouraging bad patterns, or giving the programmer more tools to detect bugs. gVisor, for example, has a really interesting tool called checklocks: https://github.com/google/gvisor/tree/master/tools/checklocks https://github.com/google/gvisor/tree/master/tools/checklock... While it definitely has some caveats, ideas like these should help Go programs achieve a greater degree of robustness. Obviously, this class of error would be effectively prevented by borrow checking, but I suppose if you want programming language tradeoffs more tilted towards robustness, Rust already has a lot of that covered.

likeabbas 4y ago
> I suppose if you want programming language tradeoffs more tilted towards robustness, Rust already has a lot of that covered Does anyone not want robustness of their language to cover their mistakes?
- setr 4y ago
  For free? Of course not. At cost? … depends on what you’re charging me, and how much I’m getting
  - likeabbas 4y ago
    Good point. Right now I kind of see modern programming as two fold: 1) Loosely typed to get you what you want faster, but with some mistakes, and 2) Strongly typed that forces you to try harder, but ultimately better I'm usually happier with the latter. I find I become far more frustrated when I try to write python than I do something like Rust just because I know when I write Python that I will have mistakes I'll have to fix in prod, vs when I write Rust I won't have those mistakes (although it'll take me longer to get something to prod)

metadat 4y ago

> 2. Slices are confusing types that create subtle and hard-to-diagnose data races The "Slices" example is just nasty! Like, this is just damning for Go's promise of "_relatively_ easy and carefree concurrency". Think about it for a second or two, >> The reference to the slice was resized in the middle of an append operation from another async routine. What exactly happens in these cases? How can I trust myself, as a fallible human being, to reason about such cases when I'm trying to efficiently roll up a list of results. :-/ Compared to every other remotely mainstream language, perhaps even C++, these are extremely subtle and sharp.. nigh, razor sharp edges. Yuck. One big takeaway is this harsh realization: Golang guarantees are scant more than what is offered by pure, relatively naïve and unadulterated BASH shell programming. I still will use it, but with newfound fear. As a multi-hundred-kloc-authoring-gopher: I love Go, and this article is killing me inside. Go appears extremely sloppy at the edges of the envelope and language boundaries, moreso than even I had ever realized prior to now. Full-disclosure: I am disgusted by the company that is Uber, but I'm grateful to the talented folks who've cast a light on this cesspool region of Golang. Thank you! p.s. inane aside: I never would've guessed that in 2022, Java would start looking more and more appealing in new ways. Until now I've been more or less "all-in" on Go for years.

jchw 4y ago
Slices may just be one of the best and worst parts of Go. They're cumbersome, their behavior sometimes feels 'inexplicable,' and even as an experienced developer you are likely to eventually fallen into one of the traps where your 'obvious' code isn't so obvious. That said... when programming in programming languages without a slice type, I always want to have one. And though it's confusing at times, the design does actually make sense; without a doubt, it's hard to think of how you would improve on the actual underlying design. I really wish that Go's container types were persistent immutable or some-such. It wouldn't solve everything, but it feels to me like if they could've managed to do that, it would've been a lot easier to reason about.
- masklinn 4y ago
  > And though it's confusing at times, the design does actually make sense; without a doubt, it's hard to think of how you would improve on the actual underlying design. Go slices are absolutely the worst type in Go, because out of laziness they serve as both slices and vectors rather than have a separate, independent, and opaque vector types. This schizophrenia is the source of most if not all their traps and issues. > I really wish that Go's container types were persistent immutable or some-such. That would go against everything Go holds dear, since it's allergic to immutability and provides no support whatsoever for it (aside from simple information hiding).

dubswithus 4y ago

> We developed a system to detect data races at Uber using a dynamic data race detection technique. This system, over a period of six months, detected about 2,000 data races in our Go code base, of which our developers already fixed ~1,100 data races. This isn't open source, correct?

aaronbwebber 4y ago
Yes it is, it's part of the standard go toolchain as described in the first blog post in the series: https://eng.uber.com/dynamic-data-race-detection-in-go-code/ https://eng.uber.com/dynamic-data-race-detection-in-go-code/

aaronbwebber 4y ago

At least some of these would be caught by running your tests with race detection on? I haven't read the whole article yet but as soon as I read the loop variable one I was pretty sure I have written code with that exact bug and had it caught by tests... https://go.dev/doc/articles/race_detector https://go.dev/doc/articles/race_detector Edit: at the _end_ of the post, they mention that this is the second of two blog posts talking about this, and in the first post they explain that they caught these by deploying the default race detector and why they haven't been running it as part of CI (tl;dr it's slower and more resource-expensive and they had a large backlog). https://eng.uber.com/dynamic-data-race-detection-in-go-code/ https://eng.uber.com/dynamic-data-race-detection-in-go-code/

Klasiaster 4y ago

My favorite example is the IP address type which is an alias for a slice of bytes (type IP []byte). Thus, it gets passed by reference instead of by value and you easily end up working on the same data even if you didn't plan to. This will just be a logical bug but there are data structures in Go which result in memory corruption and introduce the risk of (remote) code execution vulnerabilities.

armitron 4y ago
> Thus, it gets passed by reference instead of by value Everything in Go is passed by value, including slices. Go has no reference types, but a lot of people think it does, and that’s a problem. An example of the much bigger problem of low barrier to entry programming, where lots of folks write code but have no deep understanding of the tools they use. If there’s something that Go proves, it’s that one can’t make a language “idiot proof”. Rust has the same problem in terms of folks creating mess after mess with it, except it’s higher barrier to entry and gets a better caliber of people using it.
- tialaramex 4y ago
  You can't make a language "idiot proof" but by changing how we think about the problem we can make a huge difference. The trick is, what programs do we even want to exist? There's no need to be able to write all the programs you didn't want. In Rust, such programs get consigned to unsafe, which means that yes, sometimes to do general purpose programming (and especially e.g. in Rust's own stdlib) you must use unsafe Rust. But it already means we can constrain "idiots" (or more reasonably, new programmers) to safe Rust and rule out all those problems that aren't in the reduced domain of safe Rust. You can go much further than Rust. WUFFS isn't a general purpose language at all. While a Rust compiler written entirely in Rust isn't a priority it'll likely happen sooner or later, but a WUFFS compiler written in WUFFS is nonsense, WUFFS doesn't even have strings. WUFFS is for, well, Wrangling Untrusted File Formats Safely, hence the name. Notice not files just the file format. WUFFS has no idea what a file is, no file APIs, since it doesn't know what strings are it couldn't easily name files anyway. But inside its domain WUFFS gets to be 100% safe while also being faster than code you'd actually write in other languages. Take buffer overflow buffer[n]. In a language like C++ direct access isn't bounds checked and so overflows are common when n is too large, too dangerous. OK, in a language like (safe) Rust this access is bounds checked, now the overflow is prevented when n is too large but the bounds check cost CPU cycles, a little slower. WUFFS doesn't do either, in WUFFS that variable n was used to index into buffer therefore n is constrained to be 0 <= n < buffer size. If the compiler can see any way that n might exceed this constraint your program does not compile. As a result at runtime there's no overflow and no bounds checking. A complete idiot's WUFFS GIF decoder might be wrong - it could report spurious decoding errors, it could decode a blue dog as a pink roller skate, render images upside down or even decode JPEG instead of GIF - whatever, but it can't escape the limits of WUFFS itself. It can't go off piste and send your password database to a remote HTTP server or delete all your logs, or send spam emails or run some machine code it found inside the supposed GIF file.

jimmymommawitz 4y ago

hintymad 4y ago

> and contains approximately 2,100 unique Go services (and growing). A side topic: this is really not something to be proud of. There used to be more people than quantity of work in Uber and engineers fought for credits by building bogus decomposed services, and the sheer number of services seems indicate it's still so.

deleted 4y ago
[deleted]
jeremy_wiebe 4y ago
That number really store out to me too. I’d be very curious how they decide what becomes a separate service. I’d also be curious if the 50m lines of code included generated code.
- shp0ngle 4y ago
  I did not work in Uber, but a similar company, and... it's a very political thing, usually. Taking an existing service and making it into 2 new microservices is a "thing you did". Suddenly, you have "impact" and can claim the new service as "yours". Everyone wants to be a king of their little kingdom.
  - crabbygrabby 4y ago
    You are me and I am you, I'll always be with you
jordanbeiber 4y ago
If they still use cadence/temporal[0] extensively this kind of blurs the concept of technical ”services”. We’ve started to use it (temporal) a bit for general automations, and it’s pretty great. Monorepo with a lot of different activities (“microservices”) makes sense. The activites are orchestrated in workflows (much like DDD “sagas”) and scheduled via temporal. This gives awesome introspection and observability. [0] https://temporal.io/ https://temporal.io/

smasher164 4y ago

This definitely matches my experience using Go at my previous organization. 1. Closures and concurrency really don't mix well. The loop variable capture in particular is very pernicious. There's an open issue to change this behavior in the language: https://github.com/golang/go/issues/20733 https://github.com/golang/go/issues/20733. 2. Yep. I've seen this problem in our codebase. I've grown to just be very deliberate with data that needs to be shared. Put it all in a struct that's passed around by its pointer. 3. This issue is caught fairly easily by the race detector. Using a sync.Map or a lock around a map is pretty easy to communicate with other Go devs. 4. This should be documented better, but the convention around structs that should not be passed around by value is to embed a noCopy field inside. https://github.com/golang/go/issues/8005#issuecomment-190753527 https://github.com/golang/go/issues/8005#issuecomment-190753... This will get caught by go vet, since it'll treat it like a Locker. 5 & 6. Go makes it pretty easy to do ad-hoc concurrency as you see fit. This makes it possible for people to just create channels, waitgroups, and goroutines willy-nilly. It's really important to design upfront how you're gonna do an operation concurrently, especially because there aren't many guardrails. I'd suggest that many newcomers stick with x/sync.ErrGroup (which forces you to use its Go method, and can now set a cap on the # of goroutines), and use a *sync.Mutex inside a struct in 99% of cases. 7. Didn't encounter this that often, but sharing a bunch of state between (sub)tests should already be a red flag. Either there's something global that you initialized at the very beginning (like opening a connection), or that state should be scoped and passed down to that individual test, so it can't really infect everything around it.

0xFACEFEED 4y ago
> 3. This issue is caught fairly easily by the race detector. Using a sync.Map or a lock around a map is pretty easy to communicate with other Go devs. I run my Go development server with the -race flag as a default. If it affects performance I'll turn it off but that's very rare in practice. Unfortunately a lot of applications don't run tests against their HTTP endpoints (like only internal library stuff) which is bad bad bad, but the -race flag at least helps mitigate. To anyone reading who cares: 1) Always run your tests with the -race flag! 2) Always write tests for your HTTP handling code too! 3) Run your dev server with -race for a week and see what happens. This will hard crash your Go program and there is nothing you can do about it. You can't recover(). Go vet will not catch anything. The -race flag will! package main import "time" func main() { m := map[int]int{} go poop(m) go poop(m) time.Sleep(5 * time.Second) } func poop(m map[int]int) { for i := 0; i < 1e10; i++ { m[i] = i } }

fmakunbound 4y ago

> contains approximately 2,100 unique Go services (and growing) How is that possible and what do they do???

Tomis02 4y ago
Sadly, people get rewarded for what they built (even if makes everyone's lives harder in the long run) than for exercising restraint and saying "no".
- erik_seaberg 4y ago
  There’s a perception that big tech pays longtime employees less than they’re willing to offer new candidates, making promotion or job hopping the best ways to earn the current market rate. And if you copy Google’s promo process, you get promo-driven development, because there aren’t enough projects that actually need that level of complexity.
  - eru 4y ago
    And sadly, Google isn't even the worst. At least they seemed to allocate quite a few resources for turning off old systems and decommissioning them. At many other enterprises, old systems never properly die. It's just as hard, or often harder, to migrate off the last few uses of a system compared to launching a new system. But while you can get promoted for launching a great-enough new system in almost any organisation, good luck getting promoted for your heroic efforts in shutting down obsolete systems. (I guess it's technically possible. Just unlikely in most places.)

fellellor 4y ago

https://youtu.be/-bCkha6U70o?t=787 https://youtu.be/-bCkha6U70o?t=787

DominoTree 4y ago

Seems like Rob Pike and co may have failed "The key point here is our programmers... They’re not capable of understanding a brilliant language... So, the language that we give them has to be easy for them to understand"

deleted 4y ago
[deleted]
laerus 4y ago
Yep, if Google wasn't behind Go the language would have already been history like so many other half baked technologies. It won't be long untill people will talk about Golang like they do about JavaScript.
- crabbygrabby 4y ago
  JavaScript is nutty but let's be honest, js is one of the most widely used programming languages. For better and for worse. I don't think go lang will die out because it does get some things right. Unfortunately, there's still a bit of things going wrong.
  - Groxx 4y ago
    Honestly, I expect Go to be lumped together with JavaScript in terms of "weird but still in use" languages in the long run. There are so many surprising footguns and unsafe patterns that it really stands out as a risky language to me. But it has Google's (implied) backing and it works well enough to be used, and the performance is very good in general. By this point it can probably survive for quite a long time on momentum alone. Which makes it a moderately-safe-to-use-in-a-business language.
- icedchai 4y ago
  They said the same thing about JavaScript in the 90's. If Netscape wasn't behind it... Over 25 years later, it's still going strong, even though the entire HTML/CSS/JS model of "app development" is and always was half baked.

ohazi 4y ago

A dig against Rust I sometimes hear is "Oh, data race freedom isn't such a big deal, if you really need it, a garbage collected language like Java will give you that guarantee." So now I'm hearing that Go, a garbage collected language, doesn't guarantee data race freedom? I guess it's garbage collected but not "managed" by a runtime or something? Why go to all that effort to get off of C++ just to stop 30% short? These are C-like concurrency bugs, and you still have to use C-like "if not nil" error handling. Why do people keep adopting this language? Where's the appeal?

Santosh83 4y ago
Memory safe, excellent tooling, excellent base std library, no manual memory management, static binaries, trivial cross compilation, trivial concurrent programming etc. These advantages are still not inconsiderable over C, although not C++. Yes, however, it is not as "safe" as Rust. But downside is C++ & Rust are harder to design for upfront.
- masklinn 4y ago
  > Memory safe [...] trivial concurrent programming As this post demonstrates, data races are trivial and common in Go. Because many of the core types (interface, slice, map) are non-thread-safe multiword structures, they also break memory safety: https://blog.stalkr.net/2015/04/golang-data-races-to-break-memory-safety.html https://blog.stalkr.net/2015/04/golang-data-races-to-break-m...
  - throwaway894345 4y ago
    I’ve been programming in Go for a decade at this point, and I’m sure I’ve probably run into a data race before, but for the life of me, I can’t think of any specific instances. I’m not sure they’re as common as you think. Far more often, I’ll run into race conditions in some service (multiple processes touching some network state concurrently), but this happens as often in Go as in Rust or any other language.
    - 4y ago

rapiz 4y ago

Don't take me seriously: I heard people both saying Golang is a better C/Golang is a worse C

100xdang 4y ago

deleted 4y ago

[deleted]

baalimago 4y ago

Passing sync.Mutex and sync.WaitGroup as value is especially irritating considering that the context.Context is meant to be passed as such

freyr 4y ago

> Uber has adopted Golang (Go for short) Uber has adopted Go (Golang for long)

eurasiantiger 4y ago

”Our Go monorepo consists of about 50 million lines of code (and growing) and contains approximately 2,100 unique Go services (and growing).” What the hell is that company doing? Try to imagine an ERD or DFD of their day-to-day operations. 2,100 unique services…

gigatexal 4y ago

In the closure example does declaring a new variable and setting its value to the iterative or the thing being passed in, does that mitigate the pass by reference issue?

yencabulator 4y ago
Yes. for i := 0; i < 10; i++ { i := i ... } https://go.dev/play/p/P7TunJCL7RS https://go.dev/play/p/P7TunJCL7RS
- gigatexal 4y ago
  Thank you! The example is brilliant. Thanks.

Synersoft 4y ago

[dead]

jasonhansel 4y ago

I think the root cause of a lot of these data races is that Go has no way of marking variables/fields/pointers/etc. as immutable or constant. This makes it easy to lose track of shared mutable state. It's not just data races--it's also logical races, which are near-impossible to detect or prevent without something like transactional memory.

eru 4y ago
Yes, missing immutability is a big part of the problem in Go. Given that Go eschewed generics for the longest time, I can sort of see why they left out immutability markers: To keep your sanity, you'd want some functions to take (and return!) both mutable and immutable data, as the situation requires. But some other functions should only take mutable data or only immutable data. Thus ideally you'd need some kind of 'generic' (im-)mutability handling in your type system. (Rust's borrow checker is basically one way to really deal with this (im-)mutability genericity. For example, a function to do binary search on a sorted array doesn't change the array; thus it could take either a mutable or immutable version. But if the array changes (via another thread) while the function is running, then you might get into trouble.)

ryanschneider 4y ago

What’s up with the totally broken syntax highlighting in this post, at least on iOS? 2100 micro services and not one of them is a valid syntax highlighter for blog posts. Edit: oh I see it highlights red and underlines every keyword. I find that incredibly distracting, so much so I assumed their highlighter was broken, but also just realized they are screenshots.

stevefan1999 4y ago

can you guys put it up as a SonarQube rules?

dwrodri 4y ago

Is it just me, or is Golang's concurrency a very double edged sword? My exposure to goroutines and channels was mind-blowing, but I still really struggle reading through Go code and understanding what's going on in memory. The way that Go "abstracts away" ownership feels more like it's hiding important details rather than unnecessary minutia. Here's a simple question that's stumped me for some time: if multiple go routines are popping values out of a channel, does the channel need a mutex? Why do the "fan-out, fan-in?" examples in the "Pipelines and Cancellation" post on the Go blog not require mutex locks? Link here: https://go.dev/blog/pipelines https://go.dev/blog/pipelines Stuff like that, along with the ambiguity of intializing stuff by value vs using make, the memory semantics of some of the primitives (slices, channels, etc). None of it was like "of course". If something is a reference, I'd rather the language tell me it's a reference. Maybe I'm still too new to the language.

bombela 4y ago
I am pretty sure the channels are safe to use with multiple producers and multiple consumers (mpmc). But somehow I cannot easily find any official doc clarifying that. Go doesn't do anything to help you with memory safety around concurrency. And the design of the language is also not helping you avoid logical bugs. After using Rust, all other imperative languages feel like using an angle grinder with a wood saw blade and no guard. Sure you can do really good if you are careful. But thing will go sideways remarkably quickly. And with the constant urgency of shipping for yesterday. It makes sense most programs look like the aftermath of The Boys show.
- Groxx 4y ago
  They are MPMC except for closing (which panics if anything writes after close), yes.

sharno 4y ago

Go picked the concurrency ideas of Erlang but then ignored the main safeguard that makes Erlang's concurrency fearless: Immutability. And if you feel that Erlang's lack of type safety is an issue, then Gleam has you covered.

eru 4y ago
Erlang was fun to use! But even with Erlang, concurrency is hard. Any single process's data is immutable, but if you split a process in twain, the resulting union can behave as if it had mutable state. And let's not forget about ETS (term storage), which is basically a mutable hash table that you often have to use to get anything done. In any case, I agree that Go did _not_ improve on Erlang.
masklinn 4y ago
> Go picked the concurrency ideas of Erlang but then ignored the main safeguard that makes Erlang's concurrency fearless: Immutability. Not even immutability, isolation. Though obviously immutability makes things less weird, the real gain in terms of concurrency is that you can’t touch any data other than your own (process’s), and for the most part erlang doesn’t “cheat” either: aside from binaries, terms are actually copied over when sent, each process having its own heap. Sequential erlang could be a procedural language based around mutability and it wouldn’t much alter its reliability guarantees (assuming binaries remain immutable, or become COW).

oconnor663 4y ago

Is there a mistake in Figure 2? It looks like myResults is captured, but the capture is never used?