19 ms·

Updating the Go Memory Model

248 points by gbrown_ 5y ago

judofyr 5y ago
> In other words, Go encourages avoiding subtle bugs by avoiding subtle code. I find this philosophy strange when they talk about the communication primitives. Channels are extremely subtle in my experience. A channel can be buffered or unbuffered, it can be closed, and there’s different behavior of read/write whether it’s in a select or not. And then you often end up with multiple channels as well. Reviewing code with channels is often more confusing than code that uses mutexes in my opinion.
- richardwhiuk 5y ago
  I agree - I find Go's error handling encourages subtle bugs.
  - AlexCoventry 5y ago
    Can you give an example?
  - Philip-J-Fry 5y ago
    The only place I would agree with you is when someone accidentally shadows an `err` variable. Although, I'd probably attribute this more to shadowing than the error handling itself. Other than that, I can't think of any way the error handling subtly introduces bugs.
    - richardwhiuk 5y ago
      Go maps are wonderful examples. If you index into a map, then you get a tuple (val, err), where `err` is non nil if the value exists, and `val` is the value in the map or "the default construction of the value". Go also silently discards tuple members - e.g. `v = map[4]` is valid, and silently discards the error. This kind of means that all go maps can accidentally be defaultdicts (in Python terminology), but it's also an excellent way to introduce bugs without meaning to.
    - randomdata 5y ago
      The onus is on the programmer to understand what errors might be returned, which isn't always easily gleaned, especially when calling a function that calls other functions that call other functions, each pushing the error up the stack. The error interface does not provide much information on its own. The mistake of the programmer missing an error that should be handled in a particular situation could be considered a subtle bug, I suppose. This is different to languages that have compiler-enforced error handling, where if you forget to handle a certain case the code won't compile.
      - jerf 5y ago
        None of that is unique to Go, and, indeed, the question of "what errors may be returned" in general is not solved in any language that I know of satisfactorily, especially once you include any form of polymorphism. Compiler-enforced error handling in general only works on the direct possible errors from the function you just called, not the transitive closure of everything that could have happened, again, especially once you include any sort of polymorphism, meaning that you don't even know at the time you're writing the error handler what the possibilities are, and depending on the language, it may not even be possible to statically know. Don't mistake "option" or "sum types" for a solution to this problem; they aren't. (The only language I know where this is nominally "solved" is Java if you confine yourself to its checked exceptions, but nobody considers that solution "satisfactory", and indeed can be taken as a rough-and-ready proof that there may not even be a "static" solution to this problem that is usable. I'm beginning to think of it as "Errors don't compose, because they contain every possible pathology that would prevent values from composing." At this point, based on the consistent failures trying to create composable error solutions despite substantial effort poured into it across multiple languages, I am going to operate under the assumption there is no such solution until someone produces one.)
  - voidnullnil 5y ago
    Go's errors have basically all the problems of exception handling in previous generation languages (and a few novel ones). This explains it (tl;dr nobody bothers to formally define what errors they return, and the blindly propagate such unspecified errors, and it leads to ambiguity and unintentional program behavior) https://www.lainchan.org/%CE%BB/src/1612397614615.jpg https://www.lainchan.org/%CE%BB/src/1612397614615.jpg https://www.lainchan.org/%CE%BB/src/1612397701473.jpg https://www.lainchan.org/%CE%BB/src/1612397701473.jpg https://www.lainchan.org/%CE%BB/src/1612398029019.jpg https://www.lainchan.org/%CE%BB/src/1612398029019.jpg
    - astrange 5y ago
      Swift considered this and still decided to have untyped error returns. The issues are: - wrapping every underlying error in your own error type is not helpful - but defining every underlying error in your own type confuses your implementation (and especially dependencies) with your interface - and most errors can't be handled in code anyway There are a few errors (mostly in file operations) that are individually handled as normal parts of life, but otherwise they really are just there to send back up to the user. It's more important to know where an error can happen than what it is.
- throwaway894345 5y ago
  I agree with this. Channels have not lived up to their promises, and I mostly avoid them in Go. They're handy sometimes when you want to just block some goroutine (read from a channel that is never written to) or for a few other use cases, but mutexes are my go-to primitive.
- convolvatron 5y ago
  to me the overriding problem here is that selects need to be carefully refactored when you add in other interactions. of course. but this means there are as you say subtle non-local effects...and these kick in exactly when you don't want them - when you are trying to thread in a new feature in an existing codebase. I prefer to just ignore channels. then that statement makes sense :)
- eknkc 5y ago
  Whever I tried to use channels I regretted it at the end. I believe the channels in Go are the worst part of the language. I use mutexes.
  - tapirl 5y ago
    How could it be? Programming with channels is not only intuitive but fun for many cases: https://go101.org/article/channel-use-cases.html https://go101.org/article/channel-use-cases.html Yes, channels are not always best solutions for any case. There are cases mutexes are more suitable. Just choose the best solution for specified cases. Always sticking to one is not a good attitude.
    - jeffreybolle 5y ago
      I agree with this. We used to do a coding exercise at a previous job that was really well suited to mutexes and whenever someone tried to use channels they ended up with a much more complex solution. Equally, there are lots of cases which lend themselves to channels really well. Waiting on multiple things at once is hard to without channels and select for example.
  - nemo1618 5y ago
    I agree (well, maybe not the worst, but certainly one of the worst), but in retrospect it's hard to say whether Go would have taken off the way it did, had it not pushed the concurrency angle so hard. I think Go's focus on software engineering (as well as its "stubbornness" in general) is far more valuable than its concurrency features, but there's no question the latter is more exciting and likely to drive growth.
- tapirl 5y ago
  For some scenarios, the implementation code will be quite complicated if only mutexes are used, whereas the implementation will be quite simple by using channels. I think your frustration comes from you wanted to use channels for any scenarios. This is not recommended too. https://github.com/golang/go/wiki/MutexOrChannel https://github.com/golang/go/wiki/MutexOrChannel
  - karmakaze 5y ago
    It fails to mention the performance penalty with channels.
    - tapirl 5y ago
      For some scenarios channels and mutexes are both suitable for, mutextes are a bit more performant than channels. If you do care about the performance penalty, just choose mutexes. But the for scenarios mutextes are incapable of, still using mutexes is surely not a good idea.
      - karmakaze 5y ago
        This would be a good thing to add to the reference.
        jerf 5y ago
        Dunno. Tell people "channels are about 3 times slower than mutexes" and watch them scramble the hell out of their program trying to avoid channels, when the reality is "in general, both of them are fast enough that they will not be the bottleneck on your program". In general (not just Go!), don't send tiny amounts of work across any sort of internal boundary (goroutine, thread, spark, green thread, whatever they call it). The amount of work you send should significantly exceed the costs of sending it. Follow this advice and the perf differences between mutexes and channels will almost certainly not be relevant; fail to follow it and neither of them will be fast enough. I don't think there's actually a lot of programs in the wild where the performance difference between the two would be the make-or-break difference. Non-zero, but not many. In practice it's pretty clear developers will spend much more time worrying over it than is justified. (A common benchmark I see new programmers apply to any language that claims to be good at multithreading is to try to "parallelize" the act of adding a few million integers together by sending individual integers or addition problems out to threads, and wondering why it's 10-100 times slower even though all my CPUs are at 100%, so why does multithreading suck so hard in this language? The problem is that no matter how cheap the send operation is, it's going to be dozens or hundreds of operations, whereas a single integer addition is generally a single cycle, or possibly, amortized to less than that depending on how good your compiler is. You just need to be sure to send units of work larger than the cost of sending them, which is generally not that hard. This isn't just Go, I've seen this charge leveled at Haskell, Erlang, and Python's multiprocessing too, and I'm sure others have seen it in their own communities. It's a very common mistake.)
- whateveracct 5y ago
  > it can be closed ..at most once!
- voidfunc 5y ago
  Hard agree about channels being a mess in Go. I avoid channels as much as possible in favor of mutexes but the problem is the community is so aligned towards using channels you get funny looks when you solve problems with mutexes.
- laumars 5y ago
  It took me a loooong time to fully wrap my head around channels. They’re definitely not explicit and full of subtle yet devastating bugs when used inappropriately (which is all to easy to do too). Some of the boilerplate code around using them is just ugly too. That said, there are occasions when channels have proven invaluable. I think the real issue is they were branded as a killer feature but in reality their usefulness is a little more niche than a mutex.
  - elithrar 5y ago
    > I think the real issue is they were branded as a killer feature but in reality their usefulness is a little more niche than a mutex. Agree - and many people are surprised to learn both: a) how fast mutex ops are in practice, and b) how channels use mutexes under the covers.
    - tapirl 5y ago
      Mutexes and channels really overlap in some functionalities, but not totally overlap, okay?
  - chakkepolja 5y ago
    Correct me if I am wrong, isn't a channel basically same as blocking queues in Java? I know go but not much advanced.
    - eitland 5y ago
      Seems like there are a few differences but nothing I personally would leave normal indexing, kind of working generics, Maven and a choice of 3 world class IDEs behind for: https://stackoverflow.com/questions/10690403/go-channel-vs-java-blockingqueue https://stackoverflow.com/questions/10690403/go-channel-vs-j... To each their own though. I hear a lot of people prefer it.
    - clipradiowallet 5y ago
      I don't code Java, but I have the same sentiment about channels in golang to queue's in python.
- naikrovek 5y ago
  it always shocks me when someone says that channels are not immediately intuitive. they make so much sense to me, and they did from the instant I read about them. combined with goroutines, channels are absolutely one of the best features of this language, to me. async & await, on the other hand, still confuse me and trip me up today, even after using them for years, because of very weird implementation minutia and edge cases that I always seem to stumble into. I wish async and await would disappear from the face of the earth and I wish coroutines and channels were in every language I use.
  - throwaway894345 5y ago
    I've been using Go avidly since 2012. I still run into deadlocks and panics (writing to a closed channel) when I try to use channels beyond the simplest use cases. Error handling in a parallel context is also hairy with channels. I've learned to minimize my use of channels and prefer mutexes as my default concurrency primitive. It really only means that CSP looks nice on paper but (at least as implemented in Go) doesn't work out well in practice. There's lots to like about the language even if the CSP theory didn't pan out.
    - cp9 5y ago
      same, I like go and I like channels in theory but they are too primitive in practice. I am much more likely to use waitgroups and errgroups than anything with raw channels
    - itake 5y ago
      My understanding is that if you follow the rule of using generator functions (creator of the channel is responsible for writing and the closing) it’s impossible to write to a closed channel. When would that pattern not be useful?
      - gilgad13 5y ago
        The rule that the writer should be responsible for closing a channel is a good one to keep in mind, but it is often the case that you want to launch an indeterminate number of generators and collect the results from all of them. For example, you may want to make one connection to each server in the config, or to process each file in a directory in parallel. Because the arms of a `select{}` are determined at compile time, it cannot be used to select over the variable generator-owned channels, and you have four more difficult options: * Use `reflect.Value.Select`[1]. Having to reach for reflect feels ugly for such a common case, and the performance of the reflect-select is much lower than the native select. * Create a single channel owned by the reader, pass it to each writer, and arrange for this channel to be closed when the final writer exits, through a waitgroup. There is an example under "Parallel digestion" in the Go Concurrency Patterns blog post[2]. Note the little details to get right. We must launch a separate goroutine to monitor the waitgroup / channel closure. If we accidentally do it in-line at the wrong level, everything will work fine if the total number of items written to `c` is less than `c`'s capacity, but will hang once a worker becomes blocked on `c`. Additionally, the waitgroup is threaded directly into the writers, which may be more difficult if those are implemented in some other generic package. * Wrap the above pattern up into a `merge` function, such as the one under "Fan-in, Fan-out" in the Go Concurrency Patterns post[2]. The lack of generics means we will have to copy-paste this function everywhere we want to use it. Additionally, this launches a goroutine for every channel being watched, which strikes people as "expensive" for such a simple operation. * We can construct a function that takes two channels and launches a goroutine that selects between the two and writes to a merged output channel. By constructing a tree of these we can merge an arbitrary number of channels. This is really just an optimization of the above. None of these options are particularly intuitive. Too often I've instead seen developers create a single channel owned by the reader and either: * Assume it is never closed and the reader doesn't terminate until the application does * Rely on some external mechanism to know when to stop reading. If the reader can stop reading without confirming that the writers have stopped writing, this can lead to the writers becoming blocked on sending into this channel, which may prevent them from performing necessary cleanup actions (signaling `.Done()` on a waitgroup, for instance) that cause hangs in other areas. * Thread a cancellation ctx through every reader and writer. This ensures that nothing hangs, but can result in messages that are sitting in the the channels being dropped. If other areas of code have an assumption like, "every accepted request will receive a response", this can break that. In addition, many developers have a gut instinct to add some amount of buffering to their channels, which usually results in these backpressure / channel issues being papered over during low-load unit tests, only to rear their head during higher load integration tests or in production, when the debugging story is much more difficult. [1]: https://pkg.go.dev/reflect#Select https://pkg.go.dev/reflect#Select [2]: https://blog.golang.org/pipelines https://blog.golang.org/pipelines
- Thaxll 5y ago
  Channel are pipe if you want to use a mutex you have to re-invent a queue with a mutex which is very different, a mutext alone does not just replace a channel.
- ryanschneider 5y ago
  The thing that has bitten me the most is somewhat related: the lack of any uniform “deep copy” or immutability support. Sure I can make a `chan Foo` but if Foo is a struct with an embedded pointer in it you just aliased a pointer you probably didn’t mean to. If there was supported for optional value types rather than using pointers that would help too.
- azth 5y ago
  Not to mention that there is no way to ensure immutability when passing messages in channels, and the language doesn't help you there. This is a recipe for race conditions. Furthermore, you can easily have hierarchies of "goroutines", everything has to be painstakingly built from scratch and is very error prone. > Another aspect of Go being a useful programming environment is having well-defined semantics for the most common programming mistakes, ... Quoting Tony Hoare One of the most common programming mistakes of all, dubbed by Hoare himself as the "billion dollar mistake": null pointers, yet it was put into go without any consideration.
  - throwaway894345 5y ago
    > One of the most common programming mistakes of all, dubbed by Hoare himself as the "billion dollar mistake": null pointers, yet it was put into go without any consideration. I get the distinct impression that people who invoke this boilerplate Hoare quote argument don't understand that null pointer bugs are "a billion dollar mistake" because nullable pointers have been idiomatic in every major programming language for the last 40+ years, not because they are individually particularly expensive. Yeah, Rust-like enums would probably be strictly better than null pointers, but people talk about null pointers like they are going to doom your project when in reality they're mere papercuts--a small subset of all type errors--and there are whole projects that are written in fully dynamically typed languages! Indeed, there are worse problems than type errors in a language--you could have an impoverished ecosystem, poor tools (especially build tools), or abysmal performance, and many of the most popular programming languages have two of the three and null pointers. Go is fortunate to have only null pointers (papercuts) working against it.
    - voidnullnil 5y ago
      I've never bothered to look up the hoare thing, but null as a member of all types increases the amount of time spent studying any given API (and of course, there are lots of null pointer bugs from people who didn't bother to study the API sufficiently).
      - 5y ago
- parhamn 5y ago
  The main issue is channels are a very low level primitive (not far from mutexes and atomics). When generics come out we there will emerge a library that handles a lot of the common channel idioms that people are implementing (perhaps somewhat incorrectly). Thanks like fanouts, multidispatch, sinks, chaining, simple concurrency, etc will likely emerge in a lib in the future.
  - ra7 5y ago
    This definitely the biggest thing I'm looking forward to in Go's post-generics world. It's frustrating to write the same concurrency abstractions like fan out, error handling etc for every type and reason about channel behavior carefully every single time.
- skywhopper 5y ago
  He’s stating the goal of the design, not an assertion that they got everything right. Obviously there are lots of places where the design fell short. But the philosophy behind the approach to future design is unchanged. Channels were an attempt to add something very new to the language and so it’s not a surprise that the design ended up being problematic. I’m sure Russ has thoughts on that, too. But the vast majority of Go code doesn’t use channels. The fact that lots of people have trouble with them doesn’t change the fact that the vast majority of Go code does follow this principle.
- silasdavis 5y ago
  Exactungly, Go's flavour of channels are full of subtle bugs. I could count on N hands the number of times I've had to make a channel to drain a channel.
- saghm 5y ago
  The thing that always got me was that closing a channel on one end (I think the receiving end, but I can't remember exactly since I haven't programmed Go for a while) causes an error if the other side is open, but doing that on the other end (I think the front) doesn't give that issue. Sure, there's an explanation for that, but I feel like you could just as easily make a justification for acting consistently on both ends.
  - owl57 5y ago
    Don't know Go, but this way would be logical. "I've done sending" and "I refuse to handle more messages" aren't that symmetrical.
  - tene 5y ago
    In Go, sending a value through a closed channel panics the program, and receiving a value from a closed channel immediately returns the zero value for the channel (and a false, if you use the two-return-values form of the receive operator). There's no difference in what side closes the channel, I think, just in how a closed channel behaves when you try to send or receive on it. Personally, I think it's terrifying that the "easy" option just lies to you instead of crashing the program, but I kind of see how it fits with some of Go's other design decisions that I'd also have made differently (Option/Result instead of pervasive nullability).
- tschellenbach 5y ago
  You can write Go for years without ever touching channels. I agree though, they are extremely easy to make mistakes with.
- ferdowsi 5y ago
  I believe the guidance addresses this. > Programs that modify data being simultaneously accessed by multiple goroutines must serialize such access. > To serialize access, protect the data with channel operations or other synchronization primitives such as those in the sync and sync/atomic packages. Go gives us `sync.WaitGroup` and `errgroup.Group` as abstractions to manage concurrent operations. I always end up guiding developers towards using these unless they absolutely can't (like responding to OS signals requires interacting with channels).
  - pkaye 5y ago
    For OS signals now they added NotifyContext() which uses context instead of channels.
- voidnullnil 5y ago
  Channels and an alt operator are much more powerful than locks but eh I can't remember the edge cases in Go's channels so you might be right,
- axaxs 5y ago
  I find channels massively overcomplicate many things. Cool in theory, not so useful in practice. Anymore, I only use them as a semaphore to keep less than X routines running at once.
- AtNightWeCode 5y ago
  Is it not just basically an implicit actor model.
- sagichmal 5y ago
  > A channel can be buffered or unbuffered Yes, with predictable results. > it can be closed Yes, by the owner/sender, who then understands it as invalid. The effect of receiving on a closed channel is well-defined and predictable. > and there's different behavior of [recv/send] whether it's in a select or not I don't think that's true? What do you mean? > And you often end up with multiple channels as well Sure! That's part of their power. > Reviewing code with channels is often more confusing than code that uses mutexes I totally agree that goroutines and channels are often over-used by new Go programmers. If what you're trying to do is simply protect concurrent access to some shared state, a mutex is the far better choice.
candiddevmike 5y ago
Somewhat related, how are other gophers handling dynamic configuration in their go apps? Such as if the app is tied to consul or vault to get some config value. How do you handle the propagation of config changes cleanly, especially if it impacts something "big" like your DB pool? Do you have mutexes everywhere you read from the config pointer you pass around? I noodled on this a while ago and figured it was easier to just restart the app on config changes than try to change things on the fly =D
- spyspy 5y ago
  Usually http handlers or a database functions are pointer methods on some larger Service or DB struct type. If the config changes, you'd have a mutex within that struct and a method that safely modifies its state by locking and unlocking the mutex.
- Philip-J-Fry 5y ago
  I don't use any of them services, but any config that has multiple readers and a single writer (who updates it when it changes) can just be protected by a sync.RWMutex. The easiest thing to do would be to add a RWMutex to your Config struct. Then have public accessor methods which RLock the mutex while retrieving a field value. Then when you want to update the config have something like a `Set(c Config)` method which allows you to overwrite the config stored in pointer. This Locks the RWMutex for writing of course.
- maximente 5y ago
  viper, which nicely handles env vars, also can watch for config changes: https://github.com/spf13/viper#watching-and-re-reading-config-files https://github.com/spf13/viper#watching-and-re-reading-confi... if you're rotating creds or need to open/close the DB, this will typically just add another select case to your main method, where you also block on e.g. signal catching to cleanly shutdown the app
  - snupples 5y ago
    Be aware that viper isn't thread safe. You'd still have to copy config by value and then put a mutex on that, then handle reads and writes separately. https://github.com/spf13/viper#is-it-safe-to-concurrently-read-and-write-to-a-viper https://github.com/spf13/viper#is-it-safe-to-concurrently-re...
karmakaze 5y ago
> To serialize access, protect the data with channel operations or other synchronization primitives such as those in the sync and sync/atomic packages. > If you must read the rest of this document to understand the behavior of your program, you are being too clever. Don't be clever. I'm surprised that they list sync/atomic as a thing to use. As I recall its behaviour isn't even defined--I followed a long mailing list thread trying to find out that ended with 'these are the behaviours we know we want but it's too complicated to document so let's just keep this to ourselves'.
- rsc 5y ago
  Keep reading?
  - karmakaze 5y ago
    Yes, that was about the original Go documentation. The suggestion still doesn't mention that the difficultly with sync/atomic isn't the memory model per-se but rather the lack of yielding to another goroutine.
    - Groxx 5y ago
      Something like `runtime.Gosched()`? https://pkg.go.dev/runtime#Gosched https://pkg.go.dev/runtime#Gosched
- dragontamer 5y ago
  1. The compiler's #1 job is removing code (aka optimizing): it does this by rearranging code, merging code together, and other forms of optimizations. That is to say: "i=0; i++; i++" wants to optimize to "i=2;", but in a multithreaded context, it means that you'll "never" see "i==0", or "i==1". Turns out that "losing" these states can be an issue if you're building multithreaded primitives (lock-free code, atomics, etc. etc.) 2. The CPU and L1 cache also "effectively moves" code in relaxed-architectures like ARM / Power9 / DEC Alpha. So it turns out that #1 is true "even if the compiler wasn't involved". 3. Because of #2, might as well fix the compiler / CPU / L1 with the same set of primitives (the "memory model") that defines orderings. 4. Turns out that a large, but minority, number of programmers want to experiment with low-level multithreading primitives: researchers, speed-demon professionals, and others do want to go at "full speed" even at the cost of great complexity. Unifying the promises of the compiler + CPU + L1 cache all together in a SINGULAR model helps dramatically (that way: the programmer only fights the compiler. The compiler has to "translate" the CPU / L1 cache rearrangements into low-level barriers as appropriate) ------- It turns out that the biggest source of speed improvements exists in this realm of "rearranging code". That's why CPUs do out-of-order execution. That's why CPUs (like ARM or POWER9) want more relaxed execution, to allow the CPU to rearrange more code in more situations. That's why L1 cache exists (and similar: L1 wants to rearrange reads/writes even more aggressively for even faster operations). If you make a memory barrier (nominally preventing the L1 cache from rearranging code), you SHOULD have the CPU and compiler respect the barrier as well. After all, if the programmer says "this should not be rearranged", then that probably applies to compiler, CPU, and L1 cache all the same.
twotwotwo 5y ago
> Note that this means that races on multiword data structures can lead to inconsistent values not corresponding to a single write. When the values depend on the consistency of internal (pointer, length) or (pointer, type) pairs, as is the case for interface values, maps, slices, and strings in most Go implementations, such races can in turn lead to arbitrary memory corruption. That second sentence is one of the painful things about the Go memory model--Go is type and memory safe if you avoid explicitly unsafe things, except for data races on those structures. I wonder what you can practically do to mitigate that at this point; doesn't seem like you go back and change things to be like Java. Can there be some best-effort detection for races in those specific places that's fast enough to run in prod (like I think they did for maps)? I also recall some Go mailing-list post I can't find suggesting that an SSE 128-bit move, while not guaranteed not to tear, seemed to tear only rarely in their tests on the chips of the time; is that true and is there anything there? I imagine fully preventing the problem with atomic reads/writes of these pairs from the heap is a clear no-go performance-wise, and might involve ABI-changing alignment guarantees. [Edit: SSE2 behavior is discussed here https://stackoverflow.com/questions/7646018/sse-instructions-which-cpus-can-do-atomic-16b-memory-operations https://stackoverflow.com/questions/7646018/sse-instructions... which is mentioned in a comment at https://research.swtch.com/gorace https://research.swtch.com/gorace which seemed to inspire it. Chance of a race seemed to depend on CPU model at the time, bad on the old Core Duo, better on some server chips another answerer tried. Good on my laptop FWIW!] Probably hard for anything to happen here because any performance regression is hard to swallow, the race detector is close enough for many, and anything you do won't be a 100% solution anyway. Which is a bummer.
- ngrilly 5y ago
  > doesn't seem like you go back and change things to be like Java Java is like Go in that regard. That's why you have to use ConcurrentHashMap instead of HashMap for instance.
  - deleted 5y ago
    [deleted]
  - ibraheemdev 5y ago
draw_down 5y ago
I wish some of the chapter-and-verse about “obviousness” that gets constantly recited with regard to the language was also applied to the tooling.
none_to_remain 5y ago
I just appreciate that the Go Memory Model intro basically says "if you have to read this, you are probably wrong" and felt free to skip the rest.
- kccqzy 5y ago
  What is there to appreciate from that? That sentence is basically saying "you're too dumb to understand and use this correctly so don't bother." I find it offensive, and so characteristic of Go creators' attitude toward Go programmers.
  - rsc 5y ago
    From the start we've been building the language we want to use, ourselves. And we do use it and are happy to use it. "You're too dumb" is absolutely not our attitude toward Go programmers. I would find that offensive too. Also, if we really believed people were too dumb to understand these things I wouldn't have written so much text to try to bring people along for the decision. I'd have just put in the text I wanted and walked away.
    - kccqzy 5y ago
      Okay. Then I guess matter is just the wording then. I find the Rustonomicon warning much more inviting: > Should you wish a long and happy career of writing Rust programs, you should turn back now and forget you ever saw this book. It is not necessary. However if you intend to write unsafe code—or just want to dig into the guts of the language—this book contains lots of useful information. What I like: (1) the advice not to read this document is presented using the word "should," but in Go's case, the doc first presumes the reader as being too clever and then directly commands the reader not to be too clever; (2) the Rust version acknowledges a basic human trait that is curiosity, inviting those curious enough to explore the guts of the language to dig in, but the Go version does not. I'm sure you have great intention, but can you acknowledge the wording comes across as off-putting and condescending?
kubb 5y ago
Am I the only one who finds Go concurrency model to be extremely bug prone and difficult to work with? Writing an efficient pipeline of goroutines connected with channels that doesn't deadlock is a verbose and subtle mess with a whole bunch of code duplication. For a lot of the things that I want to achieve, functional abstractions like parallel map and reduce would be much more understandable, and easier to use. That whole talk about philosophy distracts from the actual tools that Go gives you to manage concurrency which in my opinion can be improved a lot.
- Thaxll 5y ago
  "Writing an efficient pipeline of goroutines connected with channels that doesn't deadlock is a verbose and subtle mess with a whole bunch of code duplication." It takes 30 lines of code and it's very easy.
  - 1ris 5y ago
    This has to be depended on the pipeline you want to have.
    - Thaxll 5y ago
      It depends but saying it's very complicated and it's easy to deadlock is pure bs. Where is the complexity in Go to have a pool of workers looking for jobs on a channel exactly?
      - ackfoobar 5y ago
        > Where is the complexity in Go to have a pool of workers looking for jobs on a channel exactly? This is just one kind of pipeline. If this is all you have done, you may have the impression that concurrency (in Go) is easy. BTW, this is worth reading: Understanding Real-World Concurrency Bugs in Go https://songlh.github.io/paper/go-study.pdf https://songlh.github.io/paper/go-study.pdf
        5y ago
yoursunny 5y ago
If I have a uint64 counter incremented by C code and read by Go code, is it safe to do so without using any locks and atomic?
- tapirl 5y ago
  Surely not safe.
knorker 5y ago
Good example of the Go patronizing tone. "We're not C++, with its garbage UB. Oh, but we have memory corruption abilities from data races". Oh, right. Yes, i know C++ has more interesting failures for invalid code, but it's pretty misrepresenting to say "Go sits somewhere in the middle", as if you can be a little bit pregnant. Other than that, yeah seems good, but not earth shattering incremental improvement.
- throwaway894345 5y ago
  Go’s UB is in a rare edge case. I’ve been writing Go for a decade and I’ve never run into it. No doubt some have, but this is in stark contrast to C++ where UB is all over the place. The salient detail is that UB isn’t a binary like you propose. It’s not like “am I pregnant”, it’s more like “how much alcohol have I consumed”?
  - knorker 5y ago
    The tone of the article says otherwise. It takes great detours to not call what Go has UB. It's exceptions all over again.
    - throwaway894345 5y ago
      The point isn’t whether or not it’s UB. The point is that having a little bit of UB in a rare edge case is not ideal but most programmers will never run into it. While in C++ it’s so prevalent that it will affect every programmer and lead to many difficult to troubleshoot bugs. These things are not equivalent problems.
tialaramex 5y ago
I think Go was wise to begin by telling programmers if they care about these details they're being too clever and will regret it. Detailed memory models so far over-promise and under-deliver and the main justification for them is trying to squeeze out that last few percent of performance which is not what Go is for. However: > The first, exemplified by C and C++, is that programs with data races are invalid: a compiler may break them in arbitrarily surprising ways. The problem isn't just the compiler. If you write code with a data race the CPU might break your program in arbitrarily surprising ways too. Go programs don't rely on your C++ compiler, but they do run on your CPU and not a virtual machine. Is Go really going to be able to deliver defined semantics here in the face of that? > The second, exemplified by Java and JavaScript, is that programs with data races have defined semantics, limiting the possible impact of a race and making programs more reliable and easier to debug. This was definitely the intention. But, do the results match that intention? It seems like in fact Java programmers are still confused by what happened when there's a data race, even though the semantics of a data race in Java are more limited, they're beyond the conception of the programmer anyway. Do you go less insane if you see only two Great Old Ones than if you were to see a host of them? Or does the distinction not make a difference? Later this document proposes the new Go model should say: > such races can in turn lead to arbitrary memory corruption. That's a much worse outcome than in Java. So I'd be astonished if Go programmers are able to meaningfully debug a program in which a data race has smashed unrelated data structures. In fact, I'm dubious as to what difference is even left between "arbitrary memory corruption" and C++ undefined behaviour. Experts will sometimes draw on their experience to guess correctly what went wrong. Specialist instrumentation might help find your root cause. But these are the same in C++
- yc12340 5y ago
  > Java programmers are still confused by what happened when there's a data race IMO, the most important thing is that JVM runtime isn't being confused into executing attacker's code. Most programmers are unable to precisely predict behaviour of their programs anyway, — they get accustomed to being confused.
  - astrange 5y ago

248 points by gbrown_ 5y ago

> In other words, Go encourages avoiding subtle bugs by avoiding subtle code. I find this philosophy strange when they talk about the communication primitives. Channels are extremely subtle in my experience. A channel can be buffered or unbuffered, it can be closed, and there’s different behavior of read/write whether it’s in a select or not. And then you often end up with multiple channels as well. Reviewing code with channels is often more confusing than code that uses mutexes in my opinion.

richardwhiuk 5y ago
I agree - I find Go's error handling encourages subtle bugs.
- AlexCoventry 5y ago
  Can you give an example?
- Philip-J-Fry 5y ago
  The only place I would agree with you is when someone accidentally shadows an `err` variable. Although, I'd probably attribute this more to shadowing than the error handling itself. Other than that, I can't think of any way the error handling subtly introduces bugs.
  - richardwhiuk 5y ago
    Go maps are wonderful examples. If you index into a map, then you get a tuple (val, err), where `err` is non nil if the value exists, and `val` is the value in the map or "the default construction of the value". Go also silently discards tuple members - e.g. `v = map[4]` is valid, and silently discards the error. This kind of means that all go maps can accidentally be defaultdicts (in Python terminology), but it's also an excellent way to introduce bugs without meaning to.
  - randomdata 5y ago
    The onus is on the programmer to understand what errors might be returned, which isn't always easily gleaned, especially when calling a function that calls other functions that call other functions, each pushing the error up the stack. The error interface does not provide much information on its own. The mistake of the programmer missing an error that should be handled in a particular situation could be considered a subtle bug, I suppose. This is different to languages that have compiler-enforced error handling, where if you forget to handle a certain case the code won't compile.
    - jerf 5y ago
      None of that is unique to Go, and, indeed, the question of "what errors may be returned" in general is not solved in any language that I know of satisfactorily, especially once you include any form of polymorphism. Compiler-enforced error handling in general only works on the direct possible errors from the function you just called, not the transitive closure of everything that could have happened, again, especially once you include any sort of polymorphism, meaning that you don't even know at the time you're writing the error handler what the possibilities are, and depending on the language, it may not even be possible to statically know. Don't mistake "option" or "sum types" for a solution to this problem; they aren't. (The only language I know where this is nominally "solved" is Java if you confine yourself to its checked exceptions, but nobody considers that solution "satisfactory", and indeed can be taken as a rough-and-ready proof that there may not even be a "static" solution to this problem that is usable. I'm beginning to think of it as "Errors don't compose, because they contain every possible pathology that would prevent values from composing." At this point, based on the consistent failures trying to create composable error solutions despite substantial effort poured into it across multiple languages, I am going to operate under the assumption there is no such solution until someone produces one.)
- voidnullnil 5y ago
  Go's errors have basically all the problems of exception handling in previous generation languages (and a few novel ones). This explains it (tl;dr nobody bothers to formally define what errors they return, and the blindly propagate such unspecified errors, and it leads to ambiguity and unintentional program behavior) https://www.lainchan.org/%CE%BB/src/1612397614615.jpg https://www.lainchan.org/%CE%BB/src/1612397614615.jpg https://www.lainchan.org/%CE%BB/src/1612397701473.jpg https://www.lainchan.org/%CE%BB/src/1612397701473.jpg https://www.lainchan.org/%CE%BB/src/1612398029019.jpg https://www.lainchan.org/%CE%BB/src/1612398029019.jpg
  - astrange 5y ago
    Swift considered this and still decided to have untyped error returns. The issues are: - wrapping every underlying error in your own error type is not helpful - but defining every underlying error in your own type confuses your implementation (and especially dependencies) with your interface - and most errors can't be handled in code anyway There are a few errors (mostly in file operations) that are individually handled as normal parts of life, but otherwise they really are just there to send back up to the user. It's more important to know where an error can happen than what it is.
throwaway894345 5y ago
I agree with this. Channels have not lived up to their promises, and I mostly avoid them in Go. They're handy sometimes when you want to just block some goroutine (read from a channel that is never written to) or for a few other use cases, but mutexes are my go-to primitive.
convolvatron 5y ago
to me the overriding problem here is that selects need to be carefully refactored when you add in other interactions. of course. but this means there are as you say subtle non-local effects...and these kick in exactly when you don't want them - when you are trying to thread in a new feature in an existing codebase. I prefer to just ignore channels. then that statement makes sense :)
eknkc 5y ago
Whever I tried to use channels I regretted it at the end. I believe the channels in Go are the worst part of the language. I use mutexes.
- tapirl 5y ago
  How could it be? Programming with channels is not only intuitive but fun for many cases: https://go101.org/article/channel-use-cases.html https://go101.org/article/channel-use-cases.html Yes, channels are not always best solutions for any case. There are cases mutexes are more suitable. Just choose the best solution for specified cases. Always sticking to one is not a good attitude.
  - jeffreybolle 5y ago
    I agree with this. We used to do a coding exercise at a previous job that was really well suited to mutexes and whenever someone tried to use channels they ended up with a much more complex solution. Equally, there are lots of cases which lend themselves to channels really well. Waiting on multiple things at once is hard to without channels and select for example.
- nemo1618 5y ago
  I agree (well, maybe not the worst, but certainly one of the worst), but in retrospect it's hard to say whether Go would have taken off the way it did, had it not pushed the concurrency angle so hard. I think Go's focus on software engineering (as well as its "stubbornness" in general) is far more valuable than its concurrency features, but there's no question the latter is more exciting and likely to drive growth.
tapirl 5y ago
For some scenarios, the implementation code will be quite complicated if only mutexes are used, whereas the implementation will be quite simple by using channels. I think your frustration comes from you wanted to use channels for any scenarios. This is not recommended too. https://github.com/golang/go/wiki/MutexOrChannel https://github.com/golang/go/wiki/MutexOrChannel
- karmakaze 5y ago
  It fails to mention the performance penalty with channels.
  - tapirl 5y ago
    For some scenarios channels and mutexes are both suitable for, mutextes are a bit more performant than channels. If you do care about the performance penalty, just choose mutexes. But the for scenarios mutextes are incapable of, still using mutexes is surely not a good idea.
    - karmakaze 5y ago
      This would be a good thing to add to the reference.
      - jerf 5y ago
        Dunno. Tell people "channels are about 3 times slower than mutexes" and watch them scramble the hell out of their program trying to avoid channels, when the reality is "in general, both of them are fast enough that they will not be the bottleneck on your program". In general (not just Go!), don't send tiny amounts of work across any sort of internal boundary (goroutine, thread, spark, green thread, whatever they call it). The amount of work you send should significantly exceed the costs of sending it. Follow this advice and the perf differences between mutexes and channels will almost certainly not be relevant; fail to follow it and neither of them will be fast enough. I don't think there's actually a lot of programs in the wild where the performance difference between the two would be the make-or-break difference. Non-zero, but not many. In practice it's pretty clear developers will spend much more time worrying over it than is justified. (A common benchmark I see new programmers apply to any language that claims to be good at multithreading is to try to "parallelize" the act of adding a few million integers together by sending individual integers or addition problems out to threads, and wondering why it's 10-100 times slower even though all my CPUs are at 100%, so why does multithreading suck so hard in this language? The problem is that no matter how cheap the send operation is, it's going to be dozens or hundreds of operations, whereas a single integer addition is generally a single cycle, or possibly, amortized to less than that depending on how good your compiler is. You just need to be sure to send units of work larger than the cost of sending them, which is generally not that hard. This isn't just Go, I've seen this charge leveled at Haskell, Erlang, and Python's multiprocessing too, and I'm sure others have seen it in their own communities. It's a very common mistake.)
whateveracct 5y ago
> it can be closed ..at most once!
voidfunc 5y ago
Hard agree about channels being a mess in Go. I avoid channels as much as possible in favor of mutexes but the problem is the community is so aligned towards using channels you get funny looks when you solve problems with mutexes.
laumars 5y ago
It took me a loooong time to fully wrap my head around channels. They’re definitely not explicit and full of subtle yet devastating bugs when used inappropriately (which is all to easy to do too). Some of the boilerplate code around using them is just ugly too. That said, there are occasions when channels have proven invaluable. I think the real issue is they were branded as a killer feature but in reality their usefulness is a little more niche than a mutex.
- elithrar 5y ago
  > I think the real issue is they were branded as a killer feature but in reality their usefulness is a little more niche than a mutex. Agree - and many people are surprised to learn both: a) how fast mutex ops are in practice, and b) how channels use mutexes under the covers.
  - tapirl 5y ago
    Mutexes and channels really overlap in some functionalities, but not totally overlap, okay?
- chakkepolja 5y ago
  Correct me if I am wrong, isn't a channel basically same as blocking queues in Java? I know go but not much advanced.
  - eitland 5y ago
    Seems like there are a few differences but nothing I personally would leave normal indexing, kind of working generics, Maven and a choice of 3 world class IDEs behind for: https://stackoverflow.com/questions/10690403/go-channel-vs-java-blockingqueue https://stackoverflow.com/questions/10690403/go-channel-vs-j... To each their own though. I hear a lot of people prefer it.
  - clipradiowallet 5y ago
    I don't code Java, but I have the same sentiment about channels in golang to queue's in python.
naikrovek 5y ago
it always shocks me when someone says that channels are not immediately intuitive. they make so much sense to me, and they did from the instant I read about them. combined with goroutines, channels are absolutely one of the best features of this language, to me. async & await, on the other hand, still confuse me and trip me up today, even after using them for years, because of very weird implementation minutia and edge cases that I always seem to stumble into. I wish async and await would disappear from the face of the earth and I wish coroutines and channels were in every language I use.
- throwaway894345 5y ago
  I've been using Go avidly since 2012. I still run into deadlocks and panics (writing to a closed channel) when I try to use channels beyond the simplest use cases. Error handling in a parallel context is also hairy with channels. I've learned to minimize my use of channels and prefer mutexes as my default concurrency primitive. It really only means that CSP looks nice on paper but (at least as implemented in Go) doesn't work out well in practice. There's lots to like about the language even if the CSP theory didn't pan out.
  - cp9 5y ago
    same, I like go and I like channels in theory but they are too primitive in practice. I am much more likely to use waitgroups and errgroups than anything with raw channels
  - itake 5y ago
    My understanding is that if you follow the rule of using generator functions (creator of the channel is responsible for writing and the closing) it’s impossible to write to a closed channel. When would that pattern not be useful?
    - gilgad13 5y ago
      The rule that the writer should be responsible for closing a channel is a good one to keep in mind, but it is often the case that you want to launch an indeterminate number of generators and collect the results from all of them. For example, you may want to make one connection to each server in the config, or to process each file in a directory in parallel. Because the arms of a `select{}` are determined at compile time, it cannot be used to select over the variable generator-owned channels, and you have four more difficult options: * Use `reflect.Value.Select`[1]. Having to reach for reflect feels ugly for such a common case, and the performance of the reflect-select is much lower than the native select. * Create a single channel owned by the reader, pass it to each writer, and arrange for this channel to be closed when the final writer exits, through a waitgroup. There is an example under "Parallel digestion" in the Go Concurrency Patterns blog post[2]. Note the little details to get right. We must launch a separate goroutine to monitor the waitgroup / channel closure. If we accidentally do it in-line at the wrong level, everything will work fine if the total number of items written to `c` is less than `c`'s capacity, but will hang once a worker becomes blocked on `c`. Additionally, the waitgroup is threaded directly into the writers, which may be more difficult if those are implemented in some other generic package. * Wrap the above pattern up into a `merge` function, such as the one under "Fan-in, Fan-out" in the Go Concurrency Patterns post[2]. The lack of generics means we will have to copy-paste this function everywhere we want to use it. Additionally, this launches a goroutine for every channel being watched, which strikes people as "expensive" for such a simple operation. * We can construct a function that takes two channels and launches a goroutine that selects between the two and writes to a merged output channel. By constructing a tree of these we can merge an arbitrary number of channels. This is really just an optimization of the above. None of these options are particularly intuitive. Too often I've instead seen developers create a single channel owned by the reader and either: * Assume it is never closed and the reader doesn't terminate until the application does * Rely on some external mechanism to know when to stop reading. If the reader can stop reading without confirming that the writers have stopped writing, this can lead to the writers becoming blocked on sending into this channel, which may prevent them from performing necessary cleanup actions (signaling `.Done()` on a waitgroup, for instance) that cause hangs in other areas. * Thread a cancellation ctx through every reader and writer. This ensures that nothing hangs, but can result in messages that are sitting in the the channels being dropped. If other areas of code have an assumption like, "every accepted request will receive a response", this can break that. In addition, many developers have a gut instinct to add some amount of buffering to their channels, which usually results in these backpressure / channel issues being papered over during low-load unit tests, only to rear their head during higher load integration tests or in production, when the debugging story is much more difficult. [1]: https://pkg.go.dev/reflect#Select https://pkg.go.dev/reflect#Select [2]: https://blog.golang.org/pipelines https://blog.golang.org/pipelines
Thaxll 5y ago
Channel are pipe if you want to use a mutex you have to re-invent a queue with a mutex which is very different, a mutext alone does not just replace a channel.
ryanschneider 5y ago
The thing that has bitten me the most is somewhat related: the lack of any uniform “deep copy” or immutability support. Sure I can make a `chan Foo` but if Foo is a struct with an embedded pointer in it you just aliased a pointer you probably didn’t mean to. If there was supported for optional value types rather than using pointers that would help too.
azth 5y ago
Not to mention that there is no way to ensure immutability when passing messages in channels, and the language doesn't help you there. This is a recipe for race conditions. Furthermore, you can easily have hierarchies of "goroutines", everything has to be painstakingly built from scratch and is very error prone. > Another aspect of Go being a useful programming environment is having well-defined semantics for the most common programming mistakes, ... Quoting Tony Hoare One of the most common programming mistakes of all, dubbed by Hoare himself as the "billion dollar mistake": null pointers, yet it was put into go without any consideration.
- throwaway894345 5y ago
  > One of the most common programming mistakes of all, dubbed by Hoare himself as the "billion dollar mistake": null pointers, yet it was put into go without any consideration. I get the distinct impression that people who invoke this boilerplate Hoare quote argument don't understand that null pointer bugs are "a billion dollar mistake" because nullable pointers have been idiomatic in every major programming language for the last 40+ years, not because they are individually particularly expensive. Yeah, Rust-like enums would probably be strictly better than null pointers, but people talk about null pointers like they are going to doom your project when in reality they're mere papercuts--a small subset of all type errors--and there are whole projects that are written in fully dynamically typed languages! Indeed, there are worse problems than type errors in a language--you could have an impoverished ecosystem, poor tools (especially build tools), or abysmal performance, and many of the most popular programming languages have two of the three and null pointers. Go is fortunate to have only null pointers (papercuts) working against it.
  - voidnullnil 5y ago
    I've never bothered to look up the hoare thing, but null as a member of all types increases the amount of time spent studying any given API (and of course, there are lots of null pointer bugs from people who didn't bother to study the API sufficiently).
    - 5y ago
parhamn 5y ago
The main issue is channels are a very low level primitive (not far from mutexes and atomics). When generics come out we there will emerge a library that handles a lot of the common channel idioms that people are implementing (perhaps somewhat incorrectly). Thanks like fanouts, multidispatch, sinks, chaining, simple concurrency, etc will likely emerge in a lib in the future.
- ra7 5y ago
  This definitely the biggest thing I'm looking forward to in Go's post-generics world. It's frustrating to write the same concurrency abstractions like fan out, error handling etc for every type and reason about channel behavior carefully every single time.
skywhopper 5y ago
He’s stating the goal of the design, not an assertion that they got everything right. Obviously there are lots of places where the design fell short. But the philosophy behind the approach to future design is unchanged. Channels were an attempt to add something very new to the language and so it’s not a surprise that the design ended up being problematic. I’m sure Russ has thoughts on that, too. But the vast majority of Go code doesn’t use channels. The fact that lots of people have trouble with them doesn’t change the fact that the vast majority of Go code does follow this principle.
silasdavis 5y ago
Exactungly, Go's flavour of channels are full of subtle bugs. I could count on N hands the number of times I've had to make a channel to drain a channel.
saghm 5y ago
The thing that always got me was that closing a channel on one end (I think the receiving end, but I can't remember exactly since I haven't programmed Go for a while) causes an error if the other side is open, but doing that on the other end (I think the front) doesn't give that issue. Sure, there's an explanation for that, but I feel like you could just as easily make a justification for acting consistently on both ends.
- owl57 5y ago
  Don't know Go, but this way would be logical. "I've done sending" and "I refuse to handle more messages" aren't that symmetrical.
- tene 5y ago
  In Go, sending a value through a closed channel panics the program, and receiving a value from a closed channel immediately returns the zero value for the channel (and a false, if you use the two-return-values form of the receive operator). There's no difference in what side closes the channel, I think, just in how a closed channel behaves when you try to send or receive on it. Personally, I think it's terrifying that the "easy" option just lies to you instead of crashing the program, but I kind of see how it fits with some of Go's other design decisions that I'd also have made differently (Option/Result instead of pervasive nullability).
tschellenbach 5y ago
You can write Go for years without ever touching channels. I agree though, they are extremely easy to make mistakes with.
ferdowsi 5y ago
I believe the guidance addresses this. > Programs that modify data being simultaneously accessed by multiple goroutines must serialize such access. > To serialize access, protect the data with channel operations or other synchronization primitives such as those in the sync and sync/atomic packages. Go gives us `sync.WaitGroup` and `errgroup.Group` as abstractions to manage concurrent operations. I always end up guiding developers towards using these unless they absolutely can't (like responding to OS signals requires interacting with channels).
- pkaye 5y ago
  For OS signals now they added NotifyContext() which uses context instead of channels.
voidnullnil 5y ago
Channels and an alt operator are much more powerful than locks but eh I can't remember the edge cases in Go's channels so you might be right,
axaxs 5y ago
I find channels massively overcomplicate many things. Cool in theory, not so useful in practice. Anymore, I only use them as a semaphore to keep less than X routines running at once.
AtNightWeCode 5y ago
Is it not just basically an implicit actor model.
sagichmal 5y ago
> A channel can be buffered or unbuffered Yes, with predictable results. > it can be closed Yes, by the owner/sender, who then understands it as invalid. The effect of receiving on a closed channel is well-defined and predictable. > and there's different behavior of [recv/send] whether it's in a select or not I don't think that's true? What do you mean? > And you often end up with multiple channels as well Sure! That's part of their power. > Reviewing code with channels is often more confusing than code that uses mutexes I totally agree that goroutines and channels are often over-used by new Go programmers. If what you're trying to do is simply protect concurrent access to some shared state, a mutex is the far better choice.

candiddevmike 5y ago

Somewhat related, how are other gophers handling dynamic configuration in their go apps? Such as if the app is tied to consul or vault to get some config value. How do you handle the propagation of config changes cleanly, especially if it impacts something "big" like your DB pool? Do you have mutexes everywhere you read from the config pointer you pass around? I noodled on this a while ago and figured it was easier to just restart the app on config changes than try to change things on the fly =D

spyspy 5y ago
Usually http handlers or a database functions are pointer methods on some larger Service or DB struct type. If the config changes, you'd have a mutex within that struct and a method that safely modifies its state by locking and unlocking the mutex.
Philip-J-Fry 5y ago
I don't use any of them services, but any config that has multiple readers and a single writer (who updates it when it changes) can just be protected by a sync.RWMutex. The easiest thing to do would be to add a RWMutex to your Config struct. Then have public accessor methods which RLock the mutex while retrieving a field value. Then when you want to update the config have something like a `Set(c Config)` method which allows you to overwrite the config stored in pointer. This Locks the RWMutex for writing of course.
maximente 5y ago
viper, which nicely handles env vars, also can watch for config changes: https://github.com/spf13/viper#watching-and-re-reading-config-files https://github.com/spf13/viper#watching-and-re-reading-confi... if you're rotating creds or need to open/close the DB, this will typically just add another select case to your main method, where you also block on e.g. signal catching to cleanly shutdown the app
- snupples 5y ago
  Be aware that viper isn't thread safe. You'd still have to copy config by value and then put a mutex on that, then handle reads and writes separately. https://github.com/spf13/viper#is-it-safe-to-concurrently-read-and-write-to-a-viper https://github.com/spf13/viper#is-it-safe-to-concurrently-re...

karmakaze 5y ago

> To serialize access, protect the data with channel operations or other synchronization primitives such as those in the sync and sync/atomic packages. > If you must read the rest of this document to understand the behavior of your program, you are being too clever. Don't be clever. I'm surprised that they list sync/atomic as a thing to use. As I recall its behaviour isn't even defined--I followed a long mailing list thread trying to find out that ended with 'these are the behaviours we know we want but it's too complicated to document so let's just keep this to ourselves'.

rsc 5y ago
Keep reading?
- karmakaze 5y ago
  Yes, that was about the original Go documentation. The suggestion still doesn't mention that the difficultly with sync/atomic isn't the memory model per-se but rather the lack of yielding to another goroutine.
  - Groxx 5y ago
    Something like `runtime.Gosched()`? https://pkg.go.dev/runtime#Gosched https://pkg.go.dev/runtime#Gosched
dragontamer 5y ago
1. The compiler's #1 job is removing code (aka optimizing): it does this by rearranging code, merging code together, and other forms of optimizations. That is to say: "i=0; i++; i++" wants to optimize to "i=2;", but in a multithreaded context, it means that you'll "never" see "i==0", or "i==1". Turns out that "losing" these states can be an issue if you're building multithreaded primitives (lock-free code, atomics, etc. etc.) 2. The CPU and L1 cache also "effectively moves" code in relaxed-architectures like ARM / Power9 / DEC Alpha. So it turns out that #1 is true "even if the compiler wasn't involved". 3. Because of #2, might as well fix the compiler / CPU / L1 with the same set of primitives (the "memory model") that defines orderings. 4. Turns out that a large, but minority, number of programmers want to experiment with low-level multithreading primitives: researchers, speed-demon professionals, and others do want to go at "full speed" even at the cost of great complexity. Unifying the promises of the compiler + CPU + L1 cache all together in a SINGULAR model helps dramatically (that way: the programmer only fights the compiler. The compiler has to "translate" the CPU / L1 cache rearrangements into low-level barriers as appropriate) ------- It turns out that the biggest source of speed improvements exists in this realm of "rearranging code". That's why CPUs do out-of-order execution. That's why CPUs (like ARM or POWER9) want more relaxed execution, to allow the CPU to rearrange more code in more situations. That's why L1 cache exists (and similar: L1 wants to rearrange reads/writes even more aggressively for even faster operations). If you make a memory barrier (nominally preventing the L1 cache from rearranging code), you SHOULD have the CPU and compiler respect the barrier as well. After all, if the programmer says "this should not be rearranged", then that probably applies to compiler, CPU, and L1 cache all the same.

twotwotwo 5y ago

> Note that this means that races on multiword data structures can lead to inconsistent values not corresponding to a single write. When the values depend on the consistency of internal (pointer, length) or (pointer, type) pairs, as is the case for interface values, maps, slices, and strings in most Go implementations, such races can in turn lead to arbitrary memory corruption. That second sentence is one of the painful things about the Go memory model--Go is type and memory safe if you avoid explicitly unsafe things, except for data races on those structures. I wonder what you can practically do to mitigate that at this point; doesn't seem like you go back and change things to be like Java. Can there be some best-effort detection for races in those specific places that's fast enough to run in prod (like I think they did for maps)? I also recall some Go mailing-list post I can't find suggesting that an SSE 128-bit move, while not guaranteed not to tear, seemed to tear only rarely in their tests on the chips of the time; is that true and is there anything there? I imagine fully preventing the problem with atomic reads/writes of these pairs from the heap is a clear no-go performance-wise, and might involve ABI-changing alignment guarantees. [Edit: SSE2 behavior is discussed here https://stackoverflow.com/questions/7646018/sse-instructions-which-cpus-can-do-atomic-16b-memory-operations https://stackoverflow.com/questions/7646018/sse-instructions... which is mentioned in a comment at https://research.swtch.com/gorace https://research.swtch.com/gorace which seemed to inspire it. Chance of a race seemed to depend on CPU model at the time, bad on the old Core Duo, better on some server chips another answerer tried. Good on my laptop FWIW!] Probably hard for anything to happen here because any performance regression is hard to swallow, the race detector is close enough for many, and anything you do won't be a 100% solution anyway. Which is a bummer.

ngrilly 5y ago
> doesn't seem like you go back and change things to be like Java Java is like Go in that regard. That's why you have to use ConcurrentHashMap instead of HashMap for instance.
- deleted 5y ago
  [deleted]
- ibraheemdev 5y ago

draw_down 5y ago

I wish some of the chapter-and-verse about “obviousness” that gets constantly recited with regard to the language was also applied to the tooling.

none_to_remain 5y ago

I just appreciate that the Go Memory Model intro basically says "if you have to read this, you are probably wrong" and felt free to skip the rest.

kccqzy 5y ago
What is there to appreciate from that? That sentence is basically saying "you're too dumb to understand and use this correctly so don't bother." I find it offensive, and so characteristic of Go creators' attitude toward Go programmers.
- rsc 5y ago
  From the start we've been building the language we want to use, ourselves. And we do use it and are happy to use it. "You're too dumb" is absolutely not our attitude toward Go programmers. I would find that offensive too. Also, if we really believed people were too dumb to understand these things I wouldn't have written so much text to try to bring people along for the decision. I'd have just put in the text I wanted and walked away.
  - kccqzy 5y ago
    Okay. Then I guess matter is just the wording then. I find the Rustonomicon warning much more inviting: > Should you wish a long and happy career of writing Rust programs, you should turn back now and forget you ever saw this book. It is not necessary. However if you intend to write unsafe code—or just want to dig into the guts of the language—this book contains lots of useful information. What I like: (1) the advice not to read this document is presented using the word "should," but in Go's case, the doc first presumes the reader as being too clever and then directly commands the reader not to be too clever; (2) the Rust version acknowledges a basic human trait that is curiosity, inviting those curious enough to explore the guts of the language to dig in, but the Go version does not. I'm sure you have great intention, but can you acknowledge the wording comes across as off-putting and condescending?

kubb 5y ago

Am I the only one who finds Go concurrency model to be extremely bug prone and difficult to work with? Writing an efficient pipeline of goroutines connected with channels that doesn't deadlock is a verbose and subtle mess with a whole bunch of code duplication. For a lot of the things that I want to achieve, functional abstractions like parallel map and reduce would be much more understandable, and easier to use. That whole talk about philosophy distracts from the actual tools that Go gives you to manage concurrency which in my opinion can be improved a lot.

Thaxll 5y ago
"Writing an efficient pipeline of goroutines connected with channels that doesn't deadlock is a verbose and subtle mess with a whole bunch of code duplication." It takes 30 lines of code and it's very easy.
- 1ris 5y ago
  This has to be depended on the pipeline you want to have.
  - Thaxll 5y ago
    It depends but saying it's very complicated and it's easy to deadlock is pure bs. Where is the complexity in Go to have a pool of workers looking for jobs on a channel exactly?
    - ackfoobar 5y ago
      > Where is the complexity in Go to have a pool of workers looking for jobs on a channel exactly? This is just one kind of pipeline. If this is all you have done, you may have the impression that concurrency (in Go) is easy. BTW, this is worth reading: Understanding Real-World Concurrency Bugs in Go https://songlh.github.io/paper/go-study.pdf https://songlh.github.io/paper/go-study.pdf
      - 5y ago

yoursunny 5y ago

If I have a uint64 counter incremented by C code and read by Go code, is it safe to do so without using any locks and atomic?

tapirl 5y ago
Surely not safe.

knorker 5y ago

Good example of the Go patronizing tone. "We're not C++, with its garbage UB. Oh, but we have memory corruption abilities from data races". Oh, right. Yes, i know C++ has more interesting failures for invalid code, but it's pretty misrepresenting to say "Go sits somewhere in the middle", as if you can be a little bit pregnant. Other than that, yeah seems good, but not earth shattering incremental improvement.

throwaway894345 5y ago
Go’s UB is in a rare edge case. I’ve been writing Go for a decade and I’ve never run into it. No doubt some have, but this is in stark contrast to C++ where UB is all over the place. The salient detail is that UB isn’t a binary like you propose. It’s not like “am I pregnant”, it’s more like “how much alcohol have I consumed”?
- knorker 5y ago
  The tone of the article says otherwise. It takes great detours to not call what Go has UB. It's exceptions all over again.
  - throwaway894345 5y ago
    The point isn’t whether or not it’s UB. The point is that having a little bit of UB in a rare edge case is not ideal but most programmers will never run into it. While in C++ it’s so prevalent that it will affect every programmer and lead to many difficult to troubleshoot bugs. These things are not equivalent problems.

tialaramex 5y ago

I think Go was wise to begin by telling programmers if they care about these details they're being too clever and will regret it. Detailed memory models so far over-promise and under-deliver and the main justification for them is trying to squeeze out that last few percent of performance which is not what Go is for. However: > The first, exemplified by C and C++, is that programs with data races are invalid: a compiler may break them in arbitrarily surprising ways. The problem isn't just the compiler. If you write code with a data race the CPU might break your program in arbitrarily surprising ways too. Go programs don't rely on your C++ compiler, but they do run on your CPU and not a virtual machine. Is Go really going to be able to deliver defined semantics here in the face of that? > The second, exemplified by Java and JavaScript, is that programs with data races have defined semantics, limiting the possible impact of a race and making programs more reliable and easier to debug. This was definitely the intention. But, do the results match that intention? It seems like in fact Java programmers are still confused by what happened when there's a data race, even though the semantics of a data race in Java are more limited, they're beyond the conception of the programmer anyway. Do you go less insane if you see only two Great Old Ones than if you were to see a host of them? Or does the distinction not make a difference? Later this document proposes the new Go model should say: > such races can in turn lead to arbitrary memory corruption. That's a much worse outcome than in Java. So I'd be astonished if Go programmers are able to meaningfully debug a program in which a data race has smashed unrelated data structures. In fact, I'm dubious as to what difference is even left between "arbitrary memory corruption" and C++ undefined behaviour. Experts will sometimes draw on their experience to guess correctly what went wrong. Specialist instrumentation might help find your root cause. But these are the same in C++

yc12340 5y ago
> Java programmers are still confused by what happened when there's a data race IMO, the most important thing is that JVM runtime isn't being confused into executing attacker's code. Most programmers are unable to precisely predict behaviour of their programs anyway, — they get accustomed to being confused.
- astrange 5y ago