9 ms·

The Beauty of Concurrency in Go

133 points by iand 13y ago

carbocation 13y ago
This is a nice article. I would encourage the author to use the term "goroutine" instead of "thread" when referring to Go's goroutines, because they aren't threads. The pros and cons of threads do not apply to goroutines. What's cool (for me) about this article is that the author knows this fact (and states so at the outset), but he re-discovers it and internalizes it over the course of his implementation.
StavrosK 13y ago
Ugh, I just finished writing the XMPP frontend for an XMPP/IRC bot I'm working on (http://www.getinstabot.com http://www.getinstabot.com). The frontends are in Go for concurrency, and ferry messages back and forth from the channels to the backend. Let me tell you, that problem is hard. Go coped pretty well, but the final thing is a mess of global states, and it's pretty elegant for what the problem is. I was hoping to avoid having many moving parts, but it ended up needing a lot of shared state between all processes. Some problems are just hard, and, no matter how well-designed the language is, they'll still be hard. Something I miss from the language after implementing that is the ability to, say, monitor one goroutine from another to see if it returns (so the former can return as well). I know it's possible with channels, but when one goroutine is blocked on network Recv(), there's not much of a chance to listen to channels. Anyway, yes, parallelism in Go is great, but not everything is magically all rainbows and unicorns (that's Django). Some problems will be messy and dirty, and even more so when you use channels.
- burntsushi 13y ago
  > I know it's possible with channels, but when one goroutine is blocked on network Recv(), there's not much of a chance to listen to channels. Could you elaborate a bit more on this point? This seems like a natural pattern for a channel/goroutine. Spin up a goroutine that reads from a connection and send whatever is read on a channel. Then other goroutines can synchronize on the channel.
  - StavrosK 13y ago
    Sure, but what if you want to shut that goroutine down later on? The best solution that I've found is to just close the connection, the .Recv() will error out immediately.
    - lobster_johnson 13y ago
      Yes, that is afaik the only way to abort a synchronous function call such as TCPConn.Read().
    - burntsushi 13y ago
      What other solution would you want? You could use `SetDeadline` if you like. But closing the connection seems reasonable too. I think you're framing the problem wrong. You're not asking to shutdown a goroutine, you're asking "How do I abort from a synchronous read from a network connection that is blocked?"
      - StavrosK 13y ago
        Don't get too hung up on that, it was the least of the problems. Closing the connection works fine, it's just a bit unorthodox. The basic structure of the problem was much more problematic to implement.
- pandemicsyn 13y ago
  heads up. Seems like https://getinstabot.com https://getinstabot.com and http://www.getinstabot.com http://www.getinstabot.com end up in a redirect loop.
  - StavrosK 13y ago
    Oh. I forgot about this comment and picked a great time to redirect everything over https. Deploying the fix now, thanks.
- NyxWulf 13y ago
  Observing other channels, or seeing when a channel does fire is what the select structure is for. It is one of the features that makes channels and goroutines very nice in go. http://golang.org/ref/spec#Select_statements http://golang.org/ref/spec#Select_statements http://blog.golang.org/2010/09/go-concurrency-patterns-timing-out-and.html http://blog.golang.org/2010/09/go-concurrency-patterns-timin... http://talks.golang.org/2012/concurrency.slide#1 http://talks.golang.org/2012/concurrency.slide#1 Video for the slides: http://www.youtube.com/watch?v=f6kdp27TYZs http://www.youtube.com/watch?v=f6kdp27TYZs
- lobster_johnson 13y ago
  > Something I miss from the language … Does sound like you want to use channels, and break your logic into small independent parts. You will have one goroutine using blocking Read() in a loop and feeding data to some channel. When it's done, you write to another channel that exists only for signaling: defer { doneChannel <- true } for { data := make([]byte, 65535) _, err := conn.Read(data) if err != nil { errorChannel <- err break; } dataChannel <- data } and then in your other goroutine: for { select { case data := <- dataChannel: // Handle data case <- doneChannel: // Other goroutine is done } } The only things shared here are the channels. The way to avoid too much state and moving parts is to break the problem into isolated, manageable parts that communicate with channels. Often you will have hierarchical relationships like this, where one piece of dumb code exists to pass data from something lower down to somewhere higher up. It's not hard, although some of the code gets a bit ugly and disjoint at times, especially in how anything synchronous has to use channels and goroutines. For example, today I wrote a simple worker pool implementation that runs a given function in parallel via goroutines and can adjust the number of workers dynamically at runtime. That function has to be declared not just as "func()" but as "func(abortChannel chan bool)", and the worker function has to honour the abort signal when it arrives from the pool. So channels do leak everywhere, even into APIs. (Yeah, I know I can use "chan struct{}" to avoid any storage, but I think "<- true" looks nicer than "<- struct{} {}".) What is harder is to intelligently handle complex cascading failures. That's what Erlang, with its supervisor tree, is good at. Go's goroutines are "fire and forget" and cannot even be terminated programmatically from elsewhere in the program.
  - StavrosK 13y ago
    Yeah, the channel approach was rewrite #3 :P It required me to have shared state in a horrible way as well, ruining the whole thing. There really isn't an elegant/functional way to do it, and sharing channels is error-prone itself. I ended up closing the connection to terminate goroutine B from A, and messaging on an already-shared channel to terminate A from B. Having cascading failures in Go would be fantastic. The problem, unfortunately, is not very amenable to elegant solutions. I might do a writeup at some point.
  - 13y ago
octo_t 13y ago
This has finally let me figure out what annoys me about Go, its a cargo-cult language. People saw that Erlang's Actors/processes were really popular and made it easy to write good, concurrent software. They then went away and implemented their own language with lightweight processes and message passing, but missed the fact that actors are the price you have to pay for the benefits of not sharing mutable data. And Go completely skipped that part (the most important part).
- sbov 13y ago
  Channels are basically queues with beefed up language support. Not sure what's so annoying about that considering it's a very common concurrency pattern across many languages.
- timclark 13y ago
  Doesn't go just implement Hoare's communicating sequential processes, as does Erlang? They share the same inspiration. You don't need to share state data between your goroutines if you don't want to either just like you don't have to use mnesia to share state between erlang processes if you don't want to. I don't think you can really accuse go of being a cargo cult language either, Rob Pike has implemented CSP multiple times (http://swtch.com/~rsc/thread/ http://swtch.com/~rsc/thread/).
  - ridiculous_fish 13y ago
    It is easy to accidentally share state between goroutines. For example, we wish to print out elements of a list: values := []string{"a", "b", "c"} for _, v := range values { go fmt.Println(v) } Each of these goroutines shares the same variable v, so this code contains a serious race condition.
    - Jabbles 13y ago
      It may be easy, but it's not that easy. values := []string{"a", "b", "c"} for _, v := range values { go func(){fmt.Println(v);}() } Does have a race condition. values := []string{"a", "b", "c"} for _, v := range values { go func(s string){fmt.Println(s);}(v) } Does not have a race condition.
codewright 13y ago
The beauty of concurrency in Clojure: ; Rough sketch: def defines a var (pretend it's a reference) ; @ is used to dereference the future and block to wait for the result. (def f (future (Thread/sleep 10000) (println "done") 100)) user=> @f done 100 ;; Dereferencing again will return the already calculated value. => @f 100 http://clojuredocs.org/clojure_core/clojure.core/future http://clojuredocs.org/clojure_core/clojure.core/future Edit: And more importantly, there are wrappers for the standard data structures designed around different concurrency use-cases (sync, async, coordinated, uncoordinated) Refs are for Coordinated Synchronous access to Many Identities". Atoms are for Uncoordinated synchronous access to a single Identity. Agents are for Uncoordinated asynchronous access to a single Identity. Vars are for thread local isolated identities with a shared default value. http://stackoverflow.com/questions/9132346/clojure-differences-between-ref-var-agent-atom-with-examples http://stackoverflow.com/questions/9132346/clojure-differenc... And you can use all (all!) of the Java concurrency tooling as desired, including raw threads (for which Clojure has a wrapper as well). Part of the reason I use Clojure rather than Go is because it doesn't try to force you into a one-size-fits-all method for handling concurrency. I have no problem with CSP but it doesn't fit everything I do. Sometimes I just want to defer work or wrap it in a future. Or I want to use an intelligent coordinated data structure rather than trying to meld flesh and bone to steel in order to make a concurrency-naive data structure behave how I want in a concurrent environ. If I can avoid those unnecessary battles, I will. So - Clojure.
- hornetblack 13y ago
  Futures in C++11 are similar: int x = std::async([]()->int{ return 100; }); std::cout << x.wait() << std::endl: Which isn't as nice as closure. Go could probably benfit from having a standard futures tool. I guess something along the lines of: type Future { Wait() interface{} } func newFuture(func interface{}, args ...interface{}) Future
  - cerales 13y ago
    Yep, futures can be done easily in Go with a goroutine that sends a single value on a channel, but: 1) you can't dereference the value multiple times in Go, so you have to manage saving it yourself, and 2) there's no syntactic sugar. Pity.
Jabbles 13y ago
This article isn't bad... but it misses several important points of Go. I also note the article is 9 months old. In the hope that my criticism will be taken as constructive, with apologies for not writing detailed explanations: 1. Goroutines are not threads 2. type inference allows you to elide types in var declarations: var host = flag.String(... 3. Go's convention is to use camel case, not underscores. 4. Calling os.Exit all over the place is unusual imo - it may be better to panic(). 5. fmt.Fprintf exists os.Stderr.WriteString(fmt.Sprintf... 6. An explanation of why the standard log package isn't suitable would be nice, although I see the format used is slightly different. 7. Ignored errors when writing. Why do you wish to sync at every packet? 8. Massive race condition by re-using b. Line 62 overwrites b. b is then passed to c.logger (68) and c.binary_logger (70) for them to process asynchronously. c.logger is then passed another byte slice (72), which forces it to finish using b. c.binary_logger is not passed anything else, allowing it to delay until the next time something is sent on that channel, which would be after b is overwritten by the next packet. I think that simply moving line 58 to between 61,62 would fix this. The author has not quite grokked the concept of "don't share memory". b is shared, and undefined behaviour results :(
- sirclueless 13y ago
  > I think that simply moving line 58 to between 61,62 would fix this. Doing that would erase the whole benefit of allocating a static array in the first place (you really don't want to be allocating memory for every packet that comes through). The funny thing is the author correctly handles this type of synchronization problem elsewhere, see lines 97-100. The right thing to do is to wait for acknowledgement from both loggers before reusing shared memory, but that sort of defeats his broader argument: if you are explicitly guarding every shared memory access with a barrier, then you might as well have been using any shared memory language besides Go. Buffered channels in both directions used to implement barriers are every bit as error-prone and complex as using a reader-writer lock explicitly.
  - Jabbles 13y ago
    > Doing that would erase the whole benefit of allocating a static array in the first place (you really don't want to be allocating memory for every packet that comes through). The benefit is dubious. Performance is unlikely to be important in a single-connection logger. My advice is to lean on the garbage collector. Allocate away. Buffered channels are certainly more complex, but if you're never sharing memory, shouldn't hurt you.
EllaMentry 13y ago
This article and many like suffer from one of my huge pet-peeves, absolutely terrible coding conventions. I am a person who likes to scan articles, I'm busy and generally make a read now, read later, read never decision. The code from first scan was unreadable, short 1 character variable names, "why is there a hardcoded date marked 2006.01.02-15.04.05 there??", etc. Readable code takes a little more time - but it's worth it! Further, the entire example seems contrived. Am I right in thinking that simply firing up wireshark would solve this problem? Why is the author continuing to write something in nearly every language when a tool exists for exactly this purpose, is multi-featured and pluggable? Even further, message passing! With the rise in recent years of message passing libraries in nearly every language, multi-threaded, distributed applications are becoming trivial to write. I do not see the Go code presented as anything other than messy, I have seen C++ code utilizing message passing libs that are smaller, prettier and infinitely more maintainable - again with no mutex or conditional variables! If you want to sell me on Go, make the code pretty, and present a USP.
- Jabbles 13y ago
  I'm afraid many of the conventions you complain about are standard Go, err, conventions... (Although see my complaint elsewhere.) Because they are so widely used, people who use Go won't bat an eyelid. Unfortunately that doesn't make the article a great introduction to Go. I hope these talks can help sell you on Go http://blog.golang.org/2013/01/two-recent-go-talks.html http://blog.golang.org/2013/01/two-recent-go-talks.html To be more explicit: Short variable names generally reflect the idea that you know what a variable is for just by knowing its type. Thus you have a file named f, a time t, a variadic argument called v. When the type is not enough, a longer name is recommended. Naming things is hard though... The hardcoded date is a wonderful piece of the time package, which I fully appreciate will look bizarre at first. (And therefore isn't a great thing to use in a first look at Go, without explanation). See the official documentation http://golang.org/pkg/time/#pkg-constants http://golang.org/pkg/time/#pkg-constants
  - deleted 13y ago
    [deleted]
- 13y ago
laureny 13y ago
> > Notably, if a package is included but not used, Go treats this as an error and enforces removing unused declarations A good illustration that the Go designers didn't think their ideas through. This is a real pain in the butt when you are writing code and regularly commenting in and out sections of code while you are testing things. And every time you do this, you need to remove or restore the imports. And since Go's tooling is nonexistent, there is no IDE to do this automatically for you. This kind of thing belongs in a compiler plug-in (if it was designed with such a thing in mind, which is not the case for Go), macros (if the languages supports them, ideally the hygienic and statically typed kind) or an external tool, not in the compiler.
- burntsushi 13y ago
  > A good illustration that the Go designers didn't think their ideas through. Please don't misconstrue disagreement as sloppiness. If you read the mailing list, it's pretty clear they thought it through. > This is a real pain in the butt when you are writing code and regularly commenting in and out sections of code while you are testing things. Not for me. I love it, actually. > And since Go's tooling is nonexistent, there is no IDE to do this automatically for you. Vim does this for me. Also, Go has some of the most wonderful tools of any programming language I've ever used. > This kind of thing belongs in a compiler plug-in (if it was designed with such a thing in mind, which is not the case for Go), macros (if the languages supports them, ideally the hygienic and statically typed kind) or an external tool, not in the compiler. I think reasonable people can disagree on this point.
- jbarham 13y ago
  False. The Go designers have explained many times why they made unused imports an error. In particular unused dependencies slow down compilation. There's even a FAQ: http://golang.org/doc/faq#unused_variables_and_imports http://golang.org/doc/faq#unused_variables_and_imports. A trivial workaround to silence the compiler errors during development is to use blank identifiers (http://golang.org/doc/effective_go.html#blank_unused http://golang.org/doc/effective_go.html#blank_unused). I use them all the time. You may disagree with their decision, but it's disingenuous to claim that it's because they didn't think it through.
deleted 13y ago
[deleted]
dlsspy 13y ago
There's so much code here. If I weren't sick, I'd submit a new version that: 1. Didn't reimplement io.Copy 2. Didn't avoid io.TeeReader 3. Didn't do weird things to avoid regular channel ranges. 4. Didn't do non-standard date formatting. 5. Didn't reinvent the log package. 6. Didn't try to convince anyone runtime.GOMAXPROCS(runtime.NumCPU()) was a good idea (it's not) In fact, maybe I will anyway. brb
- dlsspy 13y ago
  Here's the version I threw together in (what it says here), 12 minutes: https://gist.github.com/dustin/5478818 https://gist.github.com/dustin/5478818
- cpeterso 13y ago
  > 6. Didn't try to convince anyone runtime.GOMAXPROCS(runtime.NumCPU()) was a good idea (it's not) So what is an appropriate GOMAXPROCS? As someone who has only dabbled in a few Go tutorials, I would imagine that you would want GOMAXPROCS to be NumCPU() (or even greater) so the goroutine thread pool could "fire on all pistons". Why does Go's scheduler default to GOMAXPROCS=1 instead of NumCPU()?
  - dlsspy 13y ago
    Or turning that around, do you think it requires all 8 of my cores to copy data over the network? Do you think the two lines of code + justification text provides sufficient value for this application to distract from the point of it in order to show why someone should override the default behavior? Do you believe users shouldn't have any control over the number of cores any particular application consumes? Have you measured the CPU contention of the application and determined that using more cores is worth the overhead of increased overhead of multi-thread exclusions (vs. more simple things happening directly in the scheduler)? Overall, it has nothing to do with this article and now even more people are going to copy it in more unnecessary places as a cargo-cult "turbo button" for their programs. If you are going to use an idiom like that, the least you could do is check for the GOMAXPROCS environment variable and only do this as a default when the user hasn't specified otherwise.
  - 13y ago
diamondhead 13y ago
this code examples aren't good looking

133 points by iand 13y ago

This is a nice article. I would encourage the author to use the term "goroutine" instead of "thread" when referring to Go's goroutines, because they aren't threads. The pros and cons of threads do not apply to goroutines. What's cool (for me) about this article is that the author knows this fact (and states so at the outset), but he re-discovers it and internalizes it over the course of his implementation.

StavrosK 13y ago

Ugh, I just finished writing the XMPP frontend for an XMPP/IRC bot I'm working on (http://www.getinstabot.com http://www.getinstabot.com). The frontends are in Go for concurrency, and ferry messages back and forth from the channels to the backend. Let me tell you, that problem is hard. Go coped pretty well, but the final thing is a mess of global states, and it's pretty elegant for what the problem is. I was hoping to avoid having many moving parts, but it ended up needing a lot of shared state between all processes. Some problems are just hard, and, no matter how well-designed the language is, they'll still be hard. Something I miss from the language after implementing that is the ability to, say, monitor one goroutine from another to see if it returns (so the former can return as well). I know it's possible with channels, but when one goroutine is blocked on network Recv(), there's not much of a chance to listen to channels. Anyway, yes, parallelism in Go is great, but not everything is magically all rainbows and unicorns (that's Django). Some problems will be messy and dirty, and even more so when you use channels.

burntsushi 13y ago
> I know it's possible with channels, but when one goroutine is blocked on network Recv(), there's not much of a chance to listen to channels. Could you elaborate a bit more on this point? This seems like a natural pattern for a channel/goroutine. Spin up a goroutine that reads from a connection and send whatever is read on a channel. Then other goroutines can synchronize on the channel.
- StavrosK 13y ago
  Sure, but what if you want to shut that goroutine down later on? The best solution that I've found is to just close the connection, the .Recv() will error out immediately.
  - lobster_johnson 13y ago
    Yes, that is afaik the only way to abort a synchronous function call such as TCPConn.Read().
  - burntsushi 13y ago
    What other solution would you want? You could use `SetDeadline` if you like. But closing the connection seems reasonable too. I think you're framing the problem wrong. You're not asking to shutdown a goroutine, you're asking "How do I abort from a synchronous read from a network connection that is blocked?"
    - StavrosK 13y ago
      Don't get too hung up on that, it was the least of the problems. Closing the connection works fine, it's just a bit unorthodox. The basic structure of the problem was much more problematic to implement.
pandemicsyn 13y ago
heads up. Seems like https://getinstabot.com https://getinstabot.com and http://www.getinstabot.com http://www.getinstabot.com end up in a redirect loop.
- StavrosK 13y ago
  Oh. I forgot about this comment and picked a great time to redirect everything over https. Deploying the fix now, thanks.
NyxWulf 13y ago
Observing other channels, or seeing when a channel does fire is what the select structure is for. It is one of the features that makes channels and goroutines very nice in go. http://golang.org/ref/spec#Select_statements http://golang.org/ref/spec#Select_statements http://blog.golang.org/2010/09/go-concurrency-patterns-timing-out-and.html http://blog.golang.org/2010/09/go-concurrency-patterns-timin... http://talks.golang.org/2012/concurrency.slide#1 http://talks.golang.org/2012/concurrency.slide#1 Video for the slides: http://www.youtube.com/watch?v=f6kdp27TYZs http://www.youtube.com/watch?v=f6kdp27TYZs
lobster_johnson 13y ago
> Something I miss from the language … Does sound like you want to use channels, and break your logic into small independent parts. You will have one goroutine using blocking Read() in a loop and feeding data to some channel. When it's done, you write to another channel that exists only for signaling: defer { doneChannel <- true } for { data := make([]byte, 65535) _, err := conn.Read(data) if err != nil { errorChannel <- err break; } dataChannel <- data } and then in your other goroutine: for { select { case data := <- dataChannel: // Handle data case <- doneChannel: // Other goroutine is done } } The only things shared here are the channels. The way to avoid too much state and moving parts is to break the problem into isolated, manageable parts that communicate with channels. Often you will have hierarchical relationships like this, where one piece of dumb code exists to pass data from something lower down to somewhere higher up. It's not hard, although some of the code gets a bit ugly and disjoint at times, especially in how anything synchronous has to use channels and goroutines. For example, today I wrote a simple worker pool implementation that runs a given function in parallel via goroutines and can adjust the number of workers dynamically at runtime. That function has to be declared not just as "func()" but as "func(abortChannel chan bool)", and the worker function has to honour the abort signal when it arrives from the pool. So channels do leak everywhere, even into APIs. (Yeah, I know I can use "chan struct{}" to avoid any storage, but I think "<- true" looks nicer than "<- struct{} {}".) What is harder is to intelligently handle complex cascading failures. That's what Erlang, with its supervisor tree, is good at. Go's goroutines are "fire and forget" and cannot even be terminated programmatically from elsewhere in the program.
- StavrosK 13y ago
  Yeah, the channel approach was rewrite #3 :P It required me to have shared state in a horrible way as well, ruining the whole thing. There really isn't an elegant/functional way to do it, and sharing channels is error-prone itself. I ended up closing the connection to terminate goroutine B from A, and messaging on an already-shared channel to terminate A from B. Having cascading failures in Go would be fantastic. The problem, unfortunately, is not very amenable to elegant solutions. I might do a writeup at some point.
- 13y ago

octo_t 13y ago

This has finally let me figure out what annoys me about Go, its a cargo-cult language. People saw that Erlang's Actors/processes were really popular and made it easy to write good, concurrent software. They then went away and implemented their own language with lightweight processes and message passing, but missed the fact that actors are the price you have to pay for the benefits of not sharing mutable data. And Go completely skipped that part (the most important part).

sbov 13y ago
Channels are basically queues with beefed up language support. Not sure what's so annoying about that considering it's a very common concurrency pattern across many languages.
timclark 13y ago
Doesn't go just implement Hoare's communicating sequential processes, as does Erlang? They share the same inspiration. You don't need to share state data between your goroutines if you don't want to either just like you don't have to use mnesia to share state between erlang processes if you don't want to. I don't think you can really accuse go of being a cargo cult language either, Rob Pike has implemented CSP multiple times (http://swtch.com/~rsc/thread/ http://swtch.com/~rsc/thread/).
- ridiculous_fish 13y ago
  It is easy to accidentally share state between goroutines. For example, we wish to print out elements of a list: values := []string{"a", "b", "c"} for _, v := range values { go fmt.Println(v) } Each of these goroutines shares the same variable v, so this code contains a serious race condition.
  - Jabbles 13y ago
    It may be easy, but it's not that easy. values := []string{"a", "b", "c"} for _, v := range values { go func(){fmt.Println(v);}() } Does have a race condition. values := []string{"a", "b", "c"} for _, v := range values { go func(s string){fmt.Println(s);}(v) } Does not have a race condition.

codewright 13y ago

The beauty of concurrency in Clojure: ; Rough sketch: def defines a var (pretend it's a reference) ; @ is used to dereference the future and block to wait for the result. (def f (future (Thread/sleep 10000) (println "done") 100)) user=> @f done 100 ;; Dereferencing again will return the already calculated value. => @f 100 http://clojuredocs.org/clojure_core/clojure.core/future http://clojuredocs.org/clojure_core/clojure.core/future Edit: And more importantly, there are wrappers for the standard data structures designed around different concurrency use-cases (sync, async, coordinated, uncoordinated) Refs are for Coordinated Synchronous access to Many Identities". Atoms are for Uncoordinated synchronous access to a single Identity. Agents are for Uncoordinated asynchronous access to a single Identity. Vars are for thread local isolated identities with a shared default value. http://stackoverflow.com/questions/9132346/clojure-differences-between-ref-var-agent-atom-with-examples http://stackoverflow.com/questions/9132346/clojure-differenc... And you can use all (all!) of the Java concurrency tooling as desired, including raw threads (for which Clojure has a wrapper as well). Part of the reason I use Clojure rather than Go is because it doesn't try to force you into a one-size-fits-all method for handling concurrency. I have no problem with CSP but it doesn't fit everything I do. Sometimes I just want to defer work or wrap it in a future. Or I want to use an intelligent coordinated data structure rather than trying to meld flesh and bone to steel in order to make a concurrency-naive data structure behave how I want in a concurrent environ. If I can avoid those unnecessary battles, I will. So - Clojure.

hornetblack 13y ago
Futures in C++11 are similar: int x = std::async([]()->int{ return 100; }); std::cout << x.wait() << std::endl: Which isn't as nice as closure. Go could probably benfit from having a standard futures tool. I guess something along the lines of: type Future { Wait() interface{} } func newFuture(func interface{}, args ...interface{}) Future
- cerales 13y ago
  Yep, futures can be done easily in Go with a goroutine that sends a single value on a channel, but: 1) you can't dereference the value multiple times in Go, so you have to manage saving it yourself, and 2) there's no syntactic sugar. Pity.

Jabbles 13y ago

This article isn't bad... but it misses several important points of Go. I also note the article is 9 months old. In the hope that my criticism will be taken as constructive, with apologies for not writing detailed explanations: 1. Goroutines are not threads 2. type inference allows you to elide types in var declarations: var host = flag.String(... 3. Go's convention is to use camel case, not underscores. 4. Calling os.Exit all over the place is unusual imo - it may be better to panic(). 5. fmt.Fprintf exists os.Stderr.WriteString(fmt.Sprintf... 6. An explanation of why the standard log package isn't suitable would be nice, although I see the format used is slightly different. 7. Ignored errors when writing. Why do you wish to sync at every packet? 8. Massive race condition by re-using b. Line 62 overwrites b. b is then passed to c.logger (68) and c.binary_logger (70) for them to process asynchronously. c.logger is then passed another byte slice (72), which forces it to finish using b. c.binary_logger is not passed anything else, allowing it to delay until the next time something is sent on that channel, which would be after b is overwritten by the next packet. I think that simply moving line 58 to between 61,62 would fix this. The author has not quite grokked the concept of "don't share memory". b is shared, and undefined behaviour results :(

sirclueless 13y ago
> I think that simply moving line 58 to between 61,62 would fix this. Doing that would erase the whole benefit of allocating a static array in the first place (you really don't want to be allocating memory for every packet that comes through). The funny thing is the author correctly handles this type of synchronization problem elsewhere, see lines 97-100. The right thing to do is to wait for acknowledgement from both loggers before reusing shared memory, but that sort of defeats his broader argument: if you are explicitly guarding every shared memory access with a barrier, then you might as well have been using any shared memory language besides Go. Buffered channels in both directions used to implement barriers are every bit as error-prone and complex as using a reader-writer lock explicitly.
- Jabbles 13y ago
  > Doing that would erase the whole benefit of allocating a static array in the first place (you really don't want to be allocating memory for every packet that comes through). The benefit is dubious. Performance is unlikely to be important in a single-connection logger. My advice is to lean on the garbage collector. Allocate away. Buffered channels are certainly more complex, but if you're never sharing memory, shouldn't hurt you.

EllaMentry 13y ago

This article and many like suffer from one of my huge pet-peeves, absolutely terrible coding conventions. I am a person who likes to scan articles, I'm busy and generally make a read now, read later, read never decision. The code from first scan was unreadable, short 1 character variable names, "why is there a hardcoded date marked 2006.01.02-15.04.05 there??", etc. Readable code takes a little more time - but it's worth it! Further, the entire example seems contrived. Am I right in thinking that simply firing up wireshark would solve this problem? Why is the author continuing to write something in nearly every language when a tool exists for exactly this purpose, is multi-featured and pluggable? Even further, message passing! With the rise in recent years of message passing libraries in nearly every language, multi-threaded, distributed applications are becoming trivial to write. I do not see the Go code presented as anything other than messy, I have seen C++ code utilizing message passing libs that are smaller, prettier and infinitely more maintainable - again with no mutex or conditional variables! If you want to sell me on Go, make the code pretty, and present a USP.

Jabbles 13y ago
I'm afraid many of the conventions you complain about are standard Go, err, conventions... (Although see my complaint elsewhere.) Because they are so widely used, people who use Go won't bat an eyelid. Unfortunately that doesn't make the article a great introduction to Go. I hope these talks can help sell you on Go http://blog.golang.org/2013/01/two-recent-go-talks.html http://blog.golang.org/2013/01/two-recent-go-talks.html To be more explicit: Short variable names generally reflect the idea that you know what a variable is for just by knowing its type. Thus you have a file named f, a time t, a variadic argument called v. When the type is not enough, a longer name is recommended. Naming things is hard though... The hardcoded date is a wonderful piece of the time package, which I fully appreciate will look bizarre at first. (And therefore isn't a great thing to use in a first look at Go, without explanation). See the official documentation http://golang.org/pkg/time/#pkg-constants http://golang.org/pkg/time/#pkg-constants
- deleted 13y ago
  [deleted]
13y ago

laureny 13y ago

> > Notably, if a package is included but not used, Go treats this as an error and enforces removing unused declarations A good illustration that the Go designers didn't think their ideas through. This is a real pain in the butt when you are writing code and regularly commenting in and out sections of code while you are testing things. And every time you do this, you need to remove or restore the imports. And since Go's tooling is nonexistent, there is no IDE to do this automatically for you. This kind of thing belongs in a compiler plug-in (if it was designed with such a thing in mind, which is not the case for Go), macros (if the languages supports them, ideally the hygienic and statically typed kind) or an external tool, not in the compiler.

burntsushi 13y ago
> A good illustration that the Go designers didn't think their ideas through. Please don't misconstrue disagreement as sloppiness. If you read the mailing list, it's pretty clear they thought it through. > This is a real pain in the butt when you are writing code and regularly commenting in and out sections of code while you are testing things. Not for me. I love it, actually. > And since Go's tooling is nonexistent, there is no IDE to do this automatically for you. Vim does this for me. Also, Go has some of the most wonderful tools of any programming language I've ever used. > This kind of thing belongs in a compiler plug-in (if it was designed with such a thing in mind, which is not the case for Go), macros (if the languages supports them, ideally the hygienic and statically typed kind) or an external tool, not in the compiler. I think reasonable people can disagree on this point.
jbarham 13y ago
False. The Go designers have explained many times why they made unused imports an error. In particular unused dependencies slow down compilation. There's even a FAQ: http://golang.org/doc/faq#unused_variables_and_imports http://golang.org/doc/faq#unused_variables_and_imports. A trivial workaround to silence the compiler errors during development is to use blank identifiers (http://golang.org/doc/effective_go.html#blank_unused http://golang.org/doc/effective_go.html#blank_unused). I use them all the time. You may disagree with their decision, but it's disingenuous to claim that it's because they didn't think it through.

deleted 13y ago

[deleted]

dlsspy 13y ago

There's so much code here. If I weren't sick, I'd submit a new version that: 1. Didn't reimplement io.Copy 2. Didn't avoid io.TeeReader 3. Didn't do weird things to avoid regular channel ranges. 4. Didn't do non-standard date formatting. 5. Didn't reinvent the log package. 6. Didn't try to convince anyone runtime.GOMAXPROCS(runtime.NumCPU()) was a good idea (it's not) In fact, maybe I will anyway. brb

dlsspy 13y ago
Here's the version I threw together in (what it says here), 12 minutes: https://gist.github.com/dustin/5478818 https://gist.github.com/dustin/5478818
cpeterso 13y ago
> 6. Didn't try to convince anyone runtime.GOMAXPROCS(runtime.NumCPU()) was a good idea (it's not) So what is an appropriate GOMAXPROCS? As someone who has only dabbled in a few Go tutorials, I would imagine that you would want GOMAXPROCS to be NumCPU() (or even greater) so the goroutine thread pool could "fire on all pistons". Why does Go's scheduler default to GOMAXPROCS=1 instead of NumCPU()?
- dlsspy 13y ago
  Or turning that around, do you think it requires all 8 of my cores to copy data over the network? Do you think the two lines of code + justification text provides sufficient value for this application to distract from the point of it in order to show why someone should override the default behavior? Do you believe users shouldn't have any control over the number of cores any particular application consumes? Have you measured the CPU contention of the application and determined that using more cores is worth the overhead of increased overhead of multi-thread exclusions (vs. more simple things happening directly in the scheduler)? Overall, it has nothing to do with this article and now even more people are going to copy it in more unnecessary places as a cargo-cult "turbo button" for their programs. If you are going to use an idiom like that, the least you could do is check for the GOMAXPROCS environment variable and only do this as a default when the user hasn't specified otherwise.
- 13y ago

diamondhead 13y ago

this code examples aren't good looking