5 ms·

https://golang.org/ https://golang.org/ In case you dont know, Golang goroutines are a marvel of parallelism. They are coroutines which are dispatched on a few

https://golang.org/ https://golang.org/ In case you dont know, Golang goroutines are a marvel of parallelism. They are coroutines which are dispatched on a few OS threads. So you can use 100% of a multi-cores CPU and yet, spawn, say, 10K of those light threads without worrying about context switches PLUS have them all run concurrently. I've found that golang is one of those rare language, like Lisp, that actually change the way you think about programming. Makes you feel really more powerful. If you dont know that language, I suggest to run the following and watch your cpu activity and memory (or any metric): import "time" func main() { for i := 0; i < 10000; i++ { go func() { for { time.Sleep(time.Second) } }() } }

dx034 9y ago
And yet, Go's standard map doesn't allow concurrent access. They recently added a concurrent map feature but it's probably easier to add a lock to your code than refactoring it with the new map type. I would've preferred if they had introduced a map type that can be used exactly as the standard one (i.e. without function calls). Calling functions via go func() is really easy but handling data between goroutines can still cause headaches and is something that could be improved.
- ta2384428 9y ago
  If you squint a little, you'll notice that go is pretty big on not hiding complexity. There are obvious exceptions to this such as the garbage collector, and heap/stack control. I'm unsure if the intent of not including a map function was due to this, however with a for loop such as for x := range ch { slice = append(slice, x) } , it is immediately obvious there are allocations happening in the background. The fact that map access, and slice access is not thread safe means there is no trickery going on in the background. The fact that it is not threadsafe means I don't need to worry about a lock if I only write to a map when it's created. Sure the compiler could take care of this - they have the race detector after all, but the compile speed is one of the design goals of go. I really like being able to compile in less than 1 second. `sync.Map` however calls a function which implies there is more going on in the background. If you follow the go mantra of share memory by communicate, don't communicate by sharing memory - handling data becomes a whole lot easier. It does allow you to share memory in case you do need the extra speed. Like most things, all designs are a matter of trade offs. Sacrifice one thing for another. There are other languages that provide the functionality you desire - but I understand the frustration when one thing is 90% what you want. Go obviously has room for improvement, and perhaps a native threadsafe map is one of those areas.
- quadcore 9y ago
  Well, go isn't really a high-level language which maps can be seen as a feature of. The idea behind go, I believe, is to focus and provide innovative system-level features.
  - eropple 9y ago
    Go is not a language for "system-level features" in the first place. Its garbage collector largely precludes it from that in any modern context. That it can't do something effectively does not mean that it shouldn't or didn't mean to. (And users can't really fix it, because hey, no generics. Sigh.)
    - quadcore 9y ago
      Go is not a language for "system-level features" in the first place. Its garbage collector largely precludes it from that in any modern context. I'm pretty sure I disagree with that largely because Go is low-level enough so that the GC is easily handled. And users can't really fix it, because hey, no generics. Sigh. Users can fix it by writing languages on top of Go (particularly dynamic ones).
tzahola 9y ago
Your example will not run in parallel. The go runtime will schedule your goroutines concurrently, but they will be run by a single OS thread, and consequently on a single CPU core. Once you execute truly on multiple CPU cores (by increasing GOMAXPROCS), you'll be having the same kind of race conditions in Go as in any other imperative language (inb4 Rust Evangelism Strike Force saying "except Rust").
- quadcore 9y ago
  Wrong. Goroutines are not simply coroutines. GOMAXPROCS defaults to number of cores.
  - tzahola 9y ago
    Prove it. Prove it by replacing Sleep in your example with some number crunching, and show how it scales with the number of cores in your CPU.
    - deleted 9y ago
      [deleted]
    - gall0ws 9y ago
      https://play.golang.org/p/o8odLCRqUMA https://play.golang.org/p/o8odLCRqUMA
    - ta2384428 9y ago
      https://imgur.com/a/DNpw3 https://imgur.com/a/DNpw3 Running the following code. https://play.golang.org/p/k_rRxNAyb0i https://play.golang.org/p/k_rRxNAyb0i I can assure you that go runs across all processors by default. https://docs.google.com/document/d/1At2Ls5_fhJQ59kDK2DFVhFu3g5mATSXqqV5QrxinasI/edit https://docs.google.com/document/d/1At2Ls5_fhJQ59kDK2DFVhFu3...
      - grkvlt 9y ago
        That's not what he's saying. He knows go can use all processors if GOMAXPROCS is set correctly, the argument seems to be that there will be race conditions just like any other threading, which seems pretty self evident to me: yes, multi-threaded code can have concurrency issues, film at 11...
VHRanger 9y ago
How is that an example an improvement over c++? C++ has had pragmatic omp parallel for for two decades
- ta2384428 9y ago
  I'm not familiar with OMP, so please forgive me if I'm incorrect - but OMP from my quick search appears to be thread based, while go routines are much lighter weight than threads. They have their own scheduler - which like most things is both good and bad depending on what you want them for. If you use them as intended, the internal scheduler is a great design decision. This means I can happily spawn 10,000 without concern. Additionally, combining them with the power of channels makes quick work of many tasks. Channels of course can be implemented in C++ too, but having the compiler take care of it for you with additional tools such as the race detector is very handy. For a large set of problems, they are very nice to work with.
  - pjmlp 9y ago
    You can do the same in C++ on Windows with PPL, UNIX/Windows with Intel TBB, or any of the fiber/co-routine libraries. Then there is the ongoing work to add async/await patterns into C++20.
    - quadcore 9y ago
      As a matter of fact, you can do the same in assembly if you want to. Notice how you can always have that answer when it comes to programming languages: you can do the same in. The point is that, the way it's made in go, is awfully handy.
      - pjmlp 9y ago
        I hardly see a difference from go func () { } and Task.Run(() => { }) With the benefit that on the latter example, the runtime allows me to customise how scheduling is done.
magnetic 9y ago
In my experience the facilities to throw tasks into a scheduler that will run them in parallel was never the hard thing to accomplish (regardless of the language: some may have built-in capabilities, others may have syntactic sugar provided by a lib, but at the end of the day, most systems provide some kind of runTask(f) method). What's really hard is to break down a problem into parallelizable chunks, figure out as much independent work as possible to reduce the touchpoints, and coordinate all those tasks such that they keep the CPU as busy as possible and as a whole finish as early as possible. Beyond this "parallel breakdown design", it's the little touchpoints with shared data structures and synchronization that creates the difficulty of implementation, and I haven't seen any language or system that does magic there.
- quadcore 9y ago
  throw tasks into a scheduler that will run them in parallel was never the hard thing to accomplish Common mistake number 1. Goroutines are running in parallel AND concurrently - they are coroutines (common mistake number 2 is to think they are only coroutines). I suggest not to underestimate that, it's the big deal. Additionnaly, it's important to note that goroutines yields on sleeps (any kind of sleeps / waits, like disk reads, network requests, channel writes/reads, etc), and while it may sound like a detail, it's a wonder of cpu control. Due to that, there is also a rare elegance to the way Go solve sharing data via channels. What's really hard is to break down a problem into parallelizable chunks, figure out as much independent work as possible to reduce the touchpoints, and coordinate all those tasks such that they keep the CPU as busy as possible and as a whole finish as early as possible. That's exactly what Go is a wonder for due to the combination of parallelism, concurrency, yield-on-sleep and channels.
brobinson 9y ago
>marvel of parallelism If Go's m:n is blowing your mind, you should check out Erlang. Spawning a million "processes" is not a big deal.
- quadcore 9y ago
  Cool, thanks for the suggestion.
Const-me 9y ago
> golang is one of those rare language Not sure how rare, .NET does that as well. Below is your sample translated to C#. It requires C# 7.1 because async main, but the rest of the stuff is available for many years, since 2010. static async Task routine() { await Task.Delay(TimeSpan.FromSeconds(1)); } static Task Main() { return Task.WhenAll(Enumerable.Range(0, 1000).Select(i => routine())); }
- quadcore 9y ago
  It's unclear to me if these parallel tasks are also coroutines (there seems to be a call to yield but its unclear what it really does), and if they yield on any sleeps which is a key features of goroutines.
  - Const-me 9y ago
    > if these parallel tasks are also coroutines I don’t have hands-on experience with golang but based on what I know about it yes, they are. > there seems to be a call to yield but its unclear what it really does You mean “await”? It waits for the completion of whatever is on the right side of “await”. If the result is already available, it just continues. If the result is not yet available (e.g. Task.Delay creates a task that will complete in some moment in the future), the control goes away from the async method to the scheduler. The scheduler can then run some other task on the same OS thread. When the result of that operation becomes available (in the sample code, when 1 second delay passes), the scheduler resumes execution of that async method, on the statement after the await. > if they yield on any sleeps No, not any sleep. You can call Thread.Sleep() which will put the whole OS thread to sleep. It’s up to programmers to avoid calling blocking APIs from their async methods, i.e. use await Task.Delay() instead of Thread.Sleep(), await stream.ReadAsync() instead of stream.Read(), and so on.

dx034 9y ago

And yet, Go's standard map doesn't allow concurrent access. They recently added a concurrent map feature but it's probably easier to add a lock to your code than refactoring it with the new map type. I would've preferred if they had introduced a map type that can be used exactly as the standard one (i.e. without function calls). Calling functions via go func() is really easy but handling data between goroutines can still cause headaches and is something that could be improved.

ta2384428 9y ago
If you squint a little, you'll notice that go is pretty big on not hiding complexity. There are obvious exceptions to this such as the garbage collector, and heap/stack control. I'm unsure if the intent of not including a map function was due to this, however with a for loop such as for x := range ch { slice = append(slice, x) } , it is immediately obvious there are allocations happening in the background. The fact that map access, and slice access is not thread safe means there is no trickery going on in the background. The fact that it is not threadsafe means I don't need to worry about a lock if I only write to a map when it's created. Sure the compiler could take care of this - they have the race detector after all, but the compile speed is one of the design goals of go. I really like being able to compile in less than 1 second. `sync.Map` however calls a function which implies there is more going on in the background. If you follow the go mantra of share memory by communicate, don't communicate by sharing memory - handling data becomes a whole lot easier. It does allow you to share memory in case you do need the extra speed. Like most things, all designs are a matter of trade offs. Sacrifice one thing for another. There are other languages that provide the functionality you desire - but I understand the frustration when one thing is 90% what you want. Go obviously has room for improvement, and perhaps a native threadsafe map is one of those areas.
quadcore 9y ago
Well, go isn't really a high-level language which maps can be seen as a feature of. The idea behind go, I believe, is to focus and provide innovative system-level features.
- eropple 9y ago
  Go is not a language for "system-level features" in the first place. Its garbage collector largely precludes it from that in any modern context. That it can't do something effectively does not mean that it shouldn't or didn't mean to. (And users can't really fix it, because hey, no generics. Sigh.)
  - quadcore 9y ago
    Go is not a language for "system-level features" in the first place. Its garbage collector largely precludes it from that in any modern context. I'm pretty sure I disagree with that largely because Go is low-level enough so that the GC is easily handled. And users can't really fix it, because hey, no generics. Sigh. Users can fix it by writing languages on top of Go (particularly dynamic ones).

tzahola 9y ago

Your example will not run in parallel. The go runtime will schedule your goroutines concurrently, but they will be run by a single OS thread, and consequently on a single CPU core. Once you execute truly on multiple CPU cores (by increasing GOMAXPROCS), you'll be having the same kind of race conditions in Go as in any other imperative language (inb4 Rust Evangelism Strike Force saying "except Rust").

quadcore 9y ago
Wrong. Goroutines are not simply coroutines. GOMAXPROCS defaults to number of cores.
- tzahola 9y ago
  Prove it. Prove it by replacing Sleep in your example with some number crunching, and show how it scales with the number of cores in your CPU.
  - deleted 9y ago
    [deleted]
  - gall0ws 9y ago
    https://play.golang.org/p/o8odLCRqUMA https://play.golang.org/p/o8odLCRqUMA
  - ta2384428 9y ago
    https://imgur.com/a/DNpw3 https://imgur.com/a/DNpw3 Running the following code. https://play.golang.org/p/k_rRxNAyb0i https://play.golang.org/p/k_rRxNAyb0i I can assure you that go runs across all processors by default. https://docs.google.com/document/d/1At2Ls5_fhJQ59kDK2DFVhFu3g5mATSXqqV5QrxinasI/edit https://docs.google.com/document/d/1At2Ls5_fhJQ59kDK2DFVhFu3...
    - grkvlt 9y ago
      That's not what he's saying. He knows go can use all processors if GOMAXPROCS is set correctly, the argument seems to be that there will be race conditions just like any other threading, which seems pretty self evident to me: yes, multi-threaded code can have concurrency issues, film at 11...

VHRanger 9y ago

How is that an example an improvement over c++? C++ has had pragmatic omp parallel for for two decades

ta2384428 9y ago
I'm not familiar with OMP, so please forgive me if I'm incorrect - but OMP from my quick search appears to be thread based, while go routines are much lighter weight than threads. They have their own scheduler - which like most things is both good and bad depending on what you want them for. If you use them as intended, the internal scheduler is a great design decision. This means I can happily spawn 10,000 without concern. Additionally, combining them with the power of channels makes quick work of many tasks. Channels of course can be implemented in C++ too, but having the compiler take care of it for you with additional tools such as the race detector is very handy. For a large set of problems, they are very nice to work with.
- pjmlp 9y ago
  You can do the same in C++ on Windows with PPL, UNIX/Windows with Intel TBB, or any of the fiber/co-routine libraries. Then there is the ongoing work to add async/await patterns into C++20.
  - quadcore 9y ago
    As a matter of fact, you can do the same in assembly if you want to. Notice how you can always have that answer when it comes to programming languages: you can do the same in. The point is that, the way it's made in go, is awfully handy.
    - pjmlp 9y ago
      I hardly see a difference from go func () { } and Task.Run(() => { }) With the benefit that on the latter example, the runtime allows me to customise how scheduling is done.

magnetic 9y ago

In my experience the facilities to throw tasks into a scheduler that will run them in parallel was never the hard thing to accomplish (regardless of the language: some may have built-in capabilities, others may have syntactic sugar provided by a lib, but at the end of the day, most systems provide some kind of runTask(f) method). What's really hard is to break down a problem into parallelizable chunks, figure out as much independent work as possible to reduce the touchpoints, and coordinate all those tasks such that they keep the CPU as busy as possible and as a whole finish as early as possible. Beyond this "parallel breakdown design", it's the little touchpoints with shared data structures and synchronization that creates the difficulty of implementation, and I haven't seen any language or system that does magic there.

quadcore 9y ago
throw tasks into a scheduler that will run them in parallel was never the hard thing to accomplish Common mistake number 1. Goroutines are running in parallel AND concurrently - they are coroutines (common mistake number 2 is to think they are only coroutines). I suggest not to underestimate that, it's the big deal. Additionnaly, it's important to note that goroutines yields on sleeps (any kind of sleeps / waits, like disk reads, network requests, channel writes/reads, etc), and while it may sound like a detail, it's a wonder of cpu control. Due to that, there is also a rare elegance to the way Go solve sharing data via channels. What's really hard is to break down a problem into parallelizable chunks, figure out as much independent work as possible to reduce the touchpoints, and coordinate all those tasks such that they keep the CPU as busy as possible and as a whole finish as early as possible. That's exactly what Go is a wonder for due to the combination of parallelism, concurrency, yield-on-sleep and channels.

brobinson 9y ago

>marvel of parallelism If Go's m:n is blowing your mind, you should check out Erlang. Spawning a million "processes" is not a big deal.

quadcore 9y ago
Cool, thanks for the suggestion.

Const-me 9y ago

> golang is one of those rare language Not sure how rare, .NET does that as well. Below is your sample translated to C#. It requires C# 7.1 because async main, but the rest of the stuff is available for many years, since 2010. static async Task routine() { await Task.Delay(TimeSpan.FromSeconds(1)); } static Task Main() { return Task.WhenAll(Enumerable.Range(0, 1000).Select(i => routine())); }

quadcore 9y ago
It's unclear to me if these parallel tasks are also coroutines (there seems to be a call to yield but its unclear what it really does), and if they yield on any sleeps which is a key features of goroutines.
- Const-me 9y ago
  > if these parallel tasks are also coroutines I don’t have hands-on experience with golang but based on what I know about it yes, they are. > there seems to be a call to yield but its unclear what it really does You mean “await”? It waits for the completion of whatever is on the right side of “await”. If the result is already available, it just continues. If the result is not yet available (e.g. Task.Delay creates a task that will complete in some moment in the future), the control goes away from the async method to the scheduler. The scheduler can then run some other task on the same OS thread. When the result of that operation becomes available (in the sample code, when 1 second delay passes), the scheduler resumes execution of that async method, on the statement after the await. > if they yield on any sleeps No, not any sleep. You can call Thread.Sleep() which will put the whole OS thread to sleep. It’s up to programmers to avoid calling blocking APIs from their async methods, i.e. use await Task.Delay() instead of Thread.Sleep(), await stream.ReadAsync() instead of stream.Read(), and so on.