11 ms·

Go compiler: Initial support for concurrent backend compilation

248 points by giovannibajo1 10y ago

azinman2 10y ago
Details on the CL are a little thin, and the "don't believe the hype" indicates this isn't automatically some big win. Can someone give more details?
- euyyn 10y ago
  If you click "Show more", that's the longest CL description I've ever seen :)
  - nishs 10y ago
    Beats rsc's commit message [0] by 158 words :) [0] https://github.com/golang/go/commit/b902a63ade47cf69218c9b38c9a783fc8cfc3041 https://github.com/golang/go/commit/b902a63ade47cf69218c9b38...
- stock_toaster 10y ago
  Did you click "show more" on the commit comment? Much of the CL details are simply hidden below the fold, as it were.
  - azinman2 10y ago
    Ah I didn't see that. Thanks!
- chrisseaton 10y ago
  What's a CL?
  - timothya 10y ago
    "Changelist". Similar words are commit, diff, patch, etc. (though sometimes those other words have other implications)
    - agumonkey 10y ago
      Thanks, I was deep off, Cell Line (reading too much genetics these days).. Compiling Level
  - jaffee 10y ago
    On github, it'd be called a "pull request".
  - nowbacktowork 10y ago
    I believe in this case it's Change Log.
- gpm 10y ago
  A reason I've heard against parallelizing compilers is that you get parallelism for free in large builds as your build system will build separate code-gen units (files in C, crates in rust, I believe in go it's packages) that don't depend on eachother in parallel anyways. Adding it to the compiler as well just creates synchronization overhead.
  - DSMan195276 10y ago
    I would agree. Basically all decent build systems will allow you to compile units that aren't dependent on each other in parallel. Threading the compiler may provide some gains, but it's likely that for a good portion of your build time your CPU is already fully utilized anyway (And without any locking overhead). I feel like the real gains would be with a threaded linker instead, but I can't say I have any data to back that up. But I mean, you can't link until everything is compiled, so presumably there are free CPU cores that could be used.
    - uluyol 10y ago
      Go's compilation unit is a package, not a file. This limits the amount of parallelism you can get compared to say C outside of the compiler. You have bottleneck packages like the runtime which have to be compiled before other packages, and in the case where you do incremental compilation you probably only have a few packages to compile anyway. Some of this is briefly touched on in the CL (see comments on the amount of concurrently that should be enabled), and there are issues on the bug tracker that go into more detail.
  - weberc2 10y ago
    Still, the benchmarks seem promising. 40% improvement on std lib packages is nothing to shake a stick at. The end of the CL explains that this is a best case scenario, one that will mostly benefit recompilation of a single package, which is the most common compilation activity.
  - obstinate 10y ago
    Since most builds are incremental and only touch a single package, this should still be a win. And it may be a win whenever there is just one package on the critical path. (I don't know if go packages must be built after their dependencies, but it stands to reason if so.)
- bradfitz 10y ago
  Josh is modest and knew I'd tweet about it so we was being defensive. :)
dilap 10y ago
where does this commit actually exist to try it out?
- TheDong 10y ago
  See the link that says 'Download'? Click it for instructions on how to checkout this CL.
  - dilap 10y ago
    That does it; thanks
- wging 10y ago
  There's a Download button that pops up both a bunch of git commands and a link to .patch files. Git fetching seems to work for me. If you're interested in built binaries, that I don't know.
andrecl 10y ago
Can someone explain the significance of this post?
- twotwotwo 10y ago
  If you're compiling a package with lots of functions in it, Go can now have multiple cores working concurrently on (much of the work of) compiling different functions. Go would already work on more than one _package_ at once, using multiple processes, which helped big builds covering many packages (think building Juju, Kubernetes, or Docker from scratch). The benefit here is in the common situation where you make a change in one package and recompile: more of your cores can be put to use getting that package rebuilt. This is a changelist for review, after preliminary work to move state out of globals, etc. If you click "Show more" it will show much more, including discussion of the internals and wall- and CPU-time benchmark results.
- rurban 10y ago
  The Go compiler is already much faster than your typical C and C++ compiler. Now the compilation step, which is done on those projects via make -j4, can be done natively in the go compiler within modules by itself, compiling functions in one module in parallel. Modules itself would be already parallelizable by make -j4, but since go build is its own tool independent of make, they re-invented it by themselves. It's significantly faster than make already, but comes with maintenance costs. go build is much stricter than traditional dependency systems. It's not really significant at all. Compile-times of bigger modules will get a bit faster, such as e.g. with zapcc, which got C++ compile times 50% faster by caching.
eloff 10y ago
Roughly 40% better compilation speed. Nice. Performance numbers from the bottom of the CL: All benchmarks are from my 8 core 2.9 GHz Intel Core i7 darwin/amd64 laptop. First, going from tip to this CL with c=1 costs about 3% CPU and has almost no memory impact. Comparing this CL to itself, from c=1 to c=2 improves real times 20-30%, costs 5-10% more CPU time, and adds about 2% alloc. From c=1 to c=4, real time is down ~40%, CPU usage up 10-20%, alloc up ~5% Going beyond c=4 on my machine tends to increase CPU time and allocs without impacting real time.
- izym 10y ago
  Your CPU has 4 cores, but 8 threads due to Hyper-threading. It's quite normal that not all programs will benefit from the additional threads. It would be interesting to see how the compiler performs on more physical cores.
  - mirekrusin 10y ago
    Exactly, has anybody tried it on Ryzen 7?
  - psiclops 10y ago
    That is a quote from the CL, not the hardware of the person you replied to. That being said, I'm pretty interested to see how this would perform on other systems, specifically production servers.
  - cbhl 10y ago
    Compiling programs is usually one of the cases that does benefit from filling all the hyperthreads, though? That's why people often cargo cult running "make -j<cores * 2>".
    - chrisseaton 10y ago
      In case other people don't see why this would be the case, hyperthreads are good for running a second thread while the first one stalls while waiting for something such as a memory read. Compilers often work with indirect data structures such as irregular graphs and symbol tables which don't cache perfectly and may cause memory stalls.
ridiculous_fish 10y ago
Taking a thread-oblivious program and making it multithreaded can be very hard. It requires global transitive knowledge of the function call graph and all shared data structures. How was this achieved? Was it by a dev who was intimately familiar with the code base? Was it just applying the race detector repeatedly until it stopped squawking? Something else?
- bradfitz 10y ago
  > Was it by a dev who was intimately familiar with the code base? Was it just applying the race detector repeatedly until it stopped squawking? Both.
ceocoder 10y ago
unrelated question for any Googlers looking at this, I like how PolyGerrit has an "Old UI" option at the bottom of the page. When I enabled PolyGerrit on my internal gerrit instance that button did not show up, and docs were rather sparse. Question - is it possible to turn on PolyGerrit with ability to switch back to Old UI in open source gerrit? I found the option to turn on PolyGerrit here[0] [0] https://groups.google.com/forum/#!msg/repo-discuss/3JjvqB-mKDE/Py4YSuASBwAJ https://groups.google.com/forum/#!msg/repo-discuss/3JjvqB-mK...
- EricBurnett 10y ago
  I use [1]. Does this work for you? I'm not sure the exact criteria at play. [1] https://chrome.google.com/webstore/detail/gerrit-ui-switcher/fdjpjnlhaohkkbjnglcfehbmcmpojklf https://chrome.google.com/webstore/detail/gerrit-ui-switcher...
0xFFC 10y ago
I got very interested in Go recently, but I have question, how do you do your debugging in Go? I am using VSCode, and although VSCode support for Go is the best support between editors, but there is no official support for debugging , it seems.
- im3s 10y ago
  I don't know about VSCode but I tried Gogland a few days ago and it has integrated debugging support
- konart 10y ago
  Nothing is better than Gogland or their (JetBrans') plugin for IDEA. You can also try https://github.com/derekparker/delve https://github.com/derekparker/delve
- saturn_vk 10y ago
  I thought vscode used delve to do go debugging. Also, the vim-go plugin has a lot more useful features than the vscode one
- skj 10y ago
  I don't want to pretend debuggers aren't useful... But I haven't used a debugger in at least ten years. Production Go for the past few. I debug with good log messages. Of course, not everyone's problems are exactly like mine, so I will believe you if you say it's necessary for your situation.
  - rantanplan 10y ago
    Think vast codebases that were created by many programmers that weren't you :D
  - baconizer 10y ago
    this reflects my team's experience, initially skilled programmers (from C# mainly) complains almost hourly about lack of IDE features, in particular - Debugging. 14 months in now with 9 major services in production, I hear no more conversation on this, and still no one yet bothered to setup dlv. 'Breakpoints', to our surprises, is not actually indispensable. Also I noticed positive cultural change around, we move faster and team talks about business problem solving more than programming itself, to a point I believe rich functionality IDE over complicated things we do here.
_ph_ 10y ago
This is not only a nice enhancement to the Go compiler and thus for all Go users, but also it shows why Go as a language is so relevant. Single thread performance of processors has not increased very much in the recent years, while the core count continued to grow - I just configured a 20 core 40 thread Xeon workstation and got prices between $4k and 5k. Go is a great language which combines C-like speeds for low-level code with a good concurrency model, making your program comparatively easy to parallelize on a per-function basis. As the work on the Go compiler shows, it is not trivial for complex programs, but definitely possible.
- noir_lord 10y ago
  Interesting with the Ryzen parts coming to market as well. I'm in the market for a new developer desktop and if it shakes out the way it's looking an 8 core/16 thread Ryzen is hitting the price/perf sweetspot.
  - _ph_ 10y ago
    Yes, I was so happy to learn that AMD is finally pushing 8 cores for mainstream processors, they were too long stuck on 4 cpus. Of course, building software fully using those cpus is the challenge, but thats where Go comes in.
    - noir_lord 10y ago
      It's not just the building, it's the ability to run multiple virtualised environments (for example at the moment I'm running Xubuntu, two vagrant machines and Microsoft's Windows 10/Edge testing VM).
    - pcwalton 10y ago
      As I elaborated on elsewhere, if you focus on task parallelism to the exclusion of data parallelism, like Go tends to do, then you are not making good use of the hardware.
      - weberc2 10y ago

248 points by giovannibajo1 10y ago

Details on the CL are a little thin, and the "don't believe the hype" indicates this isn't automatically some big win. Can someone give more details?

euyyn 10y ago
If you click "Show more", that's the longest CL description I've ever seen :)
- nishs 10y ago
  Beats rsc's commit message [0] by 158 words :) [0] https://github.com/golang/go/commit/b902a63ade47cf69218c9b38c9a783fc8cfc3041 https://github.com/golang/go/commit/b902a63ade47cf69218c9b38...
stock_toaster 10y ago
Did you click "show more" on the commit comment? Much of the CL details are simply hidden below the fold, as it were.
- azinman2 10y ago
  Ah I didn't see that. Thanks!
chrisseaton 10y ago
What's a CL?
- timothya 10y ago
  "Changelist". Similar words are commit, diff, patch, etc. (though sometimes those other words have other implications)
  - agumonkey 10y ago
    Thanks, I was deep off, Cell Line (reading too much genetics these days).. Compiling Level
- jaffee 10y ago
  On github, it'd be called a "pull request".
- nowbacktowork 10y ago
  I believe in this case it's Change Log.
gpm 10y ago
A reason I've heard against parallelizing compilers is that you get parallelism for free in large builds as your build system will build separate code-gen units (files in C, crates in rust, I believe in go it's packages) that don't depend on eachother in parallel anyways. Adding it to the compiler as well just creates synchronization overhead.
- DSMan195276 10y ago
  I would agree. Basically all decent build systems will allow you to compile units that aren't dependent on each other in parallel. Threading the compiler may provide some gains, but it's likely that for a good portion of your build time your CPU is already fully utilized anyway (And without any locking overhead). I feel like the real gains would be with a threaded linker instead, but I can't say I have any data to back that up. But I mean, you can't link until everything is compiled, so presumably there are free CPU cores that could be used.
  - uluyol 10y ago
    Go's compilation unit is a package, not a file. This limits the amount of parallelism you can get compared to say C outside of the compiler. You have bottleneck packages like the runtime which have to be compiled before other packages, and in the case where you do incremental compilation you probably only have a few packages to compile anyway. Some of this is briefly touched on in the CL (see comments on the amount of concurrently that should be enabled), and there are issues on the bug tracker that go into more detail.
- weberc2 10y ago
  Still, the benchmarks seem promising. 40% improvement on std lib packages is nothing to shake a stick at. The end of the CL explains that this is a best case scenario, one that will mostly benefit recompilation of a single package, which is the most common compilation activity.
- obstinate 10y ago
  Since most builds are incremental and only touch a single package, this should still be a win. And it may be a win whenever there is just one package on the critical path. (I don't know if go packages must be built after their dependencies, but it stands to reason if so.)
bradfitz 10y ago
Josh is modest and knew I'd tweet about it so we was being defensive. :)

dilap 10y ago

where does this commit actually exist to try it out?

TheDong 10y ago
See the link that says 'Download'? Click it for instructions on how to checkout this CL.
- dilap 10y ago
  That does it; thanks
wging 10y ago
There's a Download button that pops up both a bunch of git commands and a link to .patch files. Git fetching seems to work for me. If you're interested in built binaries, that I don't know.

andrecl 10y ago

Can someone explain the significance of this post?

twotwotwo 10y ago
If you're compiling a package with lots of functions in it, Go can now have multiple cores working concurrently on (much of the work of) compiling different functions. Go would already work on more than one _package_ at once, using multiple processes, which helped big builds covering many packages (think building Juju, Kubernetes, or Docker from scratch). The benefit here is in the common situation where you make a change in one package and recompile: more of your cores can be put to use getting that package rebuilt. This is a changelist for review, after preliminary work to move state out of globals, etc. If you click "Show more" it will show much more, including discussion of the internals and wall- and CPU-time benchmark results.
rurban 10y ago
The Go compiler is already much faster than your typical C and C++ compiler. Now the compilation step, which is done on those projects via make -j4, can be done natively in the go compiler within modules by itself, compiling functions in one module in parallel. Modules itself would be already parallelizable by make -j4, but since go build is its own tool independent of make, they re-invented it by themselves. It's significantly faster than make already, but comes with maintenance costs. go build is much stricter than traditional dependency systems. It's not really significant at all. Compile-times of bigger modules will get a bit faster, such as e.g. with zapcc, which got C++ compile times 50% faster by caching.

eloff 10y ago

Roughly 40% better compilation speed. Nice. Performance numbers from the bottom of the CL: All benchmarks are from my 8 core 2.9 GHz Intel Core i7 darwin/amd64 laptop. First, going from tip to this CL with c=1 costs about 3% CPU and has almost no memory impact. Comparing this CL to itself, from c=1 to c=2 improves real times 20-30%, costs 5-10% more CPU time, and adds about 2% alloc. From c=1 to c=4, real time is down ~40%, CPU usage up 10-20%, alloc up ~5% Going beyond c=4 on my machine tends to increase CPU time and allocs without impacting real time.

izym 10y ago
Your CPU has 4 cores, but 8 threads due to Hyper-threading. It's quite normal that not all programs will benefit from the additional threads. It would be interesting to see how the compiler performs on more physical cores.
- mirekrusin 10y ago
  Exactly, has anybody tried it on Ryzen 7?
- psiclops 10y ago
  That is a quote from the CL, not the hardware of the person you replied to. That being said, I'm pretty interested to see how this would perform on other systems, specifically production servers.
- cbhl 10y ago
  Compiling programs is usually one of the cases that does benefit from filling all the hyperthreads, though? That's why people often cargo cult running "make -j<cores * 2>".
  - chrisseaton 10y ago
    In case other people don't see why this would be the case, hyperthreads are good for running a second thread while the first one stalls while waiting for something such as a memory read. Compilers often work with indirect data structures such as irregular graphs and symbol tables which don't cache perfectly and may cause memory stalls.

ridiculous_fish 10y ago

Taking a thread-oblivious program and making it multithreaded can be very hard. It requires global transitive knowledge of the function call graph and all shared data structures. How was this achieved? Was it by a dev who was intimately familiar with the code base? Was it just applying the race detector repeatedly until it stopped squawking? Something else?

bradfitz 10y ago
> Was it by a dev who was intimately familiar with the code base? Was it just applying the race detector repeatedly until it stopped squawking? Both.

ceocoder 10y ago

unrelated question for any Googlers looking at this, I like how PolyGerrit has an "Old UI" option at the bottom of the page. When I enabled PolyGerrit on my internal gerrit instance that button did not show up, and docs were rather sparse. Question - is it possible to turn on PolyGerrit with ability to switch back to Old UI in open source gerrit? I found the option to turn on PolyGerrit here[0] [0] https://groups.google.com/forum/#!msg/repo-discuss/3JjvqB-mKDE/Py4YSuASBwAJ https://groups.google.com/forum/#!msg/repo-discuss/3JjvqB-mK...

EricBurnett 10y ago
I use [1]. Does this work for you? I'm not sure the exact criteria at play. [1] https://chrome.google.com/webstore/detail/gerrit-ui-switcher/fdjpjnlhaohkkbjnglcfehbmcmpojklf https://chrome.google.com/webstore/detail/gerrit-ui-switcher...

0xFFC 10y ago

I got very interested in Go recently, but I have question, how do you do your debugging in Go? I am using VSCode, and although VSCode support for Go is the best support between editors, but there is no official support for debugging , it seems.

im3s 10y ago
I don't know about VSCode but I tried Gogland a few days ago and it has integrated debugging support
konart 10y ago
Nothing is better than Gogland or their (JetBrans') plugin for IDEA. You can also try https://github.com/derekparker/delve https://github.com/derekparker/delve
saturn_vk 10y ago
I thought vscode used delve to do go debugging. Also, the vim-go plugin has a lot more useful features than the vscode one
skj 10y ago
I don't want to pretend debuggers aren't useful... But I haven't used a debugger in at least ten years. Production Go for the past few. I debug with good log messages. Of course, not everyone's problems are exactly like mine, so I will believe you if you say it's necessary for your situation.
- rantanplan 10y ago
  Think vast codebases that were created by many programmers that weren't you :D
- baconizer 10y ago
  this reflects my team's experience, initially skilled programmers (from C# mainly) complains almost hourly about lack of IDE features, in particular - Debugging. 14 months in now with 9 major services in production, I hear no more conversation on this, and still no one yet bothered to setup dlv. 'Breakpoints', to our surprises, is not actually indispensable. Also I noticed positive cultural change around, we move faster and team talks about business problem solving more than programming itself, to a point I believe rich functionality IDE over complicated things we do here.

_ph_ 10y ago

This is not only a nice enhancement to the Go compiler and thus for all Go users, but also it shows why Go as a language is so relevant. Single thread performance of processors has not increased very much in the recent years, while the core count continued to grow - I just configured a 20 core 40 thread Xeon workstation and got prices between $4k and 5k. Go is a great language which combines C-like speeds for low-level code with a good concurrency model, making your program comparatively easy to parallelize on a per-function basis. As the work on the Go compiler shows, it is not trivial for complex programs, but definitely possible.

noir_lord 10y ago
Interesting with the Ryzen parts coming to market as well. I'm in the market for a new developer desktop and if it shakes out the way it's looking an 8 core/16 thread Ryzen is hitting the price/perf sweetspot.
- _ph_ 10y ago
  Yes, I was so happy to learn that AMD is finally pushing 8 cores for mainstream processors, they were too long stuck on 4 cpus. Of course, building software fully using those cpus is the challenge, but thats where Go comes in.
  - noir_lord 10y ago
    It's not just the building, it's the ability to run multiple virtualised environments (for example at the moment I'm running Xubuntu, two vagrant machines and Microsoft's Windows 10/Edge testing VM).
  - pcwalton 10y ago
    As I elaborated on elsewhere, if you focus on task parallelism to the exclusion of data parallelism, like Go tends to do, then you are not making good use of the hardware.
    - weberc2 10y ago