21 ms·

Building a Better Go Linker

238 points by rjammala 7y ago

cpeterso 7y ago
"The original linker was also simpler than it is now and its implementation fit in one Turing award winner’s head, so there’s little abstraction or modularity. Unfortunately, as the linker grew and evolved, it retained its lack of structure, and our sole Turing award winner retired." :)
- luuio 7y ago
  For those out of the loop, who are they talking about here?
  - p7IDD243 7y ago
    Perhaps Ken Thompson?
    - rjammala 7y ago
      Kenneth Lane Thompson is an American pioneer of computer science. Thompson worked at Bell Labs for most of his career where he designed and implemented the original Unix operating system. Books: Queen and Pawn Against Queen, Roycroft's 5-man Chess Endgame Series Awards: Turing Award, National Medal of Technology and Innovation, Japan Prize
  - insulanus 7y ago
    Definitely Ken Thompson. :) https://en.wikipedia.org/wiki/Ken_Thompson https://en.wikipedia.org/wiki/Ken_Thompson
- eeZah7Ux 7y ago
  Bragging doesn't make the language less unremarkable.
derefr 7y ago
> its implementation fit in one Turing award winner’s head What a great unit of measure :)
- ChrisSD 7y ago
  Wait, how many London buses fit in a Turing award winner's head? And can they all fit in to Wales despite the number of Olympic sized swimming pools already there?
  - ggm 7y ago
    Convert to Sydney harbours and add a banana for scale
  - solresol 7y ago
    I think we also need to convert that into elephants and blue whales. What's the conversion rate for elephants to London buses?
- rootlocus 7y ago
  Another great unit of measure is the bus factor.
  - heavenlyhash 7y ago
    I think that's part of the dry humor, here.
jstimpfle 7y ago
I don't get it. A linker's task should be straightforward. In essence, it looks up addresses from strings, whose number is bounded by the lines of code written by humans. I think that there must be a lot of incidental complexity if that task somehow becomes a bottleneck. And how can it be that a binary called "cmd/compile" has 170k symbols (that's like, global definitions, right?). Not that that's a huge number in terms of today's computing power, but how many millions of lines of source code does that correspond to? Still, 1M relocations, or 34MB of "Reloc" objects, as indicated, shouldn't be a huge issue to process. Object files should have minimal parsing overhead. Is there any indication how long this takes to link? Shouldn't 1 or 2 secs be sufficient? (assuming 100s of MB/s for sequential disk read/write, and < 1us to store each of the 170k symbols in a hashmap, and < 1us to to look up each of the 1M of relocations). - I don't think mmap should be used if it can be avoided. It means giving up control over what parts of the file are loaded in memory. And from a semantic point of view, that memory region still must be treated differently, since on-disk and in-memory data structures are not the same.
- wahern 7y ago
  > that's like, global definitions, right? No, not just global definitions. Closures need linking, too. But the linker is doing much more than linking function entry points. Many automatic (on the stack) variables need linking so the GC can (a) trace the object graph and (b) move them when resizing the stack. Likewise, type definitions require metadata generation for GC tracing. And then there's all the debugging data that needs to be generated, which basically involves everything.
  - nappy-doo 7y ago
    Go currently does DWARF generation in the linker, but is in the process of moving that into the compiler. Also, most closures linking problems are relatively easy in Go. As far as stack variables for the GC, the stack is exactly scanned for all but the last frame (I believe), but conservatively scanned for the last frame. This makes it much easier. Types -- you're partially right. It is mostly all generated in the compiler, and deduped by the linker. There's no reason Go's linker couldn't be much simpler and likely even simpler than a standard C linker. You might argue that Go will give up things like LTO, but Go designs out lots of the LTO problem (not PGO) by nature of how packages work, and the fact that there aren't cyclic dependencies. Overall, the Go linker could be quite simple. It just needs some rework is all. :)
jacobush 7y ago
I hoped they would fix how the linker uses private OS calls.
- deleted 7y ago
  [deleted]
- nappy-doo 7y ago
  Which private OS calls. Can you site what they are?
  - kccqzy 7y ago
    I'm guessing GP refers to the fact that Go binaries use raw system calls even on operating systems where the syscall boundary is considered a private implementation detail (which is the case on Windows and macOS).
    - cpuguy83 7y ago
      On Mac, since go1.11, Go uses libsystem.
    - 4ad 7y ago
      The Windows port never used raw system calls. The Solaris port never used raw system calls. The MacOS port doesn't use raw system calls since go 1.11. The remaining ports for Linux/BSDs/Plan 9 have a stable syscall interface.
  - jbn 7y ago
    I think you mean "cite".
    - nappy-doo 7y ago
      I do, but will leave it because I stand by me inability to spell. Thanks!
      - 7y ago
slx26 7y ago
I love to see that at the end the document the plugins package is mentioned. Currently it doesn't have support for Windows (among other issues), so it's really good to hear that the work in the linker might eventually improve the situation with plugins too, and that at least it's being kept in mind. On golang, interfaces being implemented implicitly is not everyone's cup of tea, and I can understand why, but when you combine that property with plugins, you can make really powerful plugin systems that use very simple code to work.
- nappy-doo 7y ago
  Plugins are hard, and very OS dependent. There's plenty about the current system that is fragile, and likely to cause problems. I think it's likely the linker rework needs to happen before plugins can be properly addressed.
  - jart 7y ago
    This doc offers great depth on why dynamic shared objects are so hard: https://software.intel.com/sites/default/files/m/a/1/e/dsohowto.pdf https://software.intel.com/sites/default/files/m/a/1/e/dsoho... and that's just for Linux & co. I'm surprised the Go team chose to add support for DSOs.
- axaxs 7y ago
  agreed. Just yesterday I was reading about shared object support in Go, and it hasn't improved since inception. Sometimes I just want an object/module, and not a git repo. Even VB had this down.
  - pushpop 7y ago
    As much as I think VisualBasic gets a lot of unfair criticism, I really don’t think the OCX model is something worth admiring. Sure, when it worked it was great but all too often they would fail between OS updates, language patches or even just installing other applications if they happened to use different OCXs. What’s more, those bastard files required registry hacks to work. If anything, OCX is an example of how not to do shared libraries.
AlexB138 7y ago
Any recommendations on resources to better understand the build process for compiled languages in general, and Go in particular?
- ori_b 7y ago
  https://www.amazon.com/dp/1558604960 https://www.amazon.com/dp/1558604960
  - journalctl 7y ago
    I just bought this! It’s fantastic and definitely still more than relevant for understanding the subject.
- nappy-doo 7y ago
  As weird as it sounds, Go is one of the easiest to understand compilers I've ever worked on. The language, and the implementation are relatively easy to understand and follow most of the standard compiler design and implementation texts. Similarly, since Go is so opinionated, the code is easy to read.
- wahern 7y ago
  Write your own simple virtual machine[1] and a tiny assembler which accepts symbolic names for jump targets (e.g. goto LABEL). You'll need a link phase to resolve the symbolic names, and all will become clear. A few hours of time and anything involving linking, addressing[2], and so much more will instantly become intuitive to you. [1] Array of opcodes indexed with a program counter (pc) with a giant switch statement for executing each opcode. A simple stack for data implemented as an array and a stack pointer/index (sp). There are plenty of examples online, just make sure you actually implement things yourself, and only go to the examples to answer questions you arrive at yourself. If you understand loops and arrays implementing the VM is trivial. If you can parse a text file line-by-line and split words, implementing the assembler is trivial. Working out the linking might take some thinking, but that's the point. It's like one night of work, max, unless you really get sucked in ;) [2] E.g. Harvard architecture vs. von Neumann architecture
benesch 7y ago
What I would give for the developers of the Go toolchain to have spent the last decade improving GCC or LLVM instead of their own bespoke toolchain. In many ways Go seems like an excuse for Google to fund the continued development of Plan 9. Three of the five most influential people on the Go team (Ken Thompson, Rob Pike, and Russ Cox) were heavily involved in Plan 9. And it shows. Go's toolchain is a direct descendant of the Plan 9 toolchain; in fact, the Go language is really just an evolution of the special C dialect that Plan 9 used [1]. Indeed, for a while, the Go compiler was written in this special dialect of C, and so building Go required building a C compiler (!) that could compile this custom dialect of C, and using that to compile the Go compiler [2]. By all rights, Plan 9 was an interesting research project, and seems well loved by those familiar with it. (I'm not personally familiar; it was well before my time.) But it never took off. What we ended up with is Linux, macOS, and Windows. Go very much wants to be Plan 9. Sure, it's not a full-fledged operating system. But it's a linker, assembler, compiler, binutils, and scheduler. All it asks of the host system is memory management, networking, and filesystem support, and it will happily replace your system's DNS resolution with a pure Go version if you ask it to [3]. I wouldn't be surprised if Go ships its own TCP/IP stack someday [4]. This is, in my opinion, craziness. What other language ships its own assembler?! [5] To make matters worse, the assembly syntax is largely undocumented, and what is documented are the strange, unnecessary quirks, like > Instructions, registers, and assembler directives are always in UPPER CASE to remind you that assembly programming is a fraught endeavor. (Exception: the g register renaming on ARM.) > In the general case, the frame size is followed by an argument size, separated by a minus sign. (It's not a subtraction, just idiosyncratic syntax.) > In Go object files and binaries, the full name of a symbol is the package path followed by a period and the symbol name: fmt.Printf or math/rand.Int. Because the assembler's parser treats period and slash as punctuation, those strings cannot be used directly as identifier names. Instead, the assembler allows the middle dot character U+00B7 and the division slash U+2215 in identifiers and rewrites them to plain period and slash. The excuse for the custom toolchain has always been twofold, that a) LLVM is too slow, and fast compiles are one of Go's main features, and b) that the core team was too unfamiliar with GCC/LLVM, at least in the early days, and attempting to build Go on top of LLVM would have slowed the speed of innovation to a degree that Go might not exist [6]. I've always been skeptical of argument (b). After all, one of Go's creators literally won a Turing award, as this document not-so-subtly mentions. I'm quite sure they could have figured out how to build an LLVM frontend, given the desire. Rust, for example, is quite a bit more complicated than Go, and Mozilla's developers have had no trouble integrating with LLVM. I suspect the real reason was that hacking on the Plan 9 toolchain was more fun and more familiar—which is a very valid personal reason to work on something! But it doesn't mean it was the right strategic decision. I will say that (a) is valid. I recently switched from writing Go to writing Rust, and I miss the compile times of Go desperately. That said—and this is what I can't get past—the state of compilers would be much better off if the folks on the Go team had invested more in improving the compile and link times of LLVM or GCC. Every improvement to lld wouldn't just speed up compiles for Go; it would speed up compiles for C, C++, Swift, Rust, Fortran, Kotlin, and anything else with an LLVM frontend. In the last year or so, the gollvm project [7] (which is exactly what you'd expect–a Go frontend for LLVM) has seen some very active development, and I'm following along excitedly. Unfortunately I still can't quite tell whether it's Than McIntosh's 20% time project or an actual staffed project of Google's, albeit a small time one. (There are really only two committers, Than and Cherry Zhang.) There are so many optimizations that will likely never be added to gc, like a register-based calling convention [8] and autovectorization, that you essentially get for "free" (i.e., with a bit of plumbing from the frontend) with a mature toolchain like LLVM. There are not many folks who have the knowledge and expertise to work on compilers and linkers these days, and those that do can command high salaries. Google is in the luxurious position of being able to afford many dozens of these people. I just wish that someone with the power to effect change at Google would realize that the priorities are backwards. gccgo/gollvm are where the primary investment should be occurring, and the gc toolchain should be a side project that makes debug builds fast... not the production compiler, where the quality of the object code is the primary objective. [0]: https://dave.cheney.net/2013/10/15/how-does-the-go-build-command-work https://dave.cheney.net/2013/10/15/how-does-the-go-build-com... [1]: http://doc.cat-v.org/plan_9/programming/c_programming_in_plan_9 http://doc.cat-v.org/plan_9/programming/c_programming_in_pla... [2]: https://docs.google.com/document/d/1P3BLR31VA8cvLJLfMibSuTdwTuF7WWLux71CYD0eeD8/edit https://docs.google.com/document/d/1P3BLR31VA8cvLJLfMibSuTdw... [3]: https://golang.org/pkg/net/ https://golang.org/pkg/net/ [4]: https://github.com/google/netstack https://github.com/google/netstack [5]: https://golang.org/doc/asm https://golang.org/doc/asm [6]: https://golang.org/doc/faq#What_compiler_technology_is_used_to_build_the_compilers https://golang.org/doc/faq#What_compiler_technology_is_used_... [7]: https://go.googlesource.com/gollvm/ https://go.googlesource.com/gollvm/ [8]: https://github.com/golang/go/issues/18597 https://github.com/golang/go/issues/18597
Solar19 7y ago
I like it. What would also be cool is another Go compiler and linker, developed by a different team not at Google. It would be good for the ecosystem and performance if there was more than one compiler and runtime. I think there would be a market for a proprietary compiler and maybe an IDE to go with it — if the performance was better than the open source one. I think this is achievable because as good as Go's performance is now, there's still a lot of headroom. Google isn't exploiting modern CPUs very well, and the linker is not doing extensive LTO. The biggest constraint is the blazing fast compile times. A compiler and toolchain that was able to take some time for optimization might deliver markedly better runtime performance.
- ksherlock 7y ago
  gccgo perhaps?
  - sdrothrock 7y ago
    I don't think that would be appropriate since Golang isn't (to my knowledge) under the GNU license. Maybe gocc?
    - jerf 7y ago
      I think you may misunderstand the nature of the post: https://golang.org/doc/install/gccgo https://golang.org/doc/install/gccgo It's a thing that exists, not a proposal.
      - sdrothrock 7y ago
        I did, in fact! Thanks. :) I thought it was just spitballing an alternative name.
    - slrz 7y ago
      Software licenses don't really matter if you implement the Go language spec with your own code. It's only when you want to share code with other implementations that software licensing enters the picture. Gccgo manages to share most of the standard library with the golang.org implementation, so they appear to have the licensing figured out.
nn3 7y ago
Who would have thought that statically linking hundreds of MB worth of binaries every build causes problems? Maybe some of it can be fixed with a lot of engineering effort, but the fact remains that it is a bad idea to redo so much work every time. Maybe the people who invented shared libraries had a point after all. Of course the real problem is software and dependency bloat, but that is unlikely to ever get fixed.
- dullgiulio 7y ago
  Besides the fact that static linking has a lot of benefits, shared libraries need to be linked too: just this process is called "loading" and the linker is called "loader". Worse, this needs to happen at every executable invokation.
miki123211 7y ago
Anyone else is getting a "you can't view this page because you're offline" error? I don't know why this is happening, how does it even load when I'm supposedly offline?
- warent 7y ago
  It's probably being overloaded with too much traffic. The link is a Google Doc, and singular google docs aren't really great at handling traffic spikes.
warent 7y ago
For anyone having problems loading the doc due to a traffic overload, here's a copy: https://docs.google.com/document/d/1y3k3nLlPpoMf_RAfi-FCpZptDZfrv9Y8W7OzQjfhYJs/edit?usp=sharing https://docs.google.com/document/d/1y3k3nLlPpoMf_RAfi-FCpZpt...
NieDzejkob 7y ago
The article often mentions how the long-lived objects are straining the garbage collector. Wouldn't a generational GC solve this?
- remus 7y ago
  Potentially, but then the linker is just one use case for the GC and GCs are a game of trade offs so switching to a generational GC would impact other use cases.

238 points by rjammala 7y ago

"The original linker was also simpler than it is now and its implementation fit in one Turing award winner’s head, so there’s little abstraction or modularity. Unfortunately, as the linker grew and evolved, it retained its lack of structure, and our sole Turing award winner retired." :)

luuio 7y ago
For those out of the loop, who are they talking about here?
- p7IDD243 7y ago
  Perhaps Ken Thompson?
  - rjammala 7y ago
    Kenneth Lane Thompson is an American pioneer of computer science. Thompson worked at Bell Labs for most of his career where he designed and implemented the original Unix operating system. Books: Queen and Pawn Against Queen, Roycroft's 5-man Chess Endgame Series Awards: Turing Award, National Medal of Technology and Innovation, Japan Prize
- insulanus 7y ago
  Definitely Ken Thompson. :) https://en.wikipedia.org/wiki/Ken_Thompson https://en.wikipedia.org/wiki/Ken_Thompson
eeZah7Ux 7y ago
Bragging doesn't make the language less unremarkable.

derefr 7y ago

> its implementation fit in one Turing award winner’s head What a great unit of measure :)

ChrisSD 7y ago
Wait, how many London buses fit in a Turing award winner's head? And can they all fit in to Wales despite the number of Olympic sized swimming pools already there?
- ggm 7y ago
  Convert to Sydney harbours and add a banana for scale
- solresol 7y ago
  I think we also need to convert that into elephants and blue whales. What's the conversion rate for elephants to London buses?
rootlocus 7y ago
Another great unit of measure is the bus factor.
- heavenlyhash 7y ago
  I think that's part of the dry humor, here.

jstimpfle 7y ago

I don't get it. A linker's task should be straightforward. In essence, it looks up addresses from strings, whose number is bounded by the lines of code written by humans. I think that there must be a lot of incidental complexity if that task somehow becomes a bottleneck. And how can it be that a binary called "cmd/compile" has 170k symbols (that's like, global definitions, right?). Not that that's a huge number in terms of today's computing power, but how many millions of lines of source code does that correspond to? Still, 1M relocations, or 34MB of "Reloc" objects, as indicated, shouldn't be a huge issue to process. Object files should have minimal parsing overhead. Is there any indication how long this takes to link? Shouldn't 1 or 2 secs be sufficient? (assuming 100s of MB/s for sequential disk read/write, and < 1us to store each of the 170k symbols in a hashmap, and < 1us to to look up each of the 1M of relocations). - I don't think mmap should be used if it can be avoided. It means giving up control over what parts of the file are loaded in memory. And from a semantic point of view, that memory region still must be treated differently, since on-disk and in-memory data structures are not the same.

wahern 7y ago
> that's like, global definitions, right? No, not just global definitions. Closures need linking, too. But the linker is doing much more than linking function entry points. Many automatic (on the stack) variables need linking so the GC can (a) trace the object graph and (b) move them when resizing the stack. Likewise, type definitions require metadata generation for GC tracing. And then there's all the debugging data that needs to be generated, which basically involves everything.
- nappy-doo 7y ago
  Go currently does DWARF generation in the linker, but is in the process of moving that into the compiler. Also, most closures linking problems are relatively easy in Go. As far as stack variables for the GC, the stack is exactly scanned for all but the last frame (I believe), but conservatively scanned for the last frame. This makes it much easier. Types -- you're partially right. It is mostly all generated in the compiler, and deduped by the linker. There's no reason Go's linker couldn't be much simpler and likely even simpler than a standard C linker. You might argue that Go will give up things like LTO, but Go designs out lots of the LTO problem (not PGO) by nature of how packages work, and the fact that there aren't cyclic dependencies. Overall, the Go linker could be quite simple. It just needs some rework is all. :)

jacobush 7y ago

I hoped they would fix how the linker uses private OS calls.

deleted 7y ago
[deleted]
nappy-doo 7y ago
Which private OS calls. Can you site what they are?
- kccqzy 7y ago
  I'm guessing GP refers to the fact that Go binaries use raw system calls even on operating systems where the syscall boundary is considered a private implementation detail (which is the case on Windows and macOS).
  - cpuguy83 7y ago
    On Mac, since go1.11, Go uses libsystem.
  - 4ad 7y ago
    The Windows port never used raw system calls. The Solaris port never used raw system calls. The MacOS port doesn't use raw system calls since go 1.11. The remaining ports for Linux/BSDs/Plan 9 have a stable syscall interface.
- jbn 7y ago
  I think you mean "cite".
  - nappy-doo 7y ago
    I do, but will leave it because I stand by me inability to spell. Thanks!
    - 7y ago

slx26 7y ago

I love to see that at the end the document the plugins package is mentioned. Currently it doesn't have support for Windows (among other issues), so it's really good to hear that the work in the linker might eventually improve the situation with plugins too, and that at least it's being kept in mind. On golang, interfaces being implemented implicitly is not everyone's cup of tea, and I can understand why, but when you combine that property with plugins, you can make really powerful plugin systems that use very simple code to work.

nappy-doo 7y ago
Plugins are hard, and very OS dependent. There's plenty about the current system that is fragile, and likely to cause problems. I think it's likely the linker rework needs to happen before plugins can be properly addressed.
- jart 7y ago
  This doc offers great depth on why dynamic shared objects are so hard: https://software.intel.com/sites/default/files/m/a/1/e/dsohowto.pdf https://software.intel.com/sites/default/files/m/a/1/e/dsoho... and that's just for Linux & co. I'm surprised the Go team chose to add support for DSOs.
axaxs 7y ago
agreed. Just yesterday I was reading about shared object support in Go, and it hasn't improved since inception. Sometimes I just want an object/module, and not a git repo. Even VB had this down.
- pushpop 7y ago
  As much as I think VisualBasic gets a lot of unfair criticism, I really don’t think the OCX model is something worth admiring. Sure, when it worked it was great but all too often they would fail between OS updates, language patches or even just installing other applications if they happened to use different OCXs. What’s more, those bastard files required registry hacks to work. If anything, OCX is an example of how not to do shared libraries.

AlexB138 7y ago

Any recommendations on resources to better understand the build process for compiled languages in general, and Go in particular?

ori_b 7y ago
https://www.amazon.com/dp/1558604960 https://www.amazon.com/dp/1558604960
- journalctl 7y ago
  I just bought this! It’s fantastic and definitely still more than relevant for understanding the subject.
nappy-doo 7y ago
As weird as it sounds, Go is one of the easiest to understand compilers I've ever worked on. The language, and the implementation are relatively easy to understand and follow most of the standard compiler design and implementation texts. Similarly, since Go is so opinionated, the code is easy to read.
wahern 7y ago
Write your own simple virtual machine[1] and a tiny assembler which accepts symbolic names for jump targets (e.g. goto LABEL). You'll need a link phase to resolve the symbolic names, and all will become clear. A few hours of time and anything involving linking, addressing[2], and so much more will instantly become intuitive to you. [1] Array of opcodes indexed with a program counter (pc) with a giant switch statement for executing each opcode. A simple stack for data implemented as an array and a stack pointer/index (sp). There are plenty of examples online, just make sure you actually implement things yourself, and only go to the examples to answer questions you arrive at yourself. If you understand loops and arrays implementing the VM is trivial. If you can parse a text file line-by-line and split words, implementing the assembler is trivial. Working out the linking might take some thinking, but that's the point. It's like one night of work, max, unless you really get sucked in ;) [2] E.g. Harvard architecture vs. von Neumann architecture

benesch 7y ago

What I would give for the developers of the Go toolchain to have spent the last decade improving GCC or LLVM instead of their own bespoke toolchain. In many ways Go seems like an excuse for Google to fund the continued development of Plan 9. Three of the five most influential people on the Go team (Ken Thompson, Rob Pike, and Russ Cox) were heavily involved in Plan 9. And it shows. Go's toolchain is a direct descendant of the Plan 9 toolchain; in fact, the Go language is really just an evolution of the special C dialect that Plan 9 used [1]. Indeed, for a while, the Go compiler was written in this special dialect of C, and so building Go required building a C compiler (!) that could compile this custom dialect of C, and using that to compile the Go compiler [2]. By all rights, Plan 9 was an interesting research project, and seems well loved by those familiar with it. (I'm not personally familiar; it was well before my time.) But it never took off. What we ended up with is Linux, macOS, and Windows. Go very much wants to be Plan 9. Sure, it's not a full-fledged operating system. But it's a linker, assembler, compiler, binutils, and scheduler. All it asks of the host system is memory management, networking, and filesystem support, and it will happily replace your system's DNS resolution with a pure Go version if you ask it to [3]. I wouldn't be surprised if Go ships its own TCP/IP stack someday [4]. This is, in my opinion, craziness. What other language ships its own assembler?! [5] To make matters worse, the assembly syntax is largely undocumented, and what is documented are the strange, unnecessary quirks, like > Instructions, registers, and assembler directives are always in UPPER CASE to remind you that assembly programming is a fraught endeavor. (Exception: the g register renaming on ARM.) > In the general case, the frame size is followed by an argument size, separated by a minus sign. (It's not a subtraction, just idiosyncratic syntax.) > In Go object files and binaries, the full name of a symbol is the package path followed by a period and the symbol name: fmt.Printf or math/rand.Int. Because the assembler's parser treats period and slash as punctuation, those strings cannot be used directly as identifier names. Instead, the assembler allows the middle dot character U+00B7 and the division slash U+2215 in identifiers and rewrites them to plain period and slash. The excuse for the custom toolchain has always been twofold, that a) LLVM is too slow, and fast compiles are one of Go's main features, and b) that the core team was too unfamiliar with GCC/LLVM, at least in the early days, and attempting to build Go on top of LLVM would have slowed the speed of innovation to a degree that Go might not exist [6]. I've always been skeptical of argument (b). After all, one of Go's creators literally won a Turing award, as this document not-so-subtly mentions. I'm quite sure they could have figured out how to build an LLVM frontend, given the desire. Rust, for example, is quite a bit more complicated than Go, and Mozilla's developers have had no trouble integrating with LLVM. I suspect the real reason was that hacking on the Plan 9 toolchain was more fun and more familiar—which is a very valid personal reason to work on something! But it doesn't mean it was the right strategic decision. I will say that (a) is valid. I recently switched from writing Go to writing Rust, and I miss the compile times of Go desperately. That said—and this is what I can't get past—the state of compilers would be much better off if the folks on the Go team had invested more in improving the compile and link times of LLVM or GCC. Every improvement to lld wouldn't just speed up compiles for Go; it would speed up compiles for C, C++, Swift, Rust, Fortran, Kotlin, and anything else with an LLVM frontend. In the last year or so, the gollvm project [7] (which is exactly what you'd expect–a Go frontend for LLVM) has seen some very active development, and I'm following along excitedly. Unfortunately I still can't quite tell whether it's Than McIntosh's 20% time project or an actual staffed project of Google's, albeit a small time one. (There are really only two committers, Than and Cherry Zhang.) There are so many optimizations that will likely never be added to gc, like a register-based calling convention [8] and autovectorization, that you essentially get for "free" (i.e., with a bit of plumbing from the frontend) with a mature toolchain like LLVM. There are not many folks who have the knowledge and expertise to work on compilers and linkers these days, and those that do can command high salaries. Google is in the luxurious position of being able to afford many dozens of these people. I just wish that someone with the power to effect change at Google would realize that the priorities are backwards. gccgo/gollvm are where the primary investment should be occurring, and the gc toolchain should be a side project that makes debug builds fast... not the production compiler, where the quality of the object code is the primary objective. [0]: https://dave.cheney.net/2013/10/15/how-does-the-go-build-command-work https://dave.cheney.net/2013/10/15/how-does-the-go-build-com... [1]: http://doc.cat-v.org/plan_9/programming/c_programming_in_plan_9 http://doc.cat-v.org/plan_9/programming/c_programming_in_pla... [2]: https://docs.google.com/document/d/1P3BLR31VA8cvLJLfMibSuTdwTuF7WWLux71CYD0eeD8/edit https://docs.google.com/document/d/1P3BLR31VA8cvLJLfMibSuTdw... [3]: https://golang.org/pkg/net/ https://golang.org/pkg/net/ [4]: https://github.com/google/netstack https://github.com/google/netstack [5]: https://golang.org/doc/asm https://golang.org/doc/asm [6]: https://golang.org/doc/faq#What_compiler_technology_is_used_to_build_the_compilers https://golang.org/doc/faq#What_compiler_technology_is_used_... [7]: https://go.googlesource.com/gollvm/ https://go.googlesource.com/gollvm/ [8]: https://github.com/golang/go/issues/18597 https://github.com/golang/go/issues/18597

Solar19 7y ago

I like it. What would also be cool is another Go compiler and linker, developed by a different team not at Google. It would be good for the ecosystem and performance if there was more than one compiler and runtime. I think there would be a market for a proprietary compiler and maybe an IDE to go with it — if the performance was better than the open source one. I think this is achievable because as good as Go's performance is now, there's still a lot of headroom. Google isn't exploiting modern CPUs very well, and the linker is not doing extensive LTO. The biggest constraint is the blazing fast compile times. A compiler and toolchain that was able to take some time for optimization might deliver markedly better runtime performance.

ksherlock 7y ago
gccgo perhaps?
- sdrothrock 7y ago
  I don't think that would be appropriate since Golang isn't (to my knowledge) under the GNU license. Maybe gocc?
  - jerf 7y ago
    I think you may misunderstand the nature of the post: https://golang.org/doc/install/gccgo https://golang.org/doc/install/gccgo It's a thing that exists, not a proposal.
    - sdrothrock 7y ago
      I did, in fact! Thanks. :) I thought it was just spitballing an alternative name.
  - slrz 7y ago
    Software licenses don't really matter if you implement the Go language spec with your own code. It's only when you want to share code with other implementations that software licensing enters the picture. Gccgo manages to share most of the standard library with the golang.org implementation, so they appear to have the licensing figured out.

nn3 7y ago

Who would have thought that statically linking hundreds of MB worth of binaries every build causes problems? Maybe some of it can be fixed with a lot of engineering effort, but the fact remains that it is a bad idea to redo so much work every time. Maybe the people who invented shared libraries had a point after all. Of course the real problem is software and dependency bloat, but that is unlikely to ever get fixed.

dullgiulio 7y ago
Besides the fact that static linking has a lot of benefits, shared libraries need to be linked too: just this process is called "loading" and the linker is called "loader". Worse, this needs to happen at every executable invokation.

miki123211 7y ago

Anyone else is getting a "you can't view this page because you're offline" error? I don't know why this is happening, how does it even load when I'm supposedly offline?

warent 7y ago
It's probably being overloaded with too much traffic. The link is a Google Doc, and singular google docs aren't really great at handling traffic spikes.

warent 7y ago

For anyone having problems loading the doc due to a traffic overload, here's a copy: https://docs.google.com/document/d/1y3k3nLlPpoMf_RAfi-FCpZptDZfrv9Y8W7OzQjfhYJs/edit?usp=sharing https://docs.google.com/document/d/1y3k3nLlPpoMf_RAfi-FCpZpt...

NieDzejkob 7y ago

The article often mentions how the long-lived objects are straining the garbage collector. Wouldn't a generational GC solve this?

remus 7y ago
Potentially, but then the linker is just one use case for the GC and GCs are a game of trade offs so switching to a generational GC would impact other use cases.