5 ms·

Sorry for the noob question, but could someone help me understand how this is (conceptual) possible? I get that you can write a program which compiles language

Sorry for the noob question, but could someone help me understand how this is (conceptual) possible? I get that you can write a program which compiles language X to machine code, e.g. how python interpreting Go to write the assembly necessary would be technically possible. My question is, how is the compiler generated? If it's written in c, gcc -o compiler, but what is the piece I'm missing when it comes to Go compiles a Go compiler -- or is this still done as "here's the machine code, now compile the Go runtime"? I suppose my question is a similie to "the chicken or the egg?".

ominous_prime 11y ago
Simple, you need a working version of go1.4 to compile go1.5. The bootstrapping process can therefore take advantage of the fact that go1.4 will continue to build as it does today, and use that to get a working go installation to build go1.5+.
- klapinat0r 11y ago
  Ah, great. Thanks for the explanation :)
comex 11y ago
On a related note, while Go at least has Go 1.4 available to allow bootstrapping from a C compiler for the near future (see [1] for more detail), many languages have no easy way to do so. For example, Rust's compiler was originally written in OCaml, and Nim's in Object Pascal, but both compilers were migrated to be written in the respective languages themselves, and AFAIK both languages have since evolved alongside their compilers for long enough that bootstrapping all the way from before said migrations to the present would require a massive chain of newer and newer compiler versions - not an undertaking anyone really wants to carry out. Instead, any Linux distribution etc. that wants to integrate the language into its build system has to start by importing a binary from somewhere else. In general this isn't a problem, because that binary is used only to compile the compiler from source (and then that compiler recompiles itself, usually), so any bugs in it are unlikely to affect the final product. However, the topic tends to come up of Ken Thompson's famous paper [2] describing a hypothetical scenario where a compiler binary is intentionally backdoored to insert a vulnerability when it's compiling some security-critical program, plus a copy of the same backdoor whenever it detects it's compiling itself. In that case, the backdoor could theoretically infect the final result of a bootstrap, despite it being compiled from pristine source; no attack of that nature has ever been detected in the wild, though. [1] https://docs.google.com/document/d/1OaatvGhEAq7VseQ9kkavxKNAfepWy2yhPUBs96FGV28/edit https://docs.google.com/document/d/1OaatvGhEAq7VseQ9kkavxKNA... [2] https://www.ece.cmu.edu/~ganger/712.fall02/papers/p761-thompson.pdf https://www.ece.cmu.edu/~ganger/712.fall02/papers/p761-thomp...
- hobarrera 11y ago
  What also intrigues me is how one would port to a new architecture. You've got no go1.4 there to boostrap you (new architecture), and you can't get the binary because you're trying to build the first one.
  - comex 11y ago
    Easy, you cross compile from Go on a different architecture. Go apparently has a good cross compilation story, though I don't have experience with it; honestly IMO the whole perception that cross compiling is an unusual or difficult thing comes from poorly designed Unix build systems. Though I suppose this would make a fully automated bootstrapping system a bit messier. (Do the first build in an emulator?)
    - hobarrera 11y ago
      Thanks for you answer. I hadn't though about cross compilation, and I guess that's the sanest approach, when possible.
ufmace 11y ago
I understand that the standard build/release procedure for compilers which are the primary compiler for the language they are written in is to: Build the new version of the compiler with the previous version of the compiler Rebuild the new version of the compiler with the new compiler from the previous step, that was built with the old compiler Rebuild the new compiler with the new compiler from the previous step, that was also built with the new compiler, and that's your official binary The initial creation of the language requires somebody to write a compiler for it in assembly or in some other language that already has one, but once you get that first one built, all the rest could be in their own language.

ominous_prime 11y ago

Simple, you need a working version of go1.4 to compile go1.5. The bootstrapping process can therefore take advantage of the fact that go1.4 will continue to build as it does today, and use that to get a working go installation to build go1.5+.

klapinat0r 11y ago
Ah, great. Thanks for the explanation :)

comex 11y ago

On a related note, while Go at least has Go 1.4 available to allow bootstrapping from a C compiler for the near future (see [1] for more detail), many languages have no easy way to do so. For example, Rust's compiler was originally written in OCaml, and Nim's in Object Pascal, but both compilers were migrated to be written in the respective languages themselves, and AFAIK both languages have since evolved alongside their compilers for long enough that bootstrapping all the way from before said migrations to the present would require a massive chain of newer and newer compiler versions - not an undertaking anyone really wants to carry out. Instead, any Linux distribution etc. that wants to integrate the language into its build system has to start by importing a binary from somewhere else. In general this isn't a problem, because that binary is used only to compile the compiler from source (and then that compiler recompiles itself, usually), so any bugs in it are unlikely to affect the final product. However, the topic tends to come up of Ken Thompson's famous paper [2] describing a hypothetical scenario where a compiler binary is intentionally backdoored to insert a vulnerability when it's compiling some security-critical program, plus a copy of the same backdoor whenever it detects it's compiling itself. In that case, the backdoor could theoretically infect the final result of a bootstrap, despite it being compiled from pristine source; no attack of that nature has ever been detected in the wild, though. [1] https://docs.google.com/document/d/1OaatvGhEAq7VseQ9kkavxKNAfepWy2yhPUBs96FGV28/edit https://docs.google.com/document/d/1OaatvGhEAq7VseQ9kkavxKNA... [2] https://www.ece.cmu.edu/~ganger/712.fall02/papers/p761-thompson.pdf https://www.ece.cmu.edu/~ganger/712.fall02/papers/p761-thomp...

hobarrera 11y ago
What also intrigues me is how one would port to a new architecture. You've got no go1.4 there to boostrap you (new architecture), and you can't get the binary because you're trying to build the first one.
- comex 11y ago
  Easy, you cross compile from Go on a different architecture. Go apparently has a good cross compilation story, though I don't have experience with it; honestly IMO the whole perception that cross compiling is an unusual or difficult thing comes from poorly designed Unix build systems. Though I suppose this would make a fully automated bootstrapping system a bit messier. (Do the first build in an emulator?)
  - hobarrera 11y ago
    Thanks for you answer. I hadn't though about cross compilation, and I guess that's the sanest approach, when possible.

ufmace 11y ago

I understand that the standard build/release procedure for compilers which are the primary compiler for the language they are written in is to: Build the new version of the compiler with the previous version of the compiler Rebuild the new version of the compiler with the new compiler from the previous step, that was built with the old compiler Rebuild the new compiler with the new compiler from the previous step, that was also built with the new compiler, and that's your official binary The initial creation of the language requires somebody to write a compiler for it in assembly or in some other language that already has one, but once you get that first one built, all the rest could be in their own language.