9 ms·

Note, for the record, none of these are correct except for the last one: 1. LLVM's SSA based compiler currently uses more linear time algorithms than Go's SSA

Note, for the record, none of these are correct except for the last one: 1. LLVM's SSA based compiler currently uses more linear time algorithms than Go's SSA backend (IE phi placement, etc), and in more places. So yeah. While parts of LLVM definitely could be sped up (particularly the backend), it was originally designed with fast compile speed in mind, just like go! Given how many years it's taken so far, you easily could have gotten the same thing out of LLVM. Would it have been more work? Probably, but not much. Are the other reasons not to do it? Yes. But the idea that their compiler is somehow the only one designed with fast compile time in mind is, honestly, pretty arrogant 2. Stack maps have been supported for quite a while, and statepoint support now exists and is used by real people in real places. But i'll also just point out he's never actually asked, or talked to the LLVM folks about it (search the mailing list, see if you can find any mail about it). Note: I honestly don't care whether they write their own compiler or not. I suspect it's better for them to do it because of the community they want to build, because of how much control they want over their toolchain, etc. But i'm not a fan of saying "someone else's infrastructure sucks, so we are doing it", when 1. That's not the real reason as far as i can tell 2. People are using, in production, the very things they claim don't exist (and they've been told this, it turns out) 3. Answers about linear time algorithms are honestly just BS.

danieldk 11y ago
Agreed. Though another argument in favor was not cited: the SSA backend is in Go, so that means that anyone who knows Go can also hack on the SSA backend. With LLVM, they would have to know Go's internals and C++. Of course, there is also gccgo and llgo, so once the SSA backend is in stable Go, useful comparisons can be made.
- disposeofnick9 11y ago
  Using llvm might introduce a dependency, but it also gives all the LLVM advantages/disadvantages across platforms for "free" (some work, fewer reinvented wheels) and more toolchain utilities.
- jheriko 11y ago
  i think that this is a gimmick rather than a reason. languages that have their own tools built with themselves are cool... but it has zero practical value imo. the number of C/C++ programmers interested in Go and capable of contributing meaningfully far outweighs the number of programmers who do not know C/C++ and are interested in Go
wyldfire 11y ago
Is "Stack maps" the same as "segmented stacks" that Mr Cox refers to?
- riscy 11y ago
  No, stack maps provide information about each stack frame (typically using the return instruction pointer left in that frame) to allow for the garbage collector to determine what is a pointer or not in that frame.
- DannyBee 11y ago
  Segmented stacks are basically what safestack offers, and that is also something llvm does :)
  - riscy 11y ago
    No it's not. Segmented stacks are a way of allocating smaller stack segments that are linked together with pointers and kept in the heap and is a means of implementing coroutines. Safestack is more complicated and it segregates the stack into 2 parts to prevent stack buffer overflow attacks.
    - deleted 11y ago
      [deleted]
- kibwen 11y ago
  Note that Go no longer uses segmented stacks (and hasn't since 1.2). It uses contiguous, copyable stacks: http://agis.io/2014/03/25/contiguous-stacks-in-go.html http://agis.io/2014/03/25/contiguous-stacks-in-go.html
geodel 11y ago
You say it is arrogant but is it untrue? Rust uses LLVM backend and it takes order of magnitude longer to compile compare to Go.
- DannyBee 11y ago
  It is untrue, given the same amount of work put into both (IE if they had spent their time doing llvm instead of their own ssa backend). Remember the SSA backend of Go was slower than the normal go compiler for the vast majority of it's life as well. In any case, people like to build shiny new things they think they can do better at, instead of fixing things that are, in actuality, probably better for what they want but need a little work. The truth is that there is no magic in building good compilers, so it's not like anyone comes along, waves a wand, and produces a compiler that is X times faster or better than anyone else's. I can think of a lot of good reasons to write their compiler in go, and not use LLVM. Technical reasons are ... not any of them :)
  - geodel 11y ago
    > Remember the SSA backend of Go was slower than the normal go compiler for the vast majority of it's life as well. Go 1.7 in August will be first release with SSA backend. So I do not understand what is that vast majority of life besides development phase of 1 year or so. > In any case, people like to build shiny new things they think they can do better at, instead of fixing things that are, in actuality, probably better for what they want but need a little work. I agree with that. It could just be a comment on our society. > I can think of a lot of good reasons to write their compiler in go, and not use LLVM. Technical reasons are ... not any of them :) I think goroutine and segmented stacks support was mentioned as technical reason.
- pcwalton 11y ago
  The reasons why Rust is slower to compile are basically (a) typechecking takes longer; (b) LLVM does way way more optimization than Go's backend does. (b) applies even at -O0 because it has to go through LLVM IR to SelectionDAG to MachineInstrs, while Go 6g/8g pre-SSA can just go from AST to MachineInstrs. (We can't use FastISel at the moment because FastISel has no support for unwinding, among other minor reasons.) If Go adds all the optimizations that LLVM does, then I have no reason to believe that it won't end up just as slow as LLVM. There's active work to fix both (a) and (b): for (a) there are ideas for an O(1) unifier in typechecking, and for (b) we're working on MIR (which is now able to self-host) to do higher-level optimizations on a language-specific IR to avoid punting them to LLVM, where it may take more work to figure out. (For instance, we can do inlining on polymorphic functions, reducing the time complexity from O(number of instantiations) to O(1).) But the biggest wins will come from incremental compilation, which there's been a ton of progress on lately and might just fix the compilation time issues once and for all in practice.
vvanders 11y ago
Not Invented Here is a powerful thing.
azakai 11y ago
Yes, LLVM was designed with speed in mind. However, look at how much faster compile times are in JS VMs compared to LLVM; LLVM's slow compile times are why none of them use it, even though they tried (e.g. WebKit which is dropping LLVM). A lot of SSA-based JITs are just a lot faster than LLVM. I think it's a matter of tradeoffs. LLVM has multiple IRs on the way to machine code, some quite large, which let it emit pretty good output, on par with gcc. But other designs are possible that avoid a lot of that overhead, and can be far faster than LLVM, like JS VMs, B3 [1] and SubZero [2]. And clang is overall more or less comparable with gcc these days in terms of speed, it's no longer clearly faster [3] (although that also takes into account the frontend). With all that said, I agree with you that making such a decision without talking to LLVM is a little silly. But, I doubt it would change the outcome: If Go wants fast compile times, has the resources to write its own compiler, and is ok with trading off throughput for compile time, then it's a reasonable decision to avoid LLVM. [1] https://webkit.org/blog/5852/introducing-the-b3-jit-compiler/ https://webkit.org/blog/5852/introducing-the-b3-jit-compiler... [2] https://github.com/stichnot/subzero https://github.com/stichnot/subzero [3] http://hubicka.blogspot.com/2016/03/building-libreoffice-with-gcc-6-and-lto.html http://hubicka.blogspot.com/2016/03/building-libreoffice-wit...
- pcwalton 11y ago
  You can certainly go faster than LLVM if you're willing to sacrifice code quality (which B3 and SubZero do... or for the extreme example there is no way on Earth LLVM will catch up to TCC). But I think that DannyBee is totally right that "we can go faster than LLVM at the same level of code quality just by rewriting it" is magical thinking. As far as I'm aware, there are no design decisions in LLVM that are pervasive mistakes that sacrifice compilation speed for no reason [1]. If there's nothing you can point to and say "THIS is what I'm going to do better than LLVM", then you're probably going to end up at the same speed as LLVM. The parent comment's point is that none of the attempts to answer that question have been valid: so far the reasons cited have been based on misconceptions about what LLVM provides. [1] Well, OK, I'm not a fan of SelectionDAGISel's interpreter. But switching that to AOT won't result in huge differences in compile times. (Note that it's been a couple of years since I really worked with LLVM closely.)
Locke1689 11y ago
To be fair, there are some limitations around stack maps that make certain things difficult (see [1] for an example), but I agree that there's nothing stopping them from just providing patches or feature requests to LLVM to improve stack map support for their use case (as [1] is doing). [1] http://lists.llvm.org/pipermail/llvm-dev/2015-July/087832.html http://lists.llvm.org/pipermail/llvm-dev/2015-July/087832.ht...

danieldk 11y ago

Agreed. Though another argument in favor was not cited: the SSA backend is in Go, so that means that anyone who knows Go can also hack on the SSA backend. With LLVM, they would have to know Go's internals and C++. Of course, there is also gccgo and llgo, so once the SSA backend is in stable Go, useful comparisons can be made.

disposeofnick9 11y ago
Using llvm might introduce a dependency, but it also gives all the LLVM advantages/disadvantages across platforms for "free" (some work, fewer reinvented wheels) and more toolchain utilities.
jheriko 11y ago
i think that this is a gimmick rather than a reason. languages that have their own tools built with themselves are cool... but it has zero practical value imo. the number of C/C++ programmers interested in Go and capable of contributing meaningfully far outweighs the number of programmers who do not know C/C++ and are interested in Go

wyldfire 11y ago

Is "Stack maps" the same as "segmented stacks" that Mr Cox refers to?

riscy 11y ago
No, stack maps provide information about each stack frame (typically using the return instruction pointer left in that frame) to allow for the garbage collector to determine what is a pointer or not in that frame.
DannyBee 11y ago
Segmented stacks are basically what safestack offers, and that is also something llvm does :)
- riscy 11y ago
  No it's not. Segmented stacks are a way of allocating smaller stack segments that are linked together with pointers and kept in the heap and is a means of implementing coroutines. Safestack is more complicated and it segregates the stack into 2 parts to prevent stack buffer overflow attacks.
  - deleted 11y ago
    [deleted]
kibwen 11y ago
Note that Go no longer uses segmented stacks (and hasn't since 1.2). It uses contiguous, copyable stacks: http://agis.io/2014/03/25/contiguous-stacks-in-go.html http://agis.io/2014/03/25/contiguous-stacks-in-go.html

geodel 11y ago

You say it is arrogant but is it untrue? Rust uses LLVM backend and it takes order of magnitude longer to compile compare to Go.

DannyBee 11y ago
It is untrue, given the same amount of work put into both (IE if they had spent their time doing llvm instead of their own ssa backend). Remember the SSA backend of Go was slower than the normal go compiler for the vast majority of it's life as well. In any case, people like to build shiny new things they think they can do better at, instead of fixing things that are, in actuality, probably better for what they want but need a little work. The truth is that there is no magic in building good compilers, so it's not like anyone comes along, waves a wand, and produces a compiler that is X times faster or better than anyone else's. I can think of a lot of good reasons to write their compiler in go, and not use LLVM. Technical reasons are ... not any of them :)
- geodel 11y ago
  > Remember the SSA backend of Go was slower than the normal go compiler for the vast majority of it's life as well. Go 1.7 in August will be first release with SSA backend. So I do not understand what is that vast majority of life besides development phase of 1 year or so. > In any case, people like to build shiny new things they think they can do better at, instead of fixing things that are, in actuality, probably better for what they want but need a little work. I agree with that. It could just be a comment on our society. > I can think of a lot of good reasons to write their compiler in go, and not use LLVM. Technical reasons are ... not any of them :) I think goroutine and segmented stacks support was mentioned as technical reason.
pcwalton 11y ago
The reasons why Rust is slower to compile are basically (a) typechecking takes longer; (b) LLVM does way way more optimization than Go's backend does. (b) applies even at -O0 because it has to go through LLVM IR to SelectionDAG to MachineInstrs, while Go 6g/8g pre-SSA can just go from AST to MachineInstrs. (We can't use FastISel at the moment because FastISel has no support for unwinding, among other minor reasons.) If Go adds all the optimizations that LLVM does, then I have no reason to believe that it won't end up just as slow as LLVM. There's active work to fix both (a) and (b): for (a) there are ideas for an O(1) unifier in typechecking, and for (b) we're working on MIR (which is now able to self-host) to do higher-level optimizations on a language-specific IR to avoid punting them to LLVM, where it may take more work to figure out. (For instance, we can do inlining on polymorphic functions, reducing the time complexity from O(number of instantiations) to O(1).) But the biggest wins will come from incremental compilation, which there's been a ton of progress on lately and might just fix the compilation time issues once and for all in practice.

vvanders 11y ago

Not Invented Here is a powerful thing.

azakai 11y ago

Yes, LLVM was designed with speed in mind. However, look at how much faster compile times are in JS VMs compared to LLVM; LLVM's slow compile times are why none of them use it, even though they tried (e.g. WebKit which is dropping LLVM). A lot of SSA-based JITs are just a lot faster than LLVM. I think it's a matter of tradeoffs. LLVM has multiple IRs on the way to machine code, some quite large, which let it emit pretty good output, on par with gcc. But other designs are possible that avoid a lot of that overhead, and can be far faster than LLVM, like JS VMs, B3 [1] and SubZero [2]. And clang is overall more or less comparable with gcc these days in terms of speed, it's no longer clearly faster [3] (although that also takes into account the frontend). With all that said, I agree with you that making such a decision without talking to LLVM is a little silly. But, I doubt it would change the outcome: If Go wants fast compile times, has the resources to write its own compiler, and is ok with trading off throughput for compile time, then it's a reasonable decision to avoid LLVM. [1] https://webkit.org/blog/5852/introducing-the-b3-jit-compiler/ https://webkit.org/blog/5852/introducing-the-b3-jit-compiler... [2] https://github.com/stichnot/subzero https://github.com/stichnot/subzero [3] http://hubicka.blogspot.com/2016/03/building-libreoffice-with-gcc-6-and-lto.html http://hubicka.blogspot.com/2016/03/building-libreoffice-wit...

pcwalton 11y ago
You can certainly go faster than LLVM if you're willing to sacrifice code quality (which B3 and SubZero do... or for the extreme example there is no way on Earth LLVM will catch up to TCC). But I think that DannyBee is totally right that "we can go faster than LLVM at the same level of code quality just by rewriting it" is magical thinking. As far as I'm aware, there are no design decisions in LLVM that are pervasive mistakes that sacrifice compilation speed for no reason [1]. If there's nothing you can point to and say "THIS is what I'm going to do better than LLVM", then you're probably going to end up at the same speed as LLVM. The parent comment's point is that none of the attempts to answer that question have been valid: so far the reasons cited have been based on misconceptions about what LLVM provides. [1] Well, OK, I'm not a fan of SelectionDAGISel's interpreter. But switching that to AOT won't result in huge differences in compile times. (Note that it's been a couple of years since I really worked with LLVM closely.)

Locke1689 11y ago

To be fair, there are some limitations around stack maps that make certain things difficult (see [1] for an example), but I agree that there's nothing stopping them from just providing patches or feature requests to LLVM to improve stack map support for their use case (as [1] is doing). [1] http://lists.llvm.org/pipermail/llvm-dev/2015-July/087832.html http://lists.llvm.org/pipermail/llvm-dev/2015-July/087832.ht...