6 ms·

Porting the Go compiler from C to Go

212 points by sqs 13y ago

beliu 13y ago
Author of the post here. Happy to answer any questions I can, and FYI, we (Sourcegraph) are liveblogging all of GopherCon at http://gophercon.sourcegraph.com http://gophercon.sourcegraph.com. Let us know if you have any questions or find it useful!
- nfoz 13y ago
  Are you at all concerned about the possibility of the Trusting Trust[1] problem manifesting in any of your Go compilers? [1] http://cm.bell-labs.com/who/ken/trust.html http://cm.bell-labs.com/who/ken/trust.html
  - 0xdeadbeefbabe 13y ago
    Since this is an exercise divorced from reality, the usual vehicle was FORTRAN. Dang it I cheated and looked up Rus's quine. For anyone not wanting to cheat a good hint would be "tail recursion"
  - zellyn 13y ago
    It was suggested long ago that having multiple compilers would make it incredibly difficult to play Trusting Trust games.
- codezero 13y ago
  You might want to clarify that you aren't Russ Cox and that this post is a summary of a presentation he gave.
  - sqs 13y ago
    Thanks for mentioning this. I initially included "(unofficial liveblog post)" in the HN title, but it apparently was edited out by a moderator. So, I added a clarification to the top of the post itself.
stcredzero 13y ago
Automated rewrite FTW! This can help you avoid freezing a project while it is being ported. Also, if you have a code base with its own idioms, then those idioms can be matched and translated, which can produce cleaner target code.
- adient 13y ago
  Go 1.0 spec is already frozen and will not change any time soon.
  - szabba 13y ago
    I believe he referred mostyl to freezing the efforts on impoving the implementation.
    - adient 13y ago
      Implementing the compiler in Go becomes the effort to improve the implementation. Obvious, no?
    - stcredzero 13y ago
      I was also talking about projects and porting in general. Incidentally, I've been paid to do exactly what we're discussing. (Porting a production program using automated translation.) It works.
ANTSANTS 13y ago
>A Union is like a struct, but you’re only supposed to use one value >(they all occupy the same space in memory). It’s up to the programmer to know which variable to use. > There’s a joke in some of the original C code: > #define struct union /* Great space saver */ > This inspired a solution: > #define union struct /* keeps code correct, just wastes some space */ Somewhere in Scotland, a sum type sheds a single tear.
- piokuc 13y ago
  > #define union struct /* keeps code correct, just wastes some space */ Not always, though...
  - rsc 13y ago
    Yes, always. And if you don't believe me, it's not my trick. I learned it from Dennis Ritchie (he was thinking about a C to Limbo converter).
    - andybalholm 13y ago
      It works when unions are used correctly. If they are used to deliberately subvert the type system (e.g., put in an int, get an array of char out) it doesn't work. But C has so many other ways to subvert the type system that there's no need to do that. Probably about the only place you'll see them used like that is in the code produced by web2c in compiling TeX. Knuth used variant records a lot to get around Pascal's type safety, and they get translated to unions.
      - piokuc 13y ago
        That's why I said "not always". Here is an example which will print different result if you define #define union struct Here you go: #include <stdio.h> union U { char a; char b; }; int main() { U u; u.a = 13; u.b = 10; printf("%d\n", int(u.a)); return 0; }
Shamanmuni 13y ago
It's great that they are aiming for an automated conversion from C to Go. It's clear they aspire to convert their code which was written in a certain way. But I think it would be a huge boost in Go usage if they could eventually aim to transpile any C code into Go code. A little dream of mine would be if in the future when Rust is stable Mozilla developed a transpiler from C++ to Rust. That would be brilliant. By the way, all the other talks at GopherCon seem pretty interesting, I hope someone uploads videos of them soon.
- NateDad 13y ago
  It has been stated that there will be videos available, but I'd not expect them until next week sometime (when the organizers get home and have recuperated).
- jerf 13y ago
  It is unclear how one would compile arbitrary C code into useful Go. The stereotyped conventions of well-written compiler code allows for a more idiomatic translation than a general translator could ever aspire to. C++ to Rust would be even crazier. (Transpile is a silly word. It's "compile". Compiling already trans-es.)
  - stcredzero 13y ago
    Good automated translations involve matching and translating the project idioms. This results in code that fits the target language's language-idioms. Automated translation only at the level of language constructs results in odd looking generated code.
  - Shamanmuni 13y ago
    I know it's crazy difficult to do it, that's why I was talking about dreams. But maybe a set of guidelines about translatable C code plus a translator which is good enough for a variety of cases would make refactoring the resulting code such a manageable task that many projects could consider switching to Go directly. You are technically correct in that compile already implicates a translation, but we usually use that term refering to a translation into a lower level language and not another language at the same level. You can say transpiler or source-to-source compiler for those cases and I think it's clearer and more accurate for the reader.
SixSigma 13y ago
Russ' paper on the subject from Dec 2013 https://docs.google.com/document/d/1P3BLR31VA8cvLJLfMibSuTdwTuF7WWLux71CYD0eeD8/preview?sle=true&pli=1# https://docs.google.com/document/d/1P3BLR31VA8cvLJLfMibSuTdw...
thinkpad20 13y ago
> There are currently 1032 goto statements in the Go compiler jumping to 241 labels. Wow, that's really striking. I know that goto statements have their uses but for something written in the last couple of years to have over a thousand of them is very surprising (it might not be surprising at all for those who write C code all the time). I guess they're mostly just for error handlers?
- azernik 13y ago
  Yeah, there are a lot of them in e.g. Linux kernel code for jumping to end-of-function cleanup. I suspect that in the process of translation to Go most will be replaced by the defer keyword.
- jgrahamc 13y ago
  For example: http://golang.org/src/cmd/6g/gsubr.c http://golang.org/src/cmd/6g/gsubr.c Some are for single error return, but others are just to make a nice structure for the code.
  - raverbashing 13y ago
    Ah the old C style of function types in a different line. This smells of very old code (as the header testifies)
    - awda 13y ago
      ??? This is modern BSD (and probably plan9) style. Here's a line from the header: > Portions Copyright © 2009 The Go Authors. All rights reserved That's pretty recent. And here's BSD style(9): http://www.freebsd.org/cgi/man.cgi?query=style&sektion=9 http://www.freebsd.org/cgi/man.cgi?query=style&sektion=9 The function type should be on a line by itself preceding the function.
    - rafekett 13y ago
      which line?
    - foxhill 13y ago
AYBABTME 13y ago
I'm really thankful for the liveblogging, as I couldn't manage to get my body to the conference. I understand the desire to promote the Sourcegraph app by doing the blogging, and I think its effective. However, the blog is real annoying to browse, as every (prominent?) link points to Sourcegraph the app instead of the blog.
- AYBABTME 13y ago
  Just to clarify, because I think my original comment is disbalanced. I'm REALLY thankful for sourcegraph's liveblogging. The above comment was a suggestion as I thought they might want to know that (at least for me), the navigation of the blog was confusing.
  - sqs 13y ago
    Thanks! We are having a fun time liveblogging and are glad you're finding it useful. We got complaints when the blog image did NOT link to Sourcegraph, too. :) It's almost the end of this conference, but next time we liveblog, we'll have 2 separate header images, or try something else to make it less confusing.
hrjet 13y ago
This might become a nice benchmark for the Go language; the same code base implemented in C and Go! It may not be fine-tuned for optimisation, neither in C nor in Go, but may still give a good ball-park estimate.
- lazyjones 13y ago
  Indeed, but it would be much more useful and impressive if they rewrote the compiler in Go by hand. I'm disappointed that they are aiming for an automatic translation instead - some people are going to ask themselves whether Go actually isn't that much fun to program in. An independent reimplementation is better for correctness too, they can compare outputs and find bugs in both implementations instead of porting over old bugs and adding new bugs where the translation goes wrong.
TempleOSV2 13y ago
I wrote my compiler in assembly language. Then, as I built the compiler, I converted the compiler to HolyC. http://www.templeos.org/Wb/Accts/TS/Wb2/History.html http://www.templeos.org/Wb/Accts/TS/Wb2/History.html
micro_cam 13y ago
I'm reminded of the fortran to c compiler f2c that was used to produce pure c implementations of lots of libraries like LAPACK. I'm curious how they will handle things like pointer arithmetic and memory safety in C vs Go. If they mange to do so in a performant way I could see translating lots of numerical or computationally intensive code to Go so that it could be run in a shared cloud environment without worries about memory safety and without having to resort to vms for separation.
- mjcohen 13y ago
  If you took arbitrary fortran, especially with i/o, the results were an unreadable mess - but they compiled and ran. For a project I had (in the early 90s iirc), I had to extensively modify the fortran and make multiple versions before the generated C was readable. Still, much easier than a rewrite.
kristianp 13y ago
"They’re deciding to automatically convert the Go compiler written in C to Go, because writing from scratch would be too much hassle." When transcribing a talk, there isn't any need to write "They're". Just use the same pronoun the presenter used, otherwise it stands out like a sore thumb.
rdc12 13y ago
"Note: There’s a book written about converting goto code to code without goto in general, but this is a sledgehammer and not necessary here." Anyone have any idea what the title of that book is?
pohl 13y ago
3) Go has turned out to be a nice general purpose language and the compiler won’t be an outsize influence on the language design. In what sort of ways does self-hosting early influence a language design? Were they hoping to avoid something in particular by delaying self-hosting?
- gizmo686 13y ago
  The general way that self hosting influences language design is that the compiler is often one of the first major projects to be built using a language. This does not give it more influence then other major projects, but if your goal is to have a language designed for use case X, it is generally best to have your early projects with it be for X. Additionally, self hosting may encourage a language design that makes bootstrapping easier (such as a stricter divide between the state-1 language and the general language).
  - steveklabnik 13y ago
    To expand a bit on your last point, self-hosting helps answer the question "can I build Real Software (tm) with this language, or is it really difficult?" earlier in the lifetime of the language.
  - pohl 13y ago
    I sort of had the general sense of that already. I guess I was hoping for something more specific about what language features are so useful when writing a compiler but get in the way for general problems. The part I quoted above almost sounds like wiping sweat from one's brow after having dodged a bullet: "phew, the language is safe from influence by those gull-durn compiler-writers..."

212 points by sqs 13y ago

beliu 13y ago

Author of the post here. Happy to answer any questions I can, and FYI, we (Sourcegraph) are liveblogging all of GopherCon at http://gophercon.sourcegraph.com http://gophercon.sourcegraph.com. Let us know if you have any questions or find it useful!

nfoz 13y ago
Are you at all concerned about the possibility of the Trusting Trust[1] problem manifesting in any of your Go compilers? [1] http://cm.bell-labs.com/who/ken/trust.html http://cm.bell-labs.com/who/ken/trust.html
- 0xdeadbeefbabe 13y ago
  Since this is an exercise divorced from reality, the usual vehicle was FORTRAN. Dang it I cheated and looked up Rus's quine. For anyone not wanting to cheat a good hint would be "tail recursion"
- zellyn 13y ago
  It was suggested long ago that having multiple compilers would make it incredibly difficult to play Trusting Trust games.
codezero 13y ago
You might want to clarify that you aren't Russ Cox and that this post is a summary of a presentation he gave.
- sqs 13y ago
  Thanks for mentioning this. I initially included "(unofficial liveblog post)" in the HN title, but it apparently was edited out by a moderator. So, I added a clarification to the top of the post itself.

stcredzero 13y ago

Automated rewrite FTW! This can help you avoid freezing a project while it is being ported. Also, if you have a code base with its own idioms, then those idioms can be matched and translated, which can produce cleaner target code.

adient 13y ago
Go 1.0 spec is already frozen and will not change any time soon.
- szabba 13y ago
  I believe he referred mostyl to freezing the efforts on impoving the implementation.
  - adient 13y ago
    Implementing the compiler in Go becomes the effort to improve the implementation. Obvious, no?
  - stcredzero 13y ago
    I was also talking about projects and porting in general. Incidentally, I've been paid to do exactly what we're discussing. (Porting a production program using automated translation.) It works.

ANTSANTS 13y ago

>A Union is like a struct, but you’re only supposed to use one value >(they all occupy the same space in memory). It’s up to the programmer to know which variable to use. > There’s a joke in some of the original C code: > #define struct union /* Great space saver */ > This inspired a solution: > #define union struct /* keeps code correct, just wastes some space */ Somewhere in Scotland, a sum type sheds a single tear.

piokuc 13y ago
> #define union struct /* keeps code correct, just wastes some space */ Not always, though...
- rsc 13y ago
  Yes, always. And if you don't believe me, it's not my trick. I learned it from Dennis Ritchie (he was thinking about a C to Limbo converter).
  - andybalholm 13y ago
    It works when unions are used correctly. If they are used to deliberately subvert the type system (e.g., put in an int, get an array of char out) it doesn't work. But C has so many other ways to subvert the type system that there's no need to do that. Probably about the only place you'll see them used like that is in the code produced by web2c in compiling TeX. Knuth used variant records a lot to get around Pascal's type safety, and they get translated to unions.
    - piokuc 13y ago
      That's why I said "not always". Here is an example which will print different result if you define #define union struct Here you go: #include <stdio.h> union U { char a; char b; }; int main() { U u; u.a = 13; u.b = 10; printf("%d\n", int(u.a)); return 0; }

Shamanmuni 13y ago

It's great that they are aiming for an automated conversion from C to Go. It's clear they aspire to convert their code which was written in a certain way. But I think it would be a huge boost in Go usage if they could eventually aim to transpile any C code into Go code. A little dream of mine would be if in the future when Rust is stable Mozilla developed a transpiler from C++ to Rust. That would be brilliant. By the way, all the other talks at GopherCon seem pretty interesting, I hope someone uploads videos of them soon.

NateDad 13y ago
It has been stated that there will be videos available, but I'd not expect them until next week sometime (when the organizers get home and have recuperated).
jerf 13y ago
It is unclear how one would compile arbitrary C code into useful Go. The stereotyped conventions of well-written compiler code allows for a more idiomatic translation than a general translator could ever aspire to. C++ to Rust would be even crazier. (Transpile is a silly word. It's "compile". Compiling already trans-es.)
- stcredzero 13y ago
  Good automated translations involve matching and translating the project idioms. This results in code that fits the target language's language-idioms. Automated translation only at the level of language constructs results in odd looking generated code.
- Shamanmuni 13y ago
  I know it's crazy difficult to do it, that's why I was talking about dreams. But maybe a set of guidelines about translatable C code plus a translator which is good enough for a variety of cases would make refactoring the resulting code such a manageable task that many projects could consider switching to Go directly. You are technically correct in that compile already implicates a translation, but we usually use that term refering to a translation into a lower level language and not another language at the same level. You can say transpiler or source-to-source compiler for those cases and I think it's clearer and more accurate for the reader.

SixSigma 13y ago

Russ' paper on the subject from Dec 2013 https://docs.google.com/document/d/1P3BLR31VA8cvLJLfMibSuTdwTuF7WWLux71CYD0eeD8/preview?sle=true&pli=1# https://docs.google.com/document/d/1P3BLR31VA8cvLJLfMibSuTdw...

thinkpad20 13y ago

> There are currently 1032 goto statements in the Go compiler jumping to 241 labels. Wow, that's really striking. I know that goto statements have their uses but for something written in the last couple of years to have over a thousand of them is very surprising (it might not be surprising at all for those who write C code all the time). I guess they're mostly just for error handlers?

azernik 13y ago
Yeah, there are a lot of them in e.g. Linux kernel code for jumping to end-of-function cleanup. I suspect that in the process of translation to Go most will be replaced by the defer keyword.
jgrahamc 13y ago
For example: http://golang.org/src/cmd/6g/gsubr.c http://golang.org/src/cmd/6g/gsubr.c Some are for single error return, but others are just to make a nice structure for the code.
- raverbashing 13y ago
  Ah the old C style of function types in a different line. This smells of very old code (as the header testifies)
  - awda 13y ago
    ??? This is modern BSD (and probably plan9) style. Here's a line from the header: > Portions Copyright © 2009 The Go Authors. All rights reserved That's pretty recent. And here's BSD style(9): http://www.freebsd.org/cgi/man.cgi?query=style&sektion=9 http://www.freebsd.org/cgi/man.cgi?query=style&sektion=9 The function type should be on a line by itself preceding the function.
  - rafekett 13y ago
    which line?
  - foxhill 13y ago

AYBABTME 13y ago

I'm really thankful for the liveblogging, as I couldn't manage to get my body to the conference. I understand the desire to promote the Sourcegraph app by doing the blogging, and I think its effective. However, the blog is real annoying to browse, as every (prominent?) link points to Sourcegraph the app instead of the blog.

AYBABTME 13y ago
Just to clarify, because I think my original comment is disbalanced. I'm REALLY thankful for sourcegraph's liveblogging. The above comment was a suggestion as I thought they might want to know that (at least for me), the navigation of the blog was confusing.
- sqs 13y ago
  Thanks! We are having a fun time liveblogging and are glad you're finding it useful. We got complaints when the blog image did NOT link to Sourcegraph, too. :) It's almost the end of this conference, but next time we liveblog, we'll have 2 separate header images, or try something else to make it less confusing.

hrjet 13y ago

This might become a nice benchmark for the Go language; the same code base implemented in C and Go! It may not be fine-tuned for optimisation, neither in C nor in Go, but may still give a good ball-park estimate.

lazyjones 13y ago
Indeed, but it would be much more useful and impressive if they rewrote the compiler in Go by hand. I'm disappointed that they are aiming for an automatic translation instead - some people are going to ask themselves whether Go actually isn't that much fun to program in. An independent reimplementation is better for correctness too, they can compare outputs and find bugs in both implementations instead of porting over old bugs and adding new bugs where the translation goes wrong.

TempleOSV2 13y ago

I wrote my compiler in assembly language. Then, as I built the compiler, I converted the compiler to HolyC. http://www.templeos.org/Wb/Accts/TS/Wb2/History.html http://www.templeos.org/Wb/Accts/TS/Wb2/History.html

micro_cam 13y ago

I'm reminded of the fortran to c compiler f2c that was used to produce pure c implementations of lots of libraries like LAPACK. I'm curious how they will handle things like pointer arithmetic and memory safety in C vs Go. If they mange to do so in a performant way I could see translating lots of numerical or computationally intensive code to Go so that it could be run in a shared cloud environment without worries about memory safety and without having to resort to vms for separation.

mjcohen 13y ago
If you took arbitrary fortran, especially with i/o, the results were an unreadable mess - but they compiled and ran. For a project I had (in the early 90s iirc), I had to extensively modify the fortran and make multiple versions before the generated C was readable. Still, much easier than a rewrite.

kristianp 13y ago

"They’re deciding to automatically convert the Go compiler written in C to Go, because writing from scratch would be too much hassle." When transcribing a talk, there isn't any need to write "They're". Just use the same pronoun the presenter used, otherwise it stands out like a sore thumb.

rdc12 13y ago

"Note: There’s a book written about converting goto code to code without goto in general, but this is a sledgehammer and not necessary here." Anyone have any idea what the title of that book is?

pohl 13y ago

3) Go has turned out to be a nice general purpose language and the compiler won’t be an outsize influence on the language design. In what sort of ways does self-hosting early influence a language design? Were they hoping to avoid something in particular by delaying self-hosting?

gizmo686 13y ago
The general way that self hosting influences language design is that the compiler is often one of the first major projects to be built using a language. This does not give it more influence then other major projects, but if your goal is to have a language designed for use case X, it is generally best to have your early projects with it be for X. Additionally, self hosting may encourage a language design that makes bootstrapping easier (such as a stricter divide between the state-1 language and the general language).
- steveklabnik 13y ago
  To expand a bit on your last point, self-hosting helps answer the question "can I build Real Software (tm) with this language, or is it really difficult?" earlier in the lifetime of the language.
- pohl 13y ago
  I sort of had the general sense of that already. I guess I was hoping for something more specific about what language features are so useful when writing a compiler but get in the way for general problems. The part I quoted above almost sounds like wiping sweat from one's brow after having dodged a bullet: "phew, the language is safe from influence by those gull-durn compiler-writers..."