13 ms·

Go’s runtime C to Go rewrite, by the numbers

116 points by davecheney 12y ago

mholt 12y ago
Cool visualization. What causes the spikes? Code merges?
- vanderZwan 12y ago
  Wouldn't make sense for the few solitary spike though. Perhaps they are found regressions, causing them to go back to the original C code for a while? That would make sense in the context of merges too - there's always a few bugs that show up under closer scrutiny.
silversmith 12y ago
What's interesting is amount of C and Go code being so proportional - for every 100 lines of C code rewritten you seem to get 100 lines of Go. It surprises me, because from what I've seen Go seems to be much higher level language. Or are those just all the `if err != nil` lines?
- thomasahle 12y ago
  It may also be an artifact of the automated translation process. I reckon it probably produces pretty "C-like" Go code. Later, when they go over it manually, some things might get cleared up more idiomatically.
  - 4ad 12y ago
    There's no automatic process, only the compilers are automatically translated, the runtime code is manually rewritten.
- Lewisham 12y ago
  I would guess it is the `if err` lines and other such gofmt artifacts that expand the LOC. Go is higher-level, but it's not terse. It's deliberately pretty explicit. IMHO 100 lines of Go might well do the same as 100 lines of C, but the Go will be much more readable code.
  - seanmcdirmid 12y ago
    That is easy to check, could the author put 100 lines of C code next to 100 lines of equivalent Go code and let us judge for ourselves?
    - sambeau 12y ago
      If we judge for ourselves we will only be checking our opinion. Could you provide a readability algorithm in some form of executable that we can run against the 100 lines of c and Go code — then we'll be able to judge correctly.
      - seanmcdirmid 12y ago
        Honestly, I just want to see them side by side. We can then debate our subjective interpretations vs. debating pure biases.
        4ad 12y ago
        A random list of CLs that dealt with conversions. https://codereview.appspot.com/99380043 https://codereview.appspot.com/99380043 https://codereview.appspot.com/140050044 https://codereview.appspot.com/140050044 https://codereview.appspot.com/136980044 https://codereview.appspot.com/136980044 https://codereview.appspot.com/139930044 https://codereview.appspot.com/139930044 https://codereview.appspot.com/139930043 https://codereview.appspot.com/139930043 https://codereview.appspot.com/135070043 https://codereview.appspot.com/135070043 https://codereview.appspot.com/123700044 https://codereview.appspot.com/123700044 https://codereview.appspot.com/126210046 https://codereview.appspot.com/126210046 https://codereview.appspot.com/130340043 https://codereview.appspot.com/130340043 https://codereview.appspot.com/125610043 https://codereview.appspot.com/125610043 https://codereview.appspot.com/98510044 https://codereview.appspot.com/98510044
      - pjc50 12y ago
        readability algorithm Sure, it's next to the objective poetry quality metric. "Shall I compare thee to a summer's day?" > TYPE ERROR
- dons 12y ago
  Well its not "much" higher level. It adds memory management and a simpler concurrency story, which helps. But e.g. no generics, no algebraic data types, no pattern matching. Plenty of things will still have to be done by hand. I'm not super surprised.
- andrewchambers 12y ago
  The runtime code probably deals with a few things that aren't as compact in Go either. For example using the unsafe package in Go is very verbose since you are trying to work around the type system. C just lets you dereference any pointer you want etc etc. There are many types of code that are significantly longer in C# or java than in C, just because C lets you do crazy things with memory mapping. To get more terse code, you need to switch from the C family of sytax imo.
  - zak_mc_kracken 12y ago
    There are plenty of C family syntax languages that are more compact. The main reason why Go is very verbose is because it doesn't support generics (so you have to duplicate a lot of code) and no exceptions (so your code is littered with ok, err := Foo() if err { ... } and then the caller has to do the same thing (basically reinventing manually what exceptions do for you).
    - _delirium 12y ago
      Absent exceptions, for quick-and-dirty stuff I wish Go had a mode like Bash's 'set -e', which just aborts on any error return from any call. When the only thing you want to do on errors is abort the program, it's really tedious to write that same "if err, exit" after every function call. But perhaps the golang people's answer would be that for those kinds of quick-and-dirty programs, I should be using a shell script anyway, not Go? Or maybe Perl, which does require an explicit check, but uses a compact idiom for it, f() or die;
      - Lewisham 12y ago
        Yeah, I'd quite like an `or die;` equivalent, as I've run into the same thing as you: scripts that can't do anything if the chain of commands fails anywhere. I have written functions that do the same thing: xOrDie(), but it's a bit sucky. As far as "Why Go for shell scripts?" I actually think Go is pretty well suited to shell scripts (aside from no or die): getting stdout/stderr pipes set up is pretty trivial with the exec package. There must be anecdotal law somewhere that states that all shell scripts tend to Turing completeness over time: what starts with "oh, a small shell script would do this" always ends up growing to a full blown 1000+ line program with multiple code paths. Writing the first script in Go is a piece of defensive programming so that when it inevitably grows, it's growing into a language that better supports its size (static typing, readability, libraries etc.) It's probably overkill for personal stuff, but it is probably the right hammer for the job over time when it comes to corporate work.
- dsymonds 12y ago
  The higher level primitives that Go provides aren't suitable to be used when implementing the runtime. The subset of Go suitable for runtime implementation is pretty close to C in most respects.
- 4ad 12y ago
  The runtime is not regular Go code, it's tricky Go code that doesn't have access to the standard library, full of pointer manipulations using unsafe, and with no heap-escaped variables. It also has to deal the fact that the C compiler didn't produce stack maps with type information (and even if it did, it would not have been that useful since it abused uintptr's), so Go code must be really careful about when the stacks are allowed to be moved and how pointers must get proper types. In short, the Go code in the runtime is about at the same level at C code. Don't forget that this is a straight 1-to-1 translation; this is not the time to introduce new bugs. After it's all done, the runtime code can be refactored and it might become cleaner and smaller. PS: there are no go errors, and hence no if err != nil statements in the runtime.
fulafel 12y ago
Spoiler: Flash plugin would let you see a timeline graph showing lines-of-C vs lines-of-Go in the go project. (Strange that a Google Docs widget requires Flash...)
- Cthulhu_ 12y ago
  Google Docs was built before HTML5 was a thing. Second, lots of companies switched to Docs without being able to switch to a modern browser, so they're stuck with Flash for proper visualisations like that - and Google can't just drop support without losing enterprise customers.
  - cokernel_hacker 12y ago
    The seem to have a workable solution for Youtube: they support both HTML5 and flash.
johan_larson 12y ago
Why Go? I had a chance to do a bit of Go a year or so ago, and found the language very (how to put it?) bland. There just didn't seem to be anything surprising or obviously better about it. If the alternative is C++, I can understand the appeal of garbage collection and getting away from the complexities of the STL. But Java provides both. Anyone want to make the case for Go over Java?
- pjc50 12y ago
  - freedom from JVM vulnerabilities and updates - C-style systems programming without C-style catastrophic security bugs - "real binaries"
  - sanderjd 12y ago
    To expand on "real binaries" a bit. It's somewhat ironic, but distributing statically compiled binaries for common systems is actually easier than distributing source (or bytecode) for VM-based languages like Python, Ruby, and Java.
    - pjc50 12y ago
      True statically compiled Linux binaries for complex applications are surprisingly hard; libnss insists on dynamically loading, and libstdc++ doesn't like it either. I think we ended up with unpleasant RPATH hacks to load the copy of the standard libraries that shipped alongside the app rather than the system ones.
      - sanderjd 12y ago
        Interesting! Thanks for the info. I've never run into trouble myself, but I've never tried to link a go application to libnss or libstdc++.
  - pjmlp 12y ago
    - there are lots of JVMs to choose from. - also possible in Java via Unsafe package, which is being promoted to official package in Java 9 - Excelsior JET, Atego JVM, RoboVM and many others AOT compilers do exist for Java.

116 points by davecheney 12y ago

mholt 12y ago

Cool visualization. What causes the spikes? Code merges?

vanderZwan 12y ago
Wouldn't make sense for the few solitary spike though. Perhaps they are found regressions, causing them to go back to the original C code for a while? That would make sense in the context of merges too - there's always a few bugs that show up under closer scrutiny.

silversmith 12y ago

What's interesting is amount of C and Go code being so proportional - for every 100 lines of C code rewritten you seem to get 100 lines of Go. It surprises me, because from what I've seen Go seems to be much higher level language. Or are those just all the `if err != nil` lines?

thomasahle 12y ago
It may also be an artifact of the automated translation process. I reckon it probably produces pretty "C-like" Go code. Later, when they go over it manually, some things might get cleared up more idiomatically.
- 4ad 12y ago
  There's no automatic process, only the compilers are automatically translated, the runtime code is manually rewritten.
Lewisham 12y ago
I would guess it is the `if err` lines and other such gofmt artifacts that expand the LOC. Go is higher-level, but it's not terse. It's deliberately pretty explicit. IMHO 100 lines of Go might well do the same as 100 lines of C, but the Go will be much more readable code.
- seanmcdirmid 12y ago
  That is easy to check, could the author put 100 lines of C code next to 100 lines of equivalent Go code and let us judge for ourselves?
  - sambeau 12y ago
    If we judge for ourselves we will only be checking our opinion. Could you provide a readability algorithm in some form of executable that we can run against the 100 lines of c and Go code — then we'll be able to judge correctly.
    - seanmcdirmid 12y ago
      Honestly, I just want to see them side by side. We can then debate our subjective interpretations vs. debating pure biases.
      - 4ad 12y ago
        A random list of CLs that dealt with conversions. https://codereview.appspot.com/99380043 https://codereview.appspot.com/99380043 https://codereview.appspot.com/140050044 https://codereview.appspot.com/140050044 https://codereview.appspot.com/136980044 https://codereview.appspot.com/136980044 https://codereview.appspot.com/139930044 https://codereview.appspot.com/139930044 https://codereview.appspot.com/139930043 https://codereview.appspot.com/139930043 https://codereview.appspot.com/135070043 https://codereview.appspot.com/135070043 https://codereview.appspot.com/123700044 https://codereview.appspot.com/123700044 https://codereview.appspot.com/126210046 https://codereview.appspot.com/126210046 https://codereview.appspot.com/130340043 https://codereview.appspot.com/130340043 https://codereview.appspot.com/125610043 https://codereview.appspot.com/125610043 https://codereview.appspot.com/98510044 https://codereview.appspot.com/98510044
    - pjc50 12y ago
      readability algorithm Sure, it's next to the objective poetry quality metric. "Shall I compare thee to a summer's day?" > TYPE ERROR
dons 12y ago
Well its not "much" higher level. It adds memory management and a simpler concurrency story, which helps. But e.g. no generics, no algebraic data types, no pattern matching. Plenty of things will still have to be done by hand. I'm not super surprised.
andrewchambers 12y ago
The runtime code probably deals with a few things that aren't as compact in Go either. For example using the unsafe package in Go is very verbose since you are trying to work around the type system. C just lets you dereference any pointer you want etc etc. There are many types of code that are significantly longer in C# or java than in C, just because C lets you do crazy things with memory mapping. To get more terse code, you need to switch from the C family of sytax imo.
- zak_mc_kracken 12y ago
  There are plenty of C family syntax languages that are more compact. The main reason why Go is very verbose is because it doesn't support generics (so you have to duplicate a lot of code) and no exceptions (so your code is littered with ok, err := Foo() if err { ... } and then the caller has to do the same thing (basically reinventing manually what exceptions do for you).
  - _delirium 12y ago
    Absent exceptions, for quick-and-dirty stuff I wish Go had a mode like Bash's 'set -e', which just aborts on any error return from any call. When the only thing you want to do on errors is abort the program, it's really tedious to write that same "if err, exit" after every function call. But perhaps the golang people's answer would be that for those kinds of quick-and-dirty programs, I should be using a shell script anyway, not Go? Or maybe Perl, which does require an explicit check, but uses a compact idiom for it, f() or die;
    - Lewisham 12y ago
      Yeah, I'd quite like an `or die;` equivalent, as I've run into the same thing as you: scripts that can't do anything if the chain of commands fails anywhere. I have written functions that do the same thing: xOrDie(), but it's a bit sucky. As far as "Why Go for shell scripts?" I actually think Go is pretty well suited to shell scripts (aside from no or die): getting stdout/stderr pipes set up is pretty trivial with the exec package. There must be anecdotal law somewhere that states that all shell scripts tend to Turing completeness over time: what starts with "oh, a small shell script would do this" always ends up growing to a full blown 1000+ line program with multiple code paths. Writing the first script in Go is a piece of defensive programming so that when it inevitably grows, it's growing into a language that better supports its size (static typing, readability, libraries etc.) It's probably overkill for personal stuff, but it is probably the right hammer for the job over time when it comes to corporate work.
dsymonds 12y ago
The higher level primitives that Go provides aren't suitable to be used when implementing the runtime. The subset of Go suitable for runtime implementation is pretty close to C in most respects.
4ad 12y ago
The runtime is not regular Go code, it's tricky Go code that doesn't have access to the standard library, full of pointer manipulations using unsafe, and with no heap-escaped variables. It also has to deal the fact that the C compiler didn't produce stack maps with type information (and even if it did, it would not have been that useful since it abused uintptr's), so Go code must be really careful about when the stacks are allowed to be moved and how pointers must get proper types. In short, the Go code in the runtime is about at the same level at C code. Don't forget that this is a straight 1-to-1 translation; this is not the time to introduce new bugs. After it's all done, the runtime code can be refactored and it might become cleaner and smaller. PS: there are no go errors, and hence no if err != nil statements in the runtime.

fulafel 12y ago

Spoiler: Flash plugin would let you see a timeline graph showing lines-of-C vs lines-of-Go in the go project. (Strange that a Google Docs widget requires Flash...)

Cthulhu_ 12y ago
Google Docs was built before HTML5 was a thing. Second, lots of companies switched to Docs without being able to switch to a modern browser, so they're stuck with Flash for proper visualisations like that - and Google can't just drop support without losing enterprise customers.
- cokernel_hacker 12y ago
  The seem to have a workable solution for Youtube: they support both HTML5 and flash.

johan_larson 12y ago

Why Go? I had a chance to do a bit of Go a year or so ago, and found the language very (how to put it?) bland. There just didn't seem to be anything surprising or obviously better about it. If the alternative is C++, I can understand the appeal of garbage collection and getting away from the complexities of the STL. But Java provides both. Anyone want to make the case for Go over Java?

pjc50 12y ago
- freedom from JVM vulnerabilities and updates - C-style systems programming without C-style catastrophic security bugs - "real binaries"
- sanderjd 12y ago
  To expand on "real binaries" a bit. It's somewhat ironic, but distributing statically compiled binaries for common systems is actually easier than distributing source (or bytecode) for VM-based languages like Python, Ruby, and Java.
  - pjc50 12y ago
    True statically compiled Linux binaries for complex applications are surprisingly hard; libnss insists on dynamically loading, and libstdc++ doesn't like it either. I think we ended up with unpleasant RPATH hacks to load the copy of the standard libraries that shipped alongside the app rather than the system ones.
    - sanderjd 12y ago
      Interesting! Thanks for the info. I've never run into trouble myself, but I've never tried to link a go application to libnss or libstdc++.
- pjmlp 12y ago
  - there are lots of JVMs to choose from. - also possible in Java via Unsafe package, which is being promoted to official package in Java 9 - Excelsior JET, Atego JVM, RoboVM and many others AOT compilers do exist for Java.