5 ms·

So what I take from this is that the previous implementation sustained a huge amount of code rot and new code got layered over it with a staple instead of prope

So what I take from this is that the previous implementation sustained a huge amount of code rot and new code got layered over it with a staple instead of proper re factoring. So he put the whole mess in a bin and re-done it cleanly with Go. Now it's much nicer. Some of Go's attributes helped along the way. Did I miss something?

rkrzr 13y ago
Yes. In my opinion the biggest win comes from goroutines and Go's runtime, which essentially allow you to write asynchronous non-blocking multi-threaded(!)code that looks like synchronous code and is as easy to read and reason about. This leads to a huge reduction in complexity compared to the callback spaghetti you often get in other async frameworks. Basically putting concurrency first in the language and not relying on libraries seems to really pay off.
- irtefa 13y ago
  Thanks for the thorough yet concise explanation.
- pcwalton 13y ago
  You don't have to build goroutines into the language to have lightweight multithreaded tasks. TBB does this for C++, for example. (Edit: I was mistaken about Java here -- apparently it does use OS threads.)
  - skybrian 13y ago
    goroutines are lighter than Java threads, though, so it's still an improvement.
    - hugoc 13y ago
      Lightweight Threads, Channels and Actors for the JVM: http://blog.paralleluniverse.co/post/49445260575/quasar-pulsar http://blog.paralleluniverse.co/post/49445260575/quasar-puls...
      - jerf 13y ago
        If you want to show that the JVM can do that, a better link is probably Akka: http://akka.io/ http://akka.io/ Your link appears to be a guy's personal project, which is great, but akka is from what I gather an industrial-strength implementation. Though as the Node people occasionally point out, it is advantageous to have this sort of thing baked into the language, so that everything done in the language supports the concepts, rather than having a relatively small corner support it. Plus you get Go, instead of Java, which I for one would find an improvement.
        karterk 13y ago
        Actually if one wants to use Akka, one should be using Scala. Akka using Java is a total pain in comparison. Atleast with Scala, one seldom gets the feeling that the library is far from the language semantics.
  - pjmlp 13y ago
    This is not true, regarding Java. The Java language specification does not require the use of OS threads for Java threads. It all depends which JVM you refer to. The JVMs that make use of green threads follow a similar threading model to Go. Don't mix languages with implementations.
- cmircea 13y ago
  C# 5 also has this, with the async/await keywords. The nice thing is that it works with older plain Task-based code (which was callback spaghetti).
  - bigdubs 13y ago
    Tasks are still OS threads, though they get lumped into a threadpool so a lot of the overhead is mitigated (in most cases).
    - balakk 13y ago
      The async/await model does not create new threads. http://msdn.microsoft.com/en-us/library/vstudio/hh191443.aspx http://msdn.microsoft.com/en-us/library/vstudio/hh191443.asp... TPL(Task parallel library) does use additional threads, and that's a different programming model. The async/await model in C# is essentially from F#'s async workflows: http://msdn.microsoft.com/en-us/library/dd233250.aspx http://msdn.microsoft.com/en-us/library/dd233250.aspx
      - bigdubs 13y ago
        I realize that, what I was referring to was the old model (TPL), which wasn't terrible to begin with.
        test-it 13y ago
        The "new" model is just syntactic sugar over the "old" TPL, so you're wrong anyway ;-)
acqq 13y ago
I guess, what you miss is that the author was the author of LiveJournal, memcached, now also of groupcache in Go. So the guy who rewrote the program was much, much more capable that any "typical" programmer.
- Cthulhu_ 13y ago
  Your comment implies that even Google, which has a reputation of hiring only the best of the best software developers, has "typical" programmers. My faith in the world has just been restored, <3.
  - ryanmolden 13y ago
    Best of the best that apply to Google. Or in other words, their hiring process is "Hire the best people that apply here", or, you know, the hiring policy of every company in the world. You could argue they uniquely attract smart people with some kind of high IQ gravity centered at the Googleplex, but that seems unlikely. Makes for good marketing copy though.
    - hung_yao 13y ago
      The degree to which it is enforced today - I'm not sure, but: http://googleresearch.blogspot.com/2006/03/hiring-lake-wobegon-strategy.html http://googleresearch.blogspot.com/2006/03/hiring-lake-wobeg...
  - asveikau 13y ago
    Right, because there's never been a Googler who has ever made a mistake. All of them are perfect and none of them are stupid, never a bad hire made or even a momentary lapse of judgment.
- fbuilesv 13y ago
  What relevance does that add to this presentation? It's interesting by itself. What do the presentation (or our discussion) gain from the fact that Brad is a super smart guy "much more capable than the 'typical' programmer"? Why are we even bringing up that point? :)
lttlrck 13y ago
The key point for me is C++ stdlib doesn't include an HTTP stack, while Go's does. Fair enough, but it was a bit of a foregone conclusion that he wouldn't want to develop/reuse a C++ HTTP stack as he went into the project with the intention of using Go...
- shin_lao 13y ago
  Here is a work in progress by another googler: http://cpp-netlib.org/ http://cpp-netlib.org/ (Dean Michael Berris)
damian2000 13y ago
Its also probably a lot easier to maintain now due to ... - Gofmt; code formatting - Smaller number of language keywords compared to C++ - Garbage collection - Built in concurrency - Simpler error handling (no C++ style exceptions)
mambodog 13y ago
The original codebase suffered from reinventing the wheel - badly. By composing together well tested high level abstractions, most of the tricky, broken parts could be removed from the responsibility of the application codebase, leaving only appication specific concerns. It could have been done without Go, but Go provides a good set of tools for the task. In C++ land, on the other hand, there isn't such a clear Right Way to do such things (though there are plenty of options).
deleted 13y ago
[deleted]
stephen_g 13y ago
Yep, I feel that the main point of the story is "we had crappy code, and rewrote it and now it's great". Could have been very similar if they'd rewritten it using C++11 features (the main difference being that you'd have to use a HTTP library instead of HTTP being in the standard library, but that doesn't make much difference)...
- deleted 13y ago
  [deleted]
- mtdewcmu 13y ago
  Rewriting is probably more important than the language you rewrite in.
moomin 13y ago
Actually, an awful lot of the "we rewrote X in language Y" stories on the internet have this as the true story, even if they don't admit it. (The LinkedIn mobile rewrite from Ruby to Node was like this. The official line was all "here's how to kick it in Node", the truth was the original implementation was single threaded across all users.) That said, I'll happily read this story over and over again, because it tends to provide insight in what works and what doesn't.

rkrzr 13y ago

Yes. In my opinion the biggest win comes from goroutines and Go's runtime, which essentially allow you to write asynchronous non-blocking multi-threaded(!)code that looks like synchronous code and is as easy to read and reason about. This leads to a huge reduction in complexity compared to the callback spaghetti you often get in other async frameworks. Basically putting concurrency first in the language and not relying on libraries seems to really pay off.

irtefa 13y ago
Thanks for the thorough yet concise explanation.
pcwalton 13y ago
You don't have to build goroutines into the language to have lightweight multithreaded tasks. TBB does this for C++, for example. (Edit: I was mistaken about Java here -- apparently it does use OS threads.)
- skybrian 13y ago
  goroutines are lighter than Java threads, though, so it's still an improvement.
  - hugoc 13y ago
    Lightweight Threads, Channels and Actors for the JVM: http://blog.paralleluniverse.co/post/49445260575/quasar-pulsar http://blog.paralleluniverse.co/post/49445260575/quasar-puls...
    - jerf 13y ago
      If you want to show that the JVM can do that, a better link is probably Akka: http://akka.io/ http://akka.io/ Your link appears to be a guy's personal project, which is great, but akka is from what I gather an industrial-strength implementation. Though as the Node people occasionally point out, it is advantageous to have this sort of thing baked into the language, so that everything done in the language supports the concepts, rather than having a relatively small corner support it. Plus you get Go, instead of Java, which I for one would find an improvement.
      - karterk 13y ago
        Actually if one wants to use Akka, one should be using Scala. Akka using Java is a total pain in comparison. Atleast with Scala, one seldom gets the feeling that the library is far from the language semantics.
- pjmlp 13y ago
  This is not true, regarding Java. The Java language specification does not require the use of OS threads for Java threads. It all depends which JVM you refer to. The JVMs that make use of green threads follow a similar threading model to Go. Don't mix languages with implementations.
cmircea 13y ago
C# 5 also has this, with the async/await keywords. The nice thing is that it works with older plain Task-based code (which was callback spaghetti).
- bigdubs 13y ago
  Tasks are still OS threads, though they get lumped into a threadpool so a lot of the overhead is mitigated (in most cases).
  - balakk 13y ago
    The async/await model does not create new threads. http://msdn.microsoft.com/en-us/library/vstudio/hh191443.aspx http://msdn.microsoft.com/en-us/library/vstudio/hh191443.asp... TPL(Task parallel library) does use additional threads, and that's a different programming model. The async/await model in C# is essentially from F#'s async workflows: http://msdn.microsoft.com/en-us/library/dd233250.aspx http://msdn.microsoft.com/en-us/library/dd233250.aspx
    - bigdubs 13y ago
      I realize that, what I was referring to was the old model (TPL), which wasn't terrible to begin with.
      - test-it 13y ago
        The "new" model is just syntactic sugar over the "old" TPL, so you're wrong anyway ;-)

acqq 13y ago

I guess, what you miss is that the author was the author of LiveJournal, memcached, now also of groupcache in Go. So the guy who rewrote the program was much, much more capable that any "typical" programmer.

Cthulhu_ 13y ago
Your comment implies that even Google, which has a reputation of hiring only the best of the best software developers, has "typical" programmers. My faith in the world has just been restored, <3.
- ryanmolden 13y ago
  Best of the best that apply to Google. Or in other words, their hiring process is "Hire the best people that apply here", or, you know, the hiring policy of every company in the world. You could argue they uniquely attract smart people with some kind of high IQ gravity centered at the Googleplex, but that seems unlikely. Makes for good marketing copy though.
  - hung_yao 13y ago
    The degree to which it is enforced today - I'm not sure, but: http://googleresearch.blogspot.com/2006/03/hiring-lake-wobegon-strategy.html http://googleresearch.blogspot.com/2006/03/hiring-lake-wobeg...
- asveikau 13y ago
  Right, because there's never been a Googler who has ever made a mistake. All of them are perfect and none of them are stupid, never a bad hire made or even a momentary lapse of judgment.
fbuilesv 13y ago
What relevance does that add to this presentation? It's interesting by itself. What do the presentation (or our discussion) gain from the fact that Brad is a super smart guy "much more capable than the 'typical' programmer"? Why are we even bringing up that point? :)

lttlrck 13y ago

The key point for me is C++ stdlib doesn't include an HTTP stack, while Go's does. Fair enough, but it was a bit of a foregone conclusion that he wouldn't want to develop/reuse a C++ HTTP stack as he went into the project with the intention of using Go...

shin_lao 13y ago
Here is a work in progress by another googler: http://cpp-netlib.org/ http://cpp-netlib.org/ (Dean Michael Berris)

damian2000 13y ago

Its also probably a lot easier to maintain now due to ... - Gofmt; code formatting - Smaller number of language keywords compared to C++ - Garbage collection - Built in concurrency - Simpler error handling (no C++ style exceptions)

mambodog 13y ago

The original codebase suffered from reinventing the wheel - badly. By composing together well tested high level abstractions, most of the tricky, broken parts could be removed from the responsibility of the application codebase, leaving only appication specific concerns. It could have been done without Go, but Go provides a good set of tools for the task. In C++ land, on the other hand, there isn't such a clear Right Way to do such things (though there are plenty of options).

deleted 13y ago

[deleted]

stephen_g 13y ago

Yep, I feel that the main point of the story is "we had crappy code, and rewrote it and now it's great". Could have been very similar if they'd rewritten it using C++11 features (the main difference being that you'd have to use a HTTP library instead of HTTP being in the standard library, but that doesn't make much difference)...

deleted 13y ago
[deleted]
mtdewcmu 13y ago
Rewriting is probably more important than the language you rewrite in.

moomin 13y ago

Actually, an awful lot of the "we rewrote X in language Y" stories on the internet have this as the true story, even if they don't admit it. (The LinkedIn mobile rewrite from Ruby to Node was like this. The official line was all "here's how to kick it in Node", the truth was the original implementation was single threaded across all users.) That said, I'll happily read this story over and over again, because it tends to provide insight in what works and what doesn't.