6 ms·

Why Go Was the Right Choice for CockroachDB

62 points by scapbi 11y ago

hectormalot 11y ago
Excuse me if this is a dumb question, but I've seen a lot on HN these days where people would suggest Erlang/Elixer for this kind of use case (high performance distributed). Any reason (beyond syntax preference) that this was not one of the languages considered?
- jerf 11y ago
  I wouldn't recommend the Erlang VM for anything that needs to itself be running at the highest performance levels. The VM is not very fast when it comes to computation. It's a great VM for what it does, and absolutely should be considered for many other such tasks, but probably not a great choice for a database. To some extent, but a lesser extent, the same could be said for Go: http://benchmarksgame.alioth.debian.org/u64q/which-programs-are-fastest.html http://benchmarksgame.alioth.debian.org/u64q/which-programs-... (It's a bit weird reading that page, because it's broken into two graphs, but they appear to be about the same thing.) If I were thinking "trendy language to write my next database in" personally I'd favor Rust by a country mile. This is exactly where you'd like all that stuff Rust provides just for correctness' sake, and this is the sort of task where you may even be able to outrun C++ if Rust helps you write correct code with fewer barriers slathered about for correctness. Edit: Ha, and just below this comment as I read it I see the team commenting about Rust.
- greenleafjacob 11y ago
  Riak is a database written in Erlang. Erlang is not good for "high performance distributed" systems. It's good for systems whose primary characteristic is uptime rather than latency or throughput.
akbar501 11y ago
In case anyone's interested in trying CockroachDB, I have been writing some docs on how to install it, use the SQL commands, etc. So far, I've found it's easy to get started with if you're familiar with another relational DB, although its definitely still (pre)alpha. CockroachDB docs: https://www.grockdoc.com/cockroachdb/alpha/articles https://www.grockdoc.com/cockroachdb/alpha/articles FYI: The installation docs are at the bottom of the list.
sportanova 11y ago
A little late, but why not Rust? Not the right libraries? I would think no gc would be one of the biggest factors
- arthursilva 11y ago
  They started the project long before Rust was stable and even today the Rust ecosystem is still lacking some libraries (understandably). The performance potential is real but not free as the productivity isn't as good.
- Zikes 11y ago
  I've often read stories about fast refactors from Python or PHP to Go. I wonder if that would translate to a reasonably fast refactor to Rust later on, if that should become necessary or desirable.
  - aidenn0 11y ago
    My experience is that refactoring from a garbage-collected language to a non-garbage-collected language are slow.
    - jerf 11y ago
      Rust is different enough as non-GC'ed languages go that I'd be interested in someone's experience, whenever they have it. I could easily see it being easier or harder than trying to convert to a "conventional" language. Heck, I could even see it sort of being both, where the conversion takes longer because the compiler keeps complaining but the final outcome is solid almost instantly.
      - aidenn0 11y ago
        I've done a port of a small amount of code (not a full project) from lisp to Rust; it really doesn't work, as Rust's ownership semantics and circular references are somewhat orthogonal. Also lisp is in an odd position where it has influence from functional style, but at the same time expects to have mutability always available, while Rust wants mutability to be constrained to known dynamic extents.
readams 11y ago
The article seems to make a lot of very dubious claims about Java. Especially when they started this project, Java performance was much better than go (Those double-digit second GC pauses weren't a concern with go?), and the Java garbage collection remains much more sophisticated than even current bleeding edge go. Also the claim that you have to add extra interfaces to Java is completely false. There's no reason at all you need to add pointless interfaces in Java. Java's type system is strictly more powerful than go's type system. This was actually a design choice in go to make the language easier. Java generics are also a huge boon when writing something like a database. Modern C++ also would have been an excellent choice for writing a database, and would have been much more productive than go.
- 59nadir 11y ago
  > The article seems to make a lot of very dubious claims about Java. I was also stumped by this confidence in that Go is supposed to be better than Java performance-wise and I do think that the claims warrant some proof. However: > Modern C++ ... would have been much more productive than go. I'm not up to date on current C++, but this seems like a claim much like the ones you objected to. Could you elaborate on how C++ is a more productive language (in this case?)?
- schmichael 11y ago
  "What remains to be seen, of course, is how all of this Go code will perform." They're pretty clear and honest that Go's performance characteristics are an unknown and not the reason it was chosen.
  - detrino 11y ago
    There is a clear bias. When talking about Java: > JVM garbage collection is notoriously expensive. > Java’s known performance issues made it unappealing. > However, in our past experience, we ported a large system from Java to Go, which greatly decreased its memory footprint and garbage collection overhead. When talking about Go: > What remains to be seen, of course, is how all of this Go code will perform. Yet: The JVM's GC is superior to Go's, Java tends to outperform Go, and a large memory footprint is a necessary evil for getting good performance out of any GC.
shockzzz 11y ago
Can anyone link a reference for known performance issues in Java? And is it issues in Java or the JVM?
- valarauca1 11y ago
  I can't provide a direct link. But generally Java itself is rarely the problem. The language is fine, the byte code is great. The JVM on the other hand with its stop the world garbage collector is a constant source of performance issues. Especially once you start getting over 2-4GB heaps. GC cycles can take 1s+
  - shockzzz 11y ago
    The Golang GC is "better" than the JVM GC, according to this blog post? Though I think it's suggesting that the Golang GC is easier to tune and configure
  - slantedview 11y ago
    There are plenty of options for storing select data off heap, which a database such as Cockroach could certainly have taken advantage of.
    - shockzzz 11y ago
      so this is an oversight on their part? it sounds like they had maybe a half-baked justification, but idk
schmichael 11y ago
"Perhaps most telling that Go is a good fit is that a lack of previous exposure to the language has not been a barrier for contributors" This. It's true at the 100% Go backend company I work at that's never hired anyone with Go experience, and I've heard many others repeat this mantra. While many people like to think of Go's lack of features in the realms of packaging, types, syntactic sugar, etc. as downsides, they're huge upsides when learning the language. There's just so much less to learn. Onboarding a new Go developer takes days, not weeks or months.
fasteo 11y ago
>>>>> Java’s known performance issues made it unappealing I have read this many times and every time I read it and it seems to me that this is just a rationalization, probably an unconscious one: "Offering false or inauthentic excuses for our claim because we know the real reasons are much less persuasive or more embarrassing to share, or more harsh than the manufactured ones given." [1] Maybe not the case in this particular instance, but every reason I have heard about choosing Go can be summarized as: "We chose Go because it is new new cool language. Ahh, and you can deploy by copying a single exe file" [1] http://www.logicallyfallacious.com/index.php/logical-fallacies/150-rationalization http://www.logicallyfallacious.com/index.php/logical-fallaci...
cs702 11y ago
The gist of it is, they believe they're more productive with Go than with C++ or Java. In particular they mention Go's small idiomatic footprint and its focus on simplicity and consistency, which they say make it easier+faster to pick up the language (for new developers) and write cleaner, understandable code (for all developers). As to performance, they expect it will be comparable to C++ and better than Java. I say "expect" because they don't provide any evidence to back that claim. PS. I often feel like I'm walking walking on eggshells when discussing computer languages on HN. For the record, I'm neither "pro-Go" nor "anti-Go." Languages are tools, not religions. Go is just a tool: https://news.ycombinator.com/item?id=10465697 https://news.ycombinator.com/item?id=10465697
vishalzone2002 11y ago
I agree to most points. One thing I struggled trying it is lack of information on interenet as compared to more mature languages like c++ or java. Since most of the cockroachdb team including founders come from google, I wonder if they have access to better advice too.

62 points by scapbi 11y ago

Excuse me if this is a dumb question, but I've seen a lot on HN these days where people would suggest Erlang/Elixer for this kind of use case (high performance distributed). Any reason (beyond syntax preference) that this was not one of the languages considered?

jerf 11y ago
I wouldn't recommend the Erlang VM for anything that needs to itself be running at the highest performance levels. The VM is not very fast when it comes to computation. It's a great VM for what it does, and absolutely should be considered for many other such tasks, but probably not a great choice for a database. To some extent, but a lesser extent, the same could be said for Go: http://benchmarksgame.alioth.debian.org/u64q/which-programs-are-fastest.html http://benchmarksgame.alioth.debian.org/u64q/which-programs-... (It's a bit weird reading that page, because it's broken into two graphs, but they appear to be about the same thing.) If I were thinking "trendy language to write my next database in" personally I'd favor Rust by a country mile. This is exactly where you'd like all that stuff Rust provides just for correctness' sake, and this is the sort of task where you may even be able to outrun C++ if Rust helps you write correct code with fewer barriers slathered about for correctness. Edit: Ha, and just below this comment as I read it I see the team commenting about Rust.
greenleafjacob 11y ago
Riak is a database written in Erlang. Erlang is not good for "high performance distributed" systems. It's good for systems whose primary characteristic is uptime rather than latency or throughput.

akbar501 11y ago

In case anyone's interested in trying CockroachDB, I have been writing some docs on how to install it, use the SQL commands, etc. So far, I've found it's easy to get started with if you're familiar with another relational DB, although its definitely still (pre)alpha. CockroachDB docs: https://www.grockdoc.com/cockroachdb/alpha/articles https://www.grockdoc.com/cockroachdb/alpha/articles FYI: The installation docs are at the bottom of the list.

sportanova 11y ago

A little late, but why not Rust? Not the right libraries? I would think no gc would be one of the biggest factors

arthursilva 11y ago
They started the project long before Rust was stable and even today the Rust ecosystem is still lacking some libraries (understandably). The performance potential is real but not free as the productivity isn't as good.
Zikes 11y ago
I've often read stories about fast refactors from Python or PHP to Go. I wonder if that would translate to a reasonably fast refactor to Rust later on, if that should become necessary or desirable.
- aidenn0 11y ago
  My experience is that refactoring from a garbage-collected language to a non-garbage-collected language are slow.
  - jerf 11y ago
    Rust is different enough as non-GC'ed languages go that I'd be interested in someone's experience, whenever they have it. I could easily see it being easier or harder than trying to convert to a "conventional" language. Heck, I could even see it sort of being both, where the conversion takes longer because the compiler keeps complaining but the final outcome is solid almost instantly.
    - aidenn0 11y ago
      I've done a port of a small amount of code (not a full project) from lisp to Rust; it really doesn't work, as Rust's ownership semantics and circular references are somewhat orthogonal. Also lisp is in an odd position where it has influence from functional style, but at the same time expects to have mutability always available, while Rust wants mutability to be constrained to known dynamic extents.

readams 11y ago

The article seems to make a lot of very dubious claims about Java. Especially when they started this project, Java performance was much better than go (Those double-digit second GC pauses weren't a concern with go?), and the Java garbage collection remains much more sophisticated than even current bleeding edge go. Also the claim that you have to add extra interfaces to Java is completely false. There's no reason at all you need to add pointless interfaces in Java. Java's type system is strictly more powerful than go's type system. This was actually a design choice in go to make the language easier. Java generics are also a huge boon when writing something like a database. Modern C++ also would have been an excellent choice for writing a database, and would have been much more productive than go.

59nadir 11y ago
> The article seems to make a lot of very dubious claims about Java. I was also stumped by this confidence in that Go is supposed to be better than Java performance-wise and I do think that the claims warrant some proof. However: > Modern C++ ... would have been much more productive than go. I'm not up to date on current C++, but this seems like a claim much like the ones you objected to. Could you elaborate on how C++ is a more productive language (in this case?)?
schmichael 11y ago
"What remains to be seen, of course, is how all of this Go code will perform." They're pretty clear and honest that Go's performance characteristics are an unknown and not the reason it was chosen.
- detrino 11y ago
  There is a clear bias. When talking about Java: > JVM garbage collection is notoriously expensive. > Java’s known performance issues made it unappealing. > However, in our past experience, we ported a large system from Java to Go, which greatly decreased its memory footprint and garbage collection overhead. When talking about Go: > What remains to be seen, of course, is how all of this Go code will perform. Yet: The JVM's GC is superior to Go's, Java tends to outperform Go, and a large memory footprint is a necessary evil for getting good performance out of any GC.

shockzzz 11y ago

Can anyone link a reference for known performance issues in Java? And is it issues in Java or the JVM?

valarauca1 11y ago
I can't provide a direct link. But generally Java itself is rarely the problem. The language is fine, the byte code is great. The JVM on the other hand with its stop the world garbage collector is a constant source of performance issues. Especially once you start getting over 2-4GB heaps. GC cycles can take 1s+
- shockzzz 11y ago
  The Golang GC is "better" than the JVM GC, according to this blog post? Though I think it's suggesting that the Golang GC is easier to tune and configure
- slantedview 11y ago
  There are plenty of options for storing select data off heap, which a database such as Cockroach could certainly have taken advantage of.
  - shockzzz 11y ago
    so this is an oversight on their part? it sounds like they had maybe a half-baked justification, but idk

schmichael 11y ago

"Perhaps most telling that Go is a good fit is that a lack of previous exposure to the language has not been a barrier for contributors" This. It's true at the 100% Go backend company I work at that's never hired anyone with Go experience, and I've heard many others repeat this mantra. While many people like to think of Go's lack of features in the realms of packaging, types, syntactic sugar, etc. as downsides, they're huge upsides when learning the language. There's just so much less to learn. Onboarding a new Go developer takes days, not weeks or months.

fasteo 11y ago

>>>>> Java’s known performance issues made it unappealing I have read this many times and every time I read it and it seems to me that this is just a rationalization, probably an unconscious one: "Offering false or inauthentic excuses for our claim because we know the real reasons are much less persuasive or more embarrassing to share, or more harsh than the manufactured ones given." [1] Maybe not the case in this particular instance, but every reason I have heard about choosing Go can be summarized as: "We chose Go because it is new new cool language. Ahh, and you can deploy by copying a single exe file" [1] http://www.logicallyfallacious.com/index.php/logical-fallacies/150-rationalization http://www.logicallyfallacious.com/index.php/logical-fallaci...

cs702 11y ago

The gist of it is, they believe they're more productive with Go than with C++ or Java. In particular they mention Go's small idiomatic footprint and its focus on simplicity and consistency, which they say make it easier+faster to pick up the language (for new developers) and write cleaner, understandable code (for all developers). As to performance, they expect it will be comparable to C++ and better than Java. I say "expect" because they don't provide any evidence to back that claim. PS. I often feel like I'm walking walking on eggshells when discussing computer languages on HN. For the record, I'm neither "pro-Go" nor "anti-Go." Languages are tools, not religions. Go is just a tool: https://news.ycombinator.com/item?id=10465697 https://news.ycombinator.com/item?id=10465697

vishalzone2002 11y ago

I agree to most points. One thing I struggled trying it is lack of information on interenet as compared to more mature languages like c++ or java. Since most of the cockroachdb team including founders come from google, I wonder if they have access to better advice too.