4 ms·
I spent a lot of time in 2011 or so struggling with GC on the JVM: http://blog.cloudera.com/blog/2011/02/avoiding-full-gcs-in-hbase-with-memstore-local-allocati
by tlipcon 11y ago
I spent a lot of time in 2011 or so struggling with GC on the JVM: http://blog.cloudera.com/blog/2011/02/avoiding-full-gcs-in-hbase-with-memstore-local-allocation-buffers-part-1/ http://blog.cloudera.com/blog/2011/02/avoiding-full-gcs-in-h... has some of the gory details. Even hacked a bit on G1: http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2011-April/002452.html http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2011-A...
With a lot of effort by many folks in the community, HBase has mostly tackled the full-GC problem, but still has occasional issues with some workloads.
So, GC was definitely one factor - not having GC means we can give 99th percentile numbers in the single-digit milliseconds, which is pretty nice. Our master process actually has shown <1ms 99.99th percentile for tablet location requests on an 80 node cluster. So again, numbers that are super difficult to get on the JVM unless you take an allocation-free approach like the HFT guys do.
Another factor was ease of integration of platform-specific code for performance reasons. For example, we make use of SSE prefetch instructions to improve scan speed in our concurrent B-tree by 30% or so. The b-tree itself would be difficult to implement in Java due to lack of control over object layout, etc. While you can eventually get the same performance with enough off-heaping and sun.misc.Unsafe, my feeling is that, by the time you've gone down that road, you might as well be using C++.
I'll admit that, after many years of not writing native code, I was a bit nervous of diving back in. Segfaults are never fun. But, we soon realized that the native code tooling has improved a _ton_ in the last decade. We run all of our tests precommit using the excellent Sanitizer tools from Google (ThreadSanitizer, AddressSanitizer, LeakSanitizer) and those make it nearly trivial to diagnose a leak or crash. We also have pretty strict guidelines around use of pointers, based on the Google C++ guidelines. Many will complain that this is a neutered form of C++, and they're right. But it's also a relatively safe form of C++.
I could probably write a lengthy blog post on our experiences of C++ vs Java, but hopefully the above gives you a taste. Overall I've been happy with the decision. Slightly more time spent on crashes. Less time spent on chasing hard-to-reproduce performance or memory consumption issues. And the thread checking tools are actually far superior, so I'd say less time spent chasing races.
- tlipcon 11y agoQuick example of how TSAN makes it easy to understand and fix races -- here's a commit message which shows its output: https://github.com/cloudera/kudu/commit/e402d5ed79a9c98283b6612ac3d3eb152e20ca38 https://github.com/cloudera/kudu/commit/e402d5ed79a9c98283b6... Apparently Google has this same tool for Java internally, but hasn't open sourced it due to their litigation with Oracle over Java stuff.
- random3 11y agoI think writing a long post about your experiences of C++ vs Java would be great (I'd pay you in [choose your drinks and count] for it ;)).
- acconsta 11y ago>I could probably write a lengthy blog post on our experiences of C++ vs Java I'd be interested in reading that. There's still a lot of FUD around using C++ for new projects.
- bankim 11y agoThanks for the response, Todd! Since most projects in Hadoop ecosystem are written in Java/Scala on JVM I was pleasantly surprised about choice of C++ :)
- vitalyd 11y ago+1 on writing up a blog post on c++ vs java. I suspect given your background in java, people may heed your words a bit more than usual. There's definitely a lot of outdated thinking in java land with regards to (modern) c++ and accompanying toolchain. Many big data projects could benefit from being written in c++ rather than java (or another jvm language).