38 ms·
Comparison of C++17, Go, and Java for a next-generation sequencing tool
- techempower 7y agothe source code: https://github.com/ExaScience/elprep-bench https://github.com/ExaScience/elprep-bench
- jolmg 7y agoThat has the Java and C++ versions. Here's the Go version: https://github.com/exascience/elprep https://github.com/exascience/elprep
- qzw 7y agoTLDR: 1. Go, 2. Java, 3. C++17. Java was slightly faster than go but used significantly more memory. C++17 was slower than both and used more memory than go. I’m still reading for more details on the implementations, but the results are certainly not what I would’ve predicted.
- adrian_b 7y agoIt seems that the performance was dominated by memory management, so the comparison is not between the languages per se, but between their current garbage collectors, and respectively the reference counting implementation for C++.
- beering 7y agoYeah, further down in the article they track down the gap between cpp and Go/Java to the ref-counting deallocation work. I'm not a cpp expert, but it seems surprising to me that GC would beat ref-counting in any scenario.
- def-lkb 7y agoThat might seem counter-intuitive, but tracing GC is the fastest way to manage memory in general (arbitrary life time, graph shapes and allocation patterns). It permits really fast allocation, defragmentation and is sometime the only way to manage memory (cyclic datastructures for which reachability cannot be known directly from program text). Reference-counted GC on the other hand is notoriously slow, incomplete, and exhibit lots of bad performance patterns (that can be somehow managed by taking some elements of tracing GCs). The drawback is that tracing GCs are very hard to implement, to tune, and sometimes not all of the benefits are available at the same time. And because they are so counter intuitive, programmers in general have a hard time understanding their runtime behavior... Even GC experts struggle (the skills required range from low-level hardware to control theory, otherwise performance at scale is unpredictable).
- Rochus 7y agoIn C/C++ we have the possibility to avoid dynamic allocation altogether and to use system features like memory mapping. If we use C++ the same way as Java (everything dynamically) it's not too surprising the result is not (much) faster than Java.
- aidenn0 7y agoTheir original implementation was in common lisp. Common Lisp also lets you use mmap (in fact it's not that uncommon to do so if you have a large amount of mostly static data) to manage your memory manually. They clearly wanted automatic memory management, so the C++ implementation is reasonable. A fairer comparison might have used MPS or boehm instead of refcounting, but I suspect the results would have been similar.
- Rochus 7y ago> Common Lisp also lets you use mmap But not without allocating dynamic memory and copying data. > They clearly wanted automatic memory management Most likely because of some misconceptions. > so the C++ implementation is reasonable. How so? > but I suspect the results would have been similar Don't forget the data sets to be filtered, sorted an analyzed are up to 200 GB.
- Rochus 7y agoI had a quick look at the C++ source code provided at https://github.com/ExaScience/elprep-bench/tree/master/cpp https://github.com/ExaScience/elprep-bench/tree/master/cpp. As suspected, everything is dynamically allocated and no memory mapping (see e.g. http://man7.org/linux/man-pages/man2/mmap.2.html http://man7.org/linux/man-pages/man2/mmap.2.html) is used. No wonder this is slow and eats a lot of memory. At the moment I have no information about why this design was chosen, if there is a justification for it, or if the developers only knew this option. Maybe I can find some hints in the paper. From what I've seen up to now it can be savely assumed that with optimal use of data structures and system functions the C++ results are at least one order of magnitude better.
- JoeSmithson 7y agoThe problem with C++, for a lot of people, is not that it's impossible to do it right, but that it's easy to do it wrong.
- Rochus 7y agoI think this applies to any kind of advanced software development or programming languages, not only C++. There may be many reasons why scientists who are not computer scientists feel more comfortable with Go than with C++, but performance is certainly not one of them.
- thelazydogsback 7y agoIt depends on your definition of "wrong". You can mess up your domain logic with any language, but the chances of messing up perf (and even some ref vs. copy semantics that effect logic) in C++ are vastly greater than in GC'd langs. I've ported C++ code to C# with much perf gain, due to c'tors of every kind getting called all over the place. Doing it right in C++ is too much cognitive overload -- at least for me. If you're writing inner game loops or device drivers, I feel I need to pay for this complexity - for any LOB apps, I can't see choosing C++, or even Rust for that matter.
- 7y ago
- jolmg 7y ago> Based on our benchmark results, we selected Go as our new implementation language for elPrep, and recommend considering Go as a good candidate for developing other bioinformatics tools for processing SAM/BAM data as well. They don't seem to take into account that their results depend on their own proficiency in each programming language.
- vardump 7y ago> They don't seem to take into account that their results depend on their own proficiency in each programming language. If they're going to implement it as well, then that's perfect. They now know which language they should be using with their proficiency. In some other team, Java might have been #1. The overall best results with a great team would almost certainly be achievable by using C++. But this is what worked for them.
- jolmg 7y agoYes, for them that's great, but their statement sounds like they're suggesting their results would apply for everyone.
- vardump 7y agoFunny how people think what works for them works for everyone... so easy to fall into that trap even with the best intentions.
- Traster 7y agoBut....why? They could've just saved themselves the trouble and said "Anyone fancy doing this in C++? No. Great, that's that tricky question off the table!" Rather thna going through this ridiculous exercise of writing bad C++ in order to justify not using C++. If we really believe that this is just a way of idnetifying a revealed preference fine, but engineers should be smart enough to not need to trick thesmlves into coming to the obvious conclusion.
- aidenn0 7y agoI thought it was well known that reference counting is slower than all but the worst tracing GCs.
- DeathArrow 7y agoTrue. C# and Java constantly beat Swift on speed.
- etse 7y agoSorry about the naive question, but if the memory management overhead is worse in Swift, is the hardware it runs on typically better? I'm assuming some of this because I've noticed Android devices tend to require more CPU/memory compared to iOS devices in the same generation.
- ken 7y agohttps://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160208/009422.html https://lists.swift.org/pipermail/swift-evolution/Week-of-Mo... Lattner on Swift (2016): "...while it is true that modern GC's can provide high performance, they can only do that when they are granted much more memory than the process is actually using. Generally, unless you give the GC 3-4x more memory than is needed, you’ll get thrashing and incredibly poor performance..."
- aidenn0 7y agoAgreed. I would say 2x the heap size is table-stakes for high-performance tracing GC, and the more you give it the better performance you can get.
- flumpcakes 7y agoApparently the custom processors used in iOS devices are actually some of the fastest out there in that form factor. Designed by apple (I think using ARM instruction set) and made using some of the latest, most advanaced, process nodes by TMSC. If you look at raw benchmarks, they handidly beat the best Qualcom/"android SOC" chips out there. I think this changes very rapidly - given the market segment of high end phones has a yearly turn around. However, I did read an article today that claimed the "cheapest iPhone" (the new SE model) is faster than the most expensive android phone currently out there.
- nemetroid 7y agoI think the original title is better, although dropping "full-fledged next-generation" would be an even better choice. > A comparison of three programming languages for a full-fledged next-generation sequencing tool
- Magnap 7y ago"next-generation sequencing" is a term of art in this case
- nemetroid 7y agoI see, it sounded like a buzzword term (which I guess it still might be). The point is that the current title makes it sound like a general comparison, while the original title makes it clear that it's comparing three implementations of a single tool.
- zwieback 7y agoThe fact that gc is well suited to this application isn't suprising but what I thought was interesting was that subtracting the time to do the deallocation from the C++ benchmark brought it into line with Go. In other words, ignoring mem management, for this application, Go and C++ performed on par.
- gpderetta 7y agoThere is a huge amount of gratuitous reference count updates and double ptr indirection (see for example their string_slice). Those add up quickly. The rest is a lot of string manipulation. If you are not taking advantage of being able to layout your objects carefully and avoid memory avoiding allocations, I wouldn't expect C++ to have any particular advantage over Go or Java in in this particular scenario.
- pizlonator 7y agoVery surprising result. I wouldn’t have bet that this is what would have happened. But anyone working on language perf should take note even though it’s just one result from one team and one application. Of course they probably used C++ in a not great way and probably use Go in a better way. But maybe that is caused by something in Go that encourages good behavior or at least encourages the kind of behavior that Go optimizes for. So, even if this result doesn’t mean that C++ devs should switch to Go to get more speed, it’s a result that is worth pondering at least a bit, particularly if you like thinking about what it is that makes languages fast or slow.
- shigeo 7y ago> I wouldn’t have bet that this is what would have happened. Which part of the results are you referring to? It's well-known that reference counting has significantly lower throughput that tracing garbage collectors, so the fact that C++ is outperformed here isn't surprising at all.
- BubRoss 7y agoIf either of those affects performance there are WAY too many memory allocations happening.
- shigeo 7y agoSure, but I guess that's the point --- the GCs in Java and Go can handle any allocation pattern you throw at them reasonably well, but there's not, as far as I know, such a "one-size-fits-all" solution in C++ (not that it needs one.)
- BubRoss 7y agoI'm not sure I would call 'using your entire CPU to do trivially small heap allocations' a pattern unless you mean that it's a pattern you see when people wonder why their software is so slow.
- 7y ago
- DeathArrow 7y agoThat only demonstrated that poor usage of the language can make C++ slow.
- pulse7 7y agoThey should benchmark Java's implementation with different GCs...
- zmower 7y ago"We have not performed a detailed comparison against the original version of elPrep implemented in Common Lisp, but based on previous performance benchmarks, the Go implementation seems to perform close to the Common Lisp implementation." LMFAO.
- gumby 7y agoWhy would that be funny?
- tomp 7y agoIf I understand correctly, new Go code was as fast as old Lisp code, with much less effort and much clearer code. > Most existing Common Lisp implementations use stop-the-world, sequential garbage collectors. To achieve good performance, it was therefore necessary to explicitly control how often and when the garbage collector would run to avoid needless interruptions of the main program, especially during parallel phases. As a consequence, we also had to avoid unnecessary memory allocations, and reuse already allocated memory as far as possible, to reduce the number of garbage collector runs. However, our more recent attempts to add more functionality to elPrep (like optical duplicate marking, base quality score recalibration, and so on) required allocating additional memory for these new steps, and it became an even more complex task and a serious productivity bottleneck to keep memory allocation and garbage collection in check.
- shrimp_emoji 7y agoNo Rust smh
- lpghatguy 7y agoIt was mentioned in the article and was not considered a candidate due to a specific API requirement: > Other mature programming languages with support for reference counting include Objective-C, Swift, and Rust [50]. However, in its algorithm for duplicate marking, elPrep requires an atomic compare-and-swap operation on reference-counted pointers, which does not exist in those languages, but exists in C++17.
- kinghajj 7y agoSo they need https://crates.io/crates/arc-swap https://crates.io/crates/arc-swap ?
- zozbot234 7y agoExcept that this "atomic" operation over shared_ptr's in the C++ standard isn't actually required to be lock free, and is in fact not lock free in common stdlib implementations. So they're not actually gaining anything over, e.g. RwLock<Arc<RwLock<T>>> in Rust.
- gpderetta 7y agoThey are atomic in the sense that it is not possible to observe the intermediate states and they are not data races. It is extremely hard though to implement them in a lock-free way without a true DCAS (which pretty much no architecture implements). I think all implementations use spinlock pools. I guess transactional hardware could be used. This is one of the reasons why GC does make it easier to implement some lock free algorithms as you do not have to worry about races during memory deallocation. Without GC, I I guess the next best thing is RCU or harzard pointers, but they are significantly more complex. From a cursory read I do not see any fancy lock free algorithm in the code though, so I'm not sure why they need atomic operations on shared_ptrs.
- voldacar 7y agoThis is not surprising. Parallel GC will almost always be faster than refcounting. If you wrote a C or C++ program to accomplish this task after carefully planning out exactly when stuff needs to get manually malloced/freed, you could outperform any of their approaches And like another commenter mentioned, if you're writing a program which streams a lot of data sequentially from disk, and where throughput is important (such as in sequencing), you should always be using mmap
- ori_b 7y agoThat's a case where mmap isn't actually all that much faster than read, and due to the inter processor interrupts needed to synchronize the memory mappings across cores, it may end up much slower. You're grabbing large chunks and flushing the TLB a whole lot. If you are seeking randomly and doing small reads, then mmap will help quite a bit: the data will be faulted in, and accessing it a second, third, or hundredth time will not cost much.
- gpderetta 7y agoread mapping are actually not too bad. remote TLBs can be sychronised lazily on a page fault.
- gpderetta 7y agowell, there is careful planning and there is going out of the way to heap allocate everything.
- voldacar 7y agoYeah i hadn't really looked into their c++ code when I wrote my comment. it looks like code written by someone who doesn't even know the stack exists. insane
- yvdriess 7y agoThat makes the assumption that any refcounted object lifetime management can be replaced with manual management, which is not the case. As the paper states: it is not possible to manually determine the object lifetimes due to the nature of the problem, data and openness of the configurable analysis pipeline. Manual alloc/free would in their case mean holding on to much more memory than a GC/refcounting solution would.
- zerr 7y agoNo AWK and Ada this time? :)
- exabrial 7y agoThey're using "Java 10", (no mention which vendor or build) any idea if they had the compressed strings option enabled?
- jayd16 7y agoWho uses GBh to measure performance? Either you can afford the ram or not. If you can't, you should bench with swap performance.
- ezoe 7y agoAfter quickly glancing the code, I concluded that they wrote C++ like there is no static type. It seems they faithfully ported the very dynamic nature of their existing code to C++ without thinking. Like what is this? https://github.com/ExaScience/elprep-bench/blob/master/cpp/filter_pipeline.cpp#L20-L33 https://github.com/ExaScience/elprep-bench/blob/master/cpp/f... auto alns = any_cast<shared_ptr<deque<shared_ptr<sam_alignment>>>>(data); So the data is sam_alignment type inside shared_ptr inside deque inside another share_ptr inside god forbid any? Why did they do that? What kind of abomination is this? Also from the context, it is the only possible type. they use: try { any_cast<abomination_type>(data) ; } catch ( bad_any_cast ) { throw runtime_error(...) ; } If you're so sure an object of any only hold exactly one type and everything else is unexpected error, you shouldn't use any at all! They really like std::deque<T> and use it everywhere even though the sizeof(T) is like a few dozen bytes at best so they should rather use std::vector. The data structure of deque is a list of array. while it can amortize the continious adding of the elements to front or back, since the element size is very small, they should rather use vector. Speaking of data structure, they also use std::unordered_map<int, any>. the unordered_map is very slow(it's a node based hash map, not suitable for the modern hardware) and the sizeof(int) + sizeof(any) is like 20 bytes(sizeof(int) + two pointers) so they got no benefit of using node based data structure here. They should rather use sorted vector and binary search it. My conclusion, it's slow because they wrote C++ like a dynamic typed language and they choose the wrong data structures.
- WesternStar 7y ago<unpopular opinion>We should break ABI on unordered map just to stop embarrassing ourselves in public and in front of new users.</unpopular opinion>
- RantyDave 7y agoWhat's wrong with unordered map?
- ezoe 7y agoIt's implementation is buckets divided node based hash map. Theoretically, its order is good, but not that efficient in modern hardware where the memory access is heavily cached so the data locality is more important than saving the memory copy or memory size. Using the linked list to save some memory copy doesn't benefit at all and the overhead is far greater than simply copying the contagious large chunk of memory. Especially so if the data type is trivial so the simple byte by byte memcpy is suffice.
- fnord123 7y agoIt would be interesting to see an output of the compiler flags used in the C++ program that was benchmarked. The build script doesn't set an optimization level so it would default to -O0: https://github.com/ExaScience/elprep-bench/blob/master/cpp/make.sh https://github.com/ExaScience/elprep-bench/blob/master/cpp/m... I am sure they didn't benchmark it like this but it would be interesting to see the flags that /were/ used.
- gpderetta 7y agoThe jmalloc script does set the opt level and that's the one discussed in the paper.
- twic 7y ago> To achieve good performance, it was therefore necessary to explicitly control how often and when the garbage collector would run to avoid needless interruptions of the main program, especially during parallel phases. Why? This is a batch program, interruptions don't matter, only the end-to-end time does. > The goal of elPrep is to simultaneously keep both the runtime and the memory use low. Why? Keeping runtime low lets you get more work done. Keeping memory use low means what? They are using a machine with 384 GB RAM, make use of it. Worth noting also that they used GCC 7.2.1, Go 1.9.5, and Java 10. That's a pretty old GCC. They don't seem to explicitly select a GC with Java, so they'll be using G1. G1 is still not entirely mature. It got much faster between 9 and 10, and somewhat faster from 10 to 11. For a batch process like this, though, the parallel collector is still probably a better choice. Using a newer JDK and a different collector should give better performance - but admittedly, probably won't reduce heap usage.
- WatchDog 7y agoYeah there are lots more knobs they could have played with in regards to Java garbage collection. I wonder if they capped the java max heap size to what the go implementation used, how much it would have affected the runtime.
- avarsheny 7y agoGuys hire better C++ programmers rather making stupid claims.
- lostmsu 7y agoOr programmers in general. I bet they did not think, that Java consuming more memory can be inconsequential as in: it would free it if pressed to.
- zmmmmm 7y agoIt has to be noted: it's quite a strange approach all round - this framework reads all the data into memory. So if you have a 100GB genome it will read 100GB into memory. Presumably it stays in memory uncompressed so we are talking hundreds of GB to process even a single whole genome sample. This may indeed have some performance benefits, but it's a very impractical approach from a hardware point of view. Few places doing processing of genomic data will have many compute nodes with > 256GB memory, yet that would barely process 1 sample with this framework. God forbid you have a family of samples or tumor/normal comparison samples to analyse and need several genomes in memory together. Genomes are for the most part massively parallelisable and nearly every other toolkit I have seen has put that first and foremost in its design approach. Ensuring tools process data in a streaming manner and pipe between each other is a basic expectation of most genomic data tools. Which is all to say ... this is a very strange beast and I'm not sure a lot of conclusions can be drawn from it that generalise to other activities or approaches.
- yvdriess 7y agoUntrue. Large memory nodes are par the course for genomics workloads. Only a few stages of the analysis pipeline can stream the in/out effectively. Even then, going to disk for the result only to be read back is going to bottleneck your pipeline. The authors go in more detail around this topic in: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0209523 https://journals.plos.org/plosone/article?id=10.1371/journal...
- imtringued 7y agoWe once asked our cluster department to give us bare metal access to their smallest machine. They gave us a dedicated server with 256GB RAM. The real cluster nodes are bigger than that because they handle multiple jobs at the same time and they have hundreds of them. This isn't some electron framework where it is unexcusable to use that much RAM. The hardware that is available to scientists is more than capable of handling these workloads.
- greendave 7y agoProbably worth noting in the title that this is a June 2019 publication, so the versions used (gcc 7.2.1, go 1.9.5) etc. are not exceptionally ancient - basically they used what was readily available in CentOS 7 around that time. That said, some of the code used is quite... odd.
- google234123 7y agoIgnoring the terrible C++, why would they bother to write the same program in three languages. I feel like spending 3x time on just one of them would have produced the best outcome.
- teleforce 7y agoIn academic publication it is a convention to compare your proposed implementation with at least the other two comparable competitors (languages, framework, algorithm, scheme, etc). I think it is naturally intriguing and refreshing to see the performance comparison of programming languages by the the non-author of the languages themselves even though the implementations are probably not optimized to death.
- typon 7y agoOn one hand this comparison sucks because their C++ is quite non idiomatic, in the other hand it reflects something about C++. Why is it so easy to write non idiomatic C++? And so much so that it's much slower than a garbage collected language.
- madhadron 7y agoHere is the key point: > elPrep allows users to specify arbitrary combinations of SAM/BAM operations as a single pipeline in one command line. The assumption is that your native environment for data analysis is bash and you have to expose anything that you might want to do in bash. Further, > elPrep also accommodates functional steps provided by third-party tool writers That is, they are attempting to provide general semantics in a bash command line that they can handle arbitrary functions being dropped into their program. Remember, the average salary of a job requiring both programming and biology knowledge is much lower than the average salary of a job requiring programming knowledge alone, so as bioinformaticists build skill to the point where they can be employable as programmers, they mostly leave bioinformatics. Those who have escaped bash have usually done so into Python these days, which creates a class system of those gluing things together in Python versus those implementing algorithms in C or C++.
- alexeiz 7y agoThey use shared_ptr pervasively in their C++ implementation. It's not the best way to manage objects in C++ by far. The only thing that this performance comparison really tells me is that shared_ptr is worse than a modern GC. This is not really a surprising find.
- teleforce 7y agoIt seems that all the languages implementation in the benchmarking exercises are utilizing garbage collection (C++ using ref counting) it will be very interesting if someone extend this benchmark using D programming language, arguably the fastest language with GC. The fact that elPrep tool is using functional software architecture [1] and D natively supports functional programming makes me think this type of computing is very well suited for it. [1] https://github.com/exascience/elprep https://github.com/exascience/elprep
- cppbert3 6y agoreply of the author https://github.com/ExaScience/elprep-bench/issues/3 https://github.com/ExaScience/elprep-bench/issues/3