3 ms·
Don’t get me started on Java performance! Java made things fast at the macro level by making threading possible. Remember trying to do any kind of portable mu
by willvarfar 6y ago
Don’t get me started on Java performance!
Java made things fast at the macro level by making threading possible. Remember trying to do any kind of portable multi core in C/C++ in 2000? It was no surprise JVM languages took over and became what we’ve built Big Data on (Hadoop, Hive, spark, etc)
But they leave so much on the table for micro performance! Take a CSV or JSON parser for example - likely spends all its time extracting string objects!
Now the Big Data community mostly doesn’t notice; for example, the Parquet code is about 5x slower than the Orc code, apparently, but we all adopt Parquet anyhow.
I’ve been going through my java codebases doing the basic stuff like “let’s write a simple fast json parser that doesn’t make a lot of garbage” and you can easily half the number of boxes you need etc. sometimes order of magnitude improvements by writing java code that is more like C.
But there’s this level you cannot reach. You look longingly at the raw mechanical sympathy C enabled.
Perhaps Rust is the future? I expect we’ll just keep using more cores than necessary though.
- karbarcca 6y agoTake a deep look at Julia! Adding multithreaded parsing capabilities to CSV.jl was really a joy; basically just chunking up the file and spawning threaded tasks to process each chunk with the existing parsing code. My favorite favorite thing about developing in Julia is the ability to write "high-level" type code and usually get decent performance, BUT THEN have the ability to fine tune for that extra boost of performance. The code inspection tools (see lowered, typed, LLVM, or native IR code levels) and super nice Task-based multithreading really make performance tuning a delight.
- willvarfar 6y ago(Some of the comments in https://news.ycombinator.com/item?id=24736559 https://news.ycombinator.com/item?id=24736559 talk about new lines inside fields, making multi-threading difficult) (My quick search to find that recent story on HN made me find this too https://news.ycombinator.com/item?id=24740527 https://news.ycombinator.com/item?id=24740527)