4 ms·
Data Processing benchmark featuring Rust, Go, Swift, Zig, Julia etc.
- UncleOxidant 3y agoC++ seems to be a glaring omission, though.
- 0cf8612b2e1e 3y agoIt’s a micro benchmark. Best to not take it too seriously as it is typically just some cheerleading for the author’s favorite language or framework.
- GuB-42 3y agoIt is in the repository but... > WIP - Excluded from chart until it's ready C and Fortran are also missing, but considering the task, that's understandable.
- fyzix 3y agoI ported the go version to c++ but it's very slow so I excluded it from the charts...as that'd be misleading. It's in the repo if you want to take a look. I have very little experience with c++.
- kelsey9876543 3y agoAlso leads in developer headache benchmark
- extasia 3y agoCare to elaborate? (I haven't use the language)
- rg111 3y agoSee this: https://archive.ph/YY3jN https://archive.ph/YY3jN
- abc_lisper 3y agoLOL. Funny as it is, it would help to know what the negatives are
- sdfghswe 3y agoYou know what they say, there's two types of languages. The ones that people complain about all the time, and the ones that nobody uses.
- IshKebab 3y agoThey say it because it sounds clever and it's sort of true on a very superficial level if you don't think about it too much. Not because popularity gives a license to be awful.
- anotherhue 3y agoFor a benchmark such as this, shouldn't there be some handcrafted ASM to compare to as the 'theoretical best'? Measuring in seconds (though easy) is not really as useful as measuring in opcodes (with retirement times). Wrong audience perhaps.
- pixelpoet 3y agoNot just that, systems programming languages like C++ can just mmap the file and then absolutely murder the simple parsing (e.g. no floating point), use a single mem alloc, tiny hashmap, radix sort for the final list. Simple stuff like this should be limited by memory bandwidth (gigabytes per second).
- Sukera 3y agoLots of languages can mmap, it's not exclusive to C++ or C. The JSON package the Julia code is using seems to mmap too, for example.
- cornstalks 3y agoHow much of this is benchmarking JSON parsing vs other processing? It'd be nice to see timing broken down based on each step in the requirements section. Not that JSON parsing isn't valid to measure, but it's not the most interesting thing to me given the large number of JSON parsers that exist in each language.
- PMunch 3y agoThe JSON parsing is actually done before the timer starts. If you look at the raw results Julia actually comes out very poorly when the JSON part is taken into account.
- cornstalks 3y agoThank you. That wasn't clear to me from the README but I can see that now looking through the code.
- markkitti 3y agoThat total timing also includes compilation, not just the JSON parsing.
- bamazizi 3y agoRust & Go are ver close to Julia. Maybe a bit of code optimization would speed them up ahead of Julia? But regardless, from LTS aspect, Rust and Go maybe the way to go
- fullstackchris 3y agoIs Julia bad for LTS? Is it something like no reverse compatibility? Genuine question; I've never used it before.
- borodi 3y agoJulia follows semver, so code written in 1.0 should work on the latest release (packages notwithstanding)
- IshKebab 3y agoNo he meant long term support as in, is Julia still going to be remotely popular in 10 years time? Go and Rust are both hugely popular and definitely will be around for decades. Julia? Debatable.
- wjholden 3y agoAnother post here indicated that these times don't include startup costs. From my experience with Julia (and I love Julia and have used it extensively at school!), the "time to first plot" thing is still a big problem. You would never want to write shell scripts in Julia, mostly because it takes several seconds to get the interpreter running.
- sundarurfriend 3y agoJust to add some nuance: > From my experience with Julia (and I love Julia and have used it extensively at school!), the "time to first plot" thing is still a big problem. This part is true - it has improved by an order of magnitude in recent versions and continues to, but time-to-first-X is still a tangible issue you have to deal with. > You would never want to write shell scripts in Julia, mostly because it takes several seconds to get the interpreter running. This part overstates the case - it takes less than half a second for the Julia runtime to start, even on my mediocre laptop. Which isn't nothing, to be sure, and maybe a non-starter for some use cases. But generally, for shell script like programs, the time taken by the runtime (the "interpreter", though it's not really one) itself isn't much of an issue. The delays come in when your code needs to load big packages to do its thing, their loading times and other time-to-first costs. And you can mitigate that too, by putting the main part of your code in a precompiled package, and just calling out to that from your script. All that is to say only that, in the past few years, Julia has gone from "you would never want to write shell scripts in Julia" to "it's mildly annoying that you have to consciously arrange your code in a certain way to avoid latencies, but it's doable and not too hard".
- PMunch 3y agoAs someone who contributed to this benchmark it really isn't a great one. First of the JSON parsing is done before the timer starts. If you look at the raw results you can see that the full runtime of the Julia solution for 60k posts is actually 9.6 seconds (compared to 3.7, 3.8, 4.4, and 7.9 seconds for Rust, Go, Zig, and Nim respectively). The only thing it really measures is hashing and hash table performance (with a sprinkling of GC and memory management). And of course with any benchmark like this how much dev time each respective language put into the benchmark.
- mcronce 3y agoThey're also benchmarking on Github Actions runners, with swings of up to ~50% from run to run, which is more than enough to shuffle the results to more or less random positions. I contributed to it last week but without any will to solve that kind of fundamental problem I don't see it as being particularly good. There's also no control on quality of contributions to the language-specific benchmarks.
- eigenspace 3y agoIt's been moved from Github actions to a Azure F4s v2 - 4vCPU-8GB-Ubuntu 22.04. Still not good for reliability in a benchmark though.
- bafe 3y agoIf I understood the code and the GitHub actions well, it also appears that they run each benchmark once. If, as you said GitHub actions runners show that much variability between runs, one should at least run the action multiple times and report the aggregate running time along with other statistics (standard deviation)...
- mcronce 3y agoAccording to another replier it's been moved to dedicated VMs in Azure, so it's not as bad, but still subject to noisy neighbors. I agree with your assessment - if I were fixing it I would do something similar.
- ddoolin 3y agoWhy is Bun so much slower than Node/Deno?
- vorticalbox 3y agoI would assume the file reading and writing but that is just a guess.
- fyzix 3y agoI was surprised by that too. They are running the same function and I/O is excluded so I believe this might just be a comparison with v8 and jscore.
- pyrolistical 3y agoOne of Bun’s key features is low startup time, which prob trades off execution time.
- mirekrusin 3y agolet top5 = Array(5).fill({ idx: 0, count: 0, }); ...this kind of code is nice source of bugs in js.
- iwatchuo 3y ago[dead]
- qweqwe14 3y agoWhen I see that Zig is slower than Go, I know for a fact that something's off. This benchmark really looks biased. Rant: As with many benchmarks, this suffers from the fact that there are multiple ways to do the same thing in multiple languages, and the most common one isn't necessarily the best for a particular use case. In an ideal world, you would have the benchmark in each language written by people that work with that language on daily basis and have all the necessary knowledge to produce a fair benchmark, which is something that a naive implementation often fails to do.