11 ms·
Someone on Lobsters suggested this too, and I don't think it would be particularly edifying. But maybe I'm wrong. Someone should go out and do that benchmark. M
by burntsushi 6y ago
Someone on Lobsters suggested this too, and I don't think it would be particularly edifying. But maybe I'm wrong. Someone should go out and do that benchmark. Maybe we'll learn something. Here is what I wrote though:
Coming up with a benchmark is hard. Benchmarks like these are models. All models are wrong. But. Some are useful.
Follow you thinking to its logical conclusion. If you open up the parallel programming flood gates, now you need to do that for all of the programs. Now your effort to contribute a sample for each language goes up. Maybe enough where actually doing the benchmark isn't practical.
So let's play the tape. If someone told me, "sure, have fun, throw threads at this!" I'd probably say, "can I use memory maps?" And let's say, "yes, sure, go for it!" I say okay, here's my first crack at it:
1. Hope stdin is a file and open it as a memory map.
2. Split the memory map slice into N chunks where N is the number of logical CPU cores on the system.
3. Run approximately the same algorithm devised in existing code submissions in N different threads, with one thread per chunk.
4. At the end, join the results and print.
If that ends up being the best strategy overall, then your model has become over-complicated because performance now likely depends on step (3). Which is exactly what the existing benchmark is. Now, maybe some languages have a harder time passing data over thread boundaries than others and maybe that impacts things. But that doesn't apply to, say, C, Rust, C++ or Go.
If that's not the best strategy, then maybe there is a cleverer approach that involves synchronizing on one shared data structure. I doubt it, but maybe. If so, your benchmark turns into one that is very different. It's no longer just about data transformation, but now you have to worry about synchronization trickiness. And now you're way outside of a standard goroutine pool that you might have used in Go.
Building a good benchmark is an art. Sometimes the constraints look dumb and arbitrary. But you have to sit down and really look at it from a bunch of different angles:
* Do the constraints approximate some real world conditions?
* Do the constraints inhibit writing code that looks like real world code?
* Do the constraints inhibit optimizing the crap out of the code?
* Do the constraints make the benchmark difficult to create in the first place?
* Do the constraints permit easy and clear measurement?
There are probably more things that I didn't think about.
- chaostheory 6y agoYes, benchmarks are hard. However, single thread benchmarks don't reflect the real world unless you're a student.
- burntsushi 6y agoTIL GNU grep is not in the real world. Consider re-reading my comment. It anticipated your criticism and responded directly to it.
- chaostheory 6y agoYes, you've basically gone over the details of why doing parallelized / concurrent performance comparisons between different programming languages is very difficult. I agree with most of what you have written. However, this doesn't change the fact that any comparison of performance between different languages, that ignores relevant concurrent and / or parallelization features, isn't very useful. This is especially true when the selling point for a few languages in this list is concurrency and / or parallization related features.
- burntsushi 6y agoNo, what I'm saying is that it is useful. Because in a concurrent program, you'll almost certainly being using the single threaded techniques developed here. And that the fastest program will be the one that uses the fastest single threaded techniques. So it makes sense to simplify the model and just look at the thing that is going to dominate the runtime of the program. If a "dumb" thread pool is indeed what ends up being the best concurrent solution, then Rust, Go, C and C++ (and probably most of the other languages) are going to have almost no problems implementing it. It's just Not That Interesting. Now, maybe I'm wrong. Maybe there is something clever here to exploit in a concurrent program, and that some languages let you do that easier than others. I doubt it. It would be interesting if it turned out to be the case, but it would be a different benchmark. It wouldn't just be about how well a language can deal with simple data transformation tasks, but also about how it deals with tricky concurrency problems. But this doesn't seem like the kind of problem that needs such things. It complicates the model. Like honestly, just saying that any benchmark that ignores parallelism "isn't very useful" is just totally ridiculous.
- injinj 6y agoAmusingly, I've done a multi-threaded version of the word counting program in order to test a shm kv store. I needed benchmark that created a lot of cross thread concurrent accesses to keys and I found a blog about this test. My version has serious constraints though, you have to create a shared memory map with enough space to hold all of the keys beforehand, as it doesn't resize the shm kv map as it runs. This is the source for it: https://github.com/raitechnology/raikv/blob/master/test/ctest.c https://github.com/raitechnology/raikv/blob/master/test/ctes... The speedup of the multi-threaded version vs the single-threaded version is about linear. The single threaded version uses 2 threads, one to read stdin and one to hash the keys, the 16 threaded version uses one thread to read, 16 to hash. $ time ctest -t 1 < ~/data/enwiki-p10p30303 18.88user 0.25system 0:09.58elapsed 199%CPU $ time ctest -t 16 < ~/data/enwiki-p10p30303 8.08user 0.25system 0:00.49elapsed 1680%CPU