4 ms·
The conclusion I take away from this is that good performance isn't necessarily more useful than predictable performance. For many language implementations, the
by skybrian 12y ago
The conclusion I take away from this is that good performance isn't necessarily more useful than predictable performance. For many language implementations, the performance characteristics are quite fragile, and you'd better be benchmarking after every change or you won't notice a regression.
Even setting up a benchmark is tricky because the same binary can have dramatically different performance depending on the environment. At the machine level, caching effects from different processors can have dramatically different effects on performance. But compilers and runtimes can easily make things even more unpredictable, requiring things like warmup time to even hope to get an idea of what's going on.
I think Go starts out a bit closer to achieving predictable performance; any ahead-of-time compiler has an advantage here. Other than the use of garbage collection, the whole language seems to be designed with predictable performance in mind over ease of use, and you can avoid GC where it matters. Also, recently they replaced segmented stacks with copying of contiguous stacks, which both improves performance and makes it more predictable (while a goroutine is running).
- pron 12y ago> good performance isn't necessarily more useful than predictable performance That may be true, but on modern machines it's pretty much a pipe dream, as even the hardware nowadays (let alone modern OSs) is unpredictable. So once you get to the compiler it's a little late to get predictability. I think we're better off foregoing predictability altogether -- unless when writing real time applications (and I'm using realtime in its original sense) -- and getting the best performance we can. JITs currently seem to be the best hope for top performance, especially for a given unit of manual programmer effort expended on optimization.
- _dps 12y ago(note: this comes mostly from a background in scientific/numerical programming in C) You are certainly right if one takes "predictable" to mean something close to "deterministic"; on the other hand I think there are numerous almost-deterministic characteristics that can be evaluated without resorting to plain-ol' empiricism (particularly involving likelihood of hits in L1/L2/L3 and likelihood of being able to speculatively execute particular branches of code). So I would agree that my a priori predictions may be a factor of 2-3x off; OTOH I often succeed in predicting that a particular tight loop will never have to leave L1, or that a low-entropy condition in the loop will be made essentially irrelevant by speculative execution. With that proviso, I share your belief that JITs seem to be the best hope for top performance (and I am a heavy user of LuaJIT).
- seanmcdirmid 12y agoIf you use CUDA + a GPU, you know for sure, since you are basically responsible for scheduling all memory movements yourself. On the other hand, even CUDA is doing some optimizations behind the scenes that can drastically effect the performance of your code in ways that are not so obvious. A lot of the HPC work has moved over to GPUs, it is amazing what one can do when you have almost complete control over the memory hierarchy.
- pron 12y agoMemory and caching is a big part of nondeterminism, but it's not just about memory. GPUs employ a far more primitive execution strategy than modern server CPUs with their branch prediction, ILP etc. Also, GPUs, while terrific for parallel workloads, are terrible for concurrent workloads, which require very intricate branching.
- seanmcdirmid 12y agoWell, you can branch as much as you want on the GPU, as long as your branches are coherent among each thread sharing the same control unit :) I wouldn't say this is "primitive", just a very different way in thinking about computation that has been very very very effective for a lot of use cases. GPUs are obviously not the solution for concurrent or irregular workloads (yet), but many are surprised how much mileage one can get out of Python + CUDA for scientific workloads. The only point I was making is that there is a world where the hardware is much more deterministic (even if most of us can't go there).
- pron 12y agoNo argument there. BTW, for anyone interested in an overview of the non-determinism built underlying modern hardware architectures, I recommend watching this great talk[1] -- A Crash Course in Modern Hardware -- by Cliff Click, one of the world's top JIT experts. [1]: http://www.infoq.com/presentations/click-crash-course-modern-hardware http://www.infoq.com/presentations/click-crash-course-modern...
- nostrademons 12y agoI remember reading a thread once about Kenton trying to optimize some protobuf deserialization routines. After much profiling, the outcome was basically "I can't believe the compiler is doing this. It means it is basically impossible to predict the performance characteristics of source code." At this point, a number of old compiler guys jumped in (we have a number of folks on the C++ standards committee at Google, and most of the team who wrote HotSpot), and said "Yeah. The best way to make sure your program runs fast is to get it into a compiler team's benchmark." Chrome/V8 has this problem as well - if you talk to really skilled web developers, they have a lot of performance "tricks" in their head, and the pitfall is that this knowledge decays much more rapidly than people think it decays, and so what was common knowledge in 2008 (or even 2013) is now no longer true. One major problem we faced with Google Search and specifically Instant was that it was optimized for the performance characteristics of the browsers of 2008; pretty much none of those rules apply to modern Chrome and Safari on mobile networks, and so performance is ridiculously bad on mobile. Go achieves predictable performance only because it's relatively new. It's true that the team has tried very hard to keep things simple and predictable, but the problem is that the hardware that Go runs on keeps changing as well. As a language it was designed to take advantage of multicore chips that are just coming into use now. What if in 5 years we're all using quantum computers, or memristors, or flash memory, or the memory hierarchy no longer applies? What if everything is peer-to-peer over mobile devices?
- pcwalton 12y ago> It's true that the team has tried very hard to keep things simple and predictable, but the problem is that the hardware that Go runs on keeps changing as well. As a language it was designed to take advantage of multicore chips that are just coming into use now. This has already happened; what many people don't realize is that we're arguably past the multicore era. Desktop-class CPUs aren't adding more cores as quickly as we had predicted, because people aren't using them (although note that this is different in mobile). What they are adding is wider and wider SIMD units, and more and more SIMD instructions. To get maximum performance on the CPUs of today, as well as those in the future, using SIMD effectively is every bit as important, or more, as using multiple cores effectively. In my opinion, programming languages have been fairly slow at responding to this.
- taeric 12y agoThere is no good takeaway on this benchmark, honestly. If you are doing any sort of mathematically focused programming and you are not delving into the math yourself, expect odd results from compiler optimizations. Heaven help you if you start using libraries that are focused on this sort of thing.