9 ms·
Linux needs a standard set of benchmarks that are vaguely representative of things users use Linux for. Phone, laptop and a few server use cases. It then nee
by londons_explore 5y ago
Linux needs a standard set of benchmarks that are vaguely representative of things users use Linux for. Phone, laptop and a few server use cases.
It then needs someone to do a big parameter tuning to select optimal settings.
Too many decent algorithms don't make it into the kernel because there are too many tunables, and the ones that do typically arent well tuned for anyone's use case.
Even big projects like Ubuntu typically don't change many tunables in the kernel.
- voldacar 5y agoI wonder if someday you could make a differentiable kernel and do gradient descent in kernel-parameter-space
- astrange 5y agoTuning is highly overrated in software performance, eg compiler optimizations don't help that much and searching their params helps even less. I think it comes from people wishing they could change their software without actually needing to learn how to change it.
- jeffbee 5y agoThat's quite a bad take. You can bank on PGO+LTO of a C++ server being good for 20% throughput easily, and not compiling for k8-generic when you have a modern machine also makes a big difference. I would agree that just flipping compiler flags might not get you very far, but there are key inputs that you shouldn't leave on the table.
- astrange 5y agoBy "not that much" I mean orders of magnitude. 20% is reasonable, especially for C++ that needs a lot of inlining. But a more modern language would benefit from defining some optimizations like inlining as mandatory, the same way tail calls are (and of course GCC has this.) Then there isn't a chance of deploying a build with extra-slow behavior.
- josephg 5y agoA performance improvement of 20% would be a massive win for something as large and widely used and as the linux kernel. That would average out into a few percentage points of performance improvement in every application across the globe which runs on linux / android. Merging a patch like that in linux would be like quietly reaching into every device across the planet and giving them a small, free CPU improvement and drop in power usage. Not game changing for any individual user, but huge in aggregate.
- arp242 5y agoAnd if you run a lot of servers it means you need fewer servers. These kind of changes can literally save millions.
- jeffbee 5y agoWhich is why the organizations which stand to save millions by building their kernels with PGO already do it.
- josephg 5y agoSome do. I bet lots of companies miss out on massive cost savings like this simply because nobody at the company has the skills and access to improve things. I heard a story - years ago Google hired an engineer who happened to be an expert from a previous life in video codecs. He wasn’t working on YouTube at Google, but out of interest he pulled up the YouTube source code to see what it did. They were just using the defaults for some of the encoding parameters. He tweaked a few of the encoding parameters and in doing so saved Google millions of dollars per year in compute/storage/network traffic. A few hours of his work was probably more beneficial for Google than years spent in his primary role. And if tuning opportunities like this abound at Google, you know they’re everywhere. It’d be much better if Linux distributions simply shipped kernels which are already compiled with PGO, with a reasonable profile based on normalish use.
- jorvi 5y agoI don't know if you would consider it tuning, but a low latency kernel is a massive UX improvement on consumer devices, where you don't want the UI to stutter and lock up under heavy load, and don't mind paying (could be wrong, top of my head) like a 5% throughout penalty.
- yokaze 5y agoCounterexamples: Running your DB with the default parameters. Not running Linux kernel parameters depending on your workload. (+10Gbit NICs, lots of connections...) There are reasons, why those knobs are there, as there are always trade-off, and having them automatically adjust is very hard. But maybe that was what you're referring to.
- cycomanic 5y agoThat highly depends on your problem. I have some scientific code that gets more than a factor 10 improvement on runtime just by turning on some compiler flags (mainly -O3 and -march, but some others as well).
- bayindirh 5y agoI also have a similar scientific code base which gets 10-15x wall-time improvement just by enabling compiler optimizations.
- IntelMiner 5y agoNot a scientific codebase, but ffmpeg seems to benefit massively from optimization I use it to encode my DVD and Blu Ray TV/movies to HEVC. Doing so reduces the filesize on DVD's by roughly 80% and Blu Rays by 60% The downside is the encoding is an incredibly CPU intensive process. Hardware encoders like Nvidia's Nvenc or Intel's QuicSync look absolutely terrible and are a non-starter for archival storage On a stock Fedora XFCE install, I would get roughly 0.5 FPS for a 1080p Blu Ray file (29.97 FPS at 1920x1080) A Gentoo Linux installation with a global -O3 -march=native as well as LTO, PGO and Graphite enabled globally boosts it from 0.5 to roughly 1.3 FPS. Still slower than realtime, but an absolutely massive improvement
- ta988 5y agoThat's an amazing compression ratio, how is the quality?
- IntelMiner 5y agoffmpeg recommends a crf of 23 to preserve 1:1 for H.264 -> h.265 I do a probably overkill CRF of 20 just to be safe. But everything looks absolutely perfect, even blown up on my 65 inch (1080p) TV
- bayindirh 5y ago
- dspillett 5y agoIt is overrated when you are tuning the wrong thing, but if it is the bottleneck in a significant process for you use case then it is often very significant especially at scale. For someone running a compute cluster for their own needs or as a cloud service a couple of percent improvement like this represents a massive gain in throughput or a saving in the need for extra kit (or power costs - there may even be a small environmental benefit). Of course on the scale of you or I, such optimisations may be little more than a curiosity most of the time. A process that normally takes 100 minutes (a video transcode, perhaps) now taking 95 followed by the machine being idle while it waits for us to look back and see the result, is not really benefiting from the benefit. > without actually needing to learn how to change it This is certainly a problem sometimes, but not an argument for tuning bring overrated in general. It usually comes down to tuning the wrong thing, like someone playing with mysql engine & kernel IO parameters to eek out a fraction of a of % bonus when they could improve index structures, or fix queries with no sargable predicates, or both in unison, to get benefits measured in orders of magnitude. It can be overrated in the case of average gamers tweaking their hardware to get a few extra FPS on top of the many tens they already get, but again this not being worth the time for some, even many, doesn't mean it can make a huge difference to a pro-gamer or someone using old kit where "a few extra" is a relatively large gain.
- rwmj 5y agoWe just improved the performance of one copying tool by a factor of 2.9, by tuning the size of a memory buffer. (BTW a shout out to both Flamegraphs and hyperfine - both excellent tools for profiling and benchmarking respectively)
- nitrogen 5y agoTuning kernel/process priorities has made a big difference in my audio code. Tweaking the GPU and USB IRQ handler priorities let me do realtime audio together with realtime OpenGL data visualization and screen recording, where before the audio would skip or the visualization would hang.
- mhh__ 5y agoOne of the things that struck me recently is that working out what a computer is actually spending it's time doing is now almost a Master's thesis to do properly.
- adrianN 5y agoDepending on the level of analysis you can even get a PhD. See for example https://core.ac.uk/download/pdf/18462805.pdf https://core.ac.uk/download/pdf/18462805.pdf
- mhh__ 5y agoNot surprising at all, sadly.
- bayindirh 5y agoI've found it the hard way when I was trying to profile my own code. If you're using a few optimized libraries and designed your code for high-speed parallel execution, the compiled thing becomes almost impossible to trace in a practical manner.
- rwmj 5y agoDidn't perf work? What difficulties did you encounter?
- bayindirh 5y agoUsed perf record for getting CPU efficiency numbers, and it worked very well, however to be able to reliably trace the code, I had to compile it in debug mode with no optimizations, and run with a single thread. That combination increased the execution time considerably. Some of the Matrix and solver code coming from Eigen is heavily optimized for SIMD operations and using it with -O0 is just painful. Moreover, I had to verify its memory sanity and used Valgrind for that. Using a full size problem meant it had to churn for days to finish execution. Having deadlines doesn't always help in this stuff.
- 5y ago
- viraptor 5y ago> Linux needs a standard set of benchmarks that are vaguely representative of things users use Linux for. Have you seen phoronix framework + benchmarks? They use common tools, but did some good work on making the tests repeatable and accessible to anyone else. https://github.com/phoronix-test-suite/phoronix-test-suite/ https://github.com/phoronix-test-suite/phoronix-test-suite/
- fgeiger 5y agoLinaro is also doing a lot in that space (at least for ARM): https://www.linaro.org/os-build-and-test/ https://www.linaro.org/os-build-and-test/
- beebeepka 5y agoJust minor nitpick. Phoronix is one guy
- acer4666 5y agoWithout knowledge of preferred pronouns, "they" seems appropriate.
- UbrtrbNchDneRle 5y agoviraptor 1 : 0 the west.
- fgeiger 5y agoAs I read it, the point was not about the wrong pronoun but about the fact that the suite is developed by one person and not a large team and that the burden of benchmarking Linux performance should maybe not rest on one person alone.
- rat9988 5y agoYou missed his point. There was no wrong pronoun, this is what he is claiming.
- 5y ago
- diegocg 5y agoThere are already plenty of benchmarks
- api 5y agoA few server use cases? It's used for like 99% of all server use cases.
- guenthert 5y agoRight, problem solved. Now we just need to identify the few representative use cases. Is it people using a phone, as, er, phone with actual voice communication? or video conferencing? Or those who play games casually or not-quite-so-casually? Or video streaming? Or just for texting and would rather prioritize security and battery runtime? And that's just the phone. I could come up with many more questions regarding desktop and server usage as I'm more familiar with those. I'm afraid, there is no one optimal set of configuration options. Not even two or three.