7 ms·
Bpftune uses BPF to auto-tune Linux systems
- 6SixTy 2y agoAs an insane Gentoo user, sign me the F*** up
- gerdesj 2y agoYou are not insane. You are running a distro that enables you to patch both the distro package definitions and the source code that is compiled by those definitions. Then there is the day to day stuff such as USE ... I have a VM that was rather unloved that I ended up using git to go back in time to gradually move it forwards in time and update. It took quite a while. It started off life in the noughties and now sports a 6.10 kernel. I won't bore you further but it did take a while but the data is still there and accessible on a modern platform. What is mad about that?
- fragmede 2y agoBack in the day when Gentoo was in favor, USE flags and recompiling the world could make a huge difference in performance on much slower computers with far more limited RAM, and you cloths really see the difference. After waiting week for the world to recompile, that it. These days, unless you're, a hyperscaler and need to eke that last 10% of performance our of your hardware, ain't nobody got time for that. emerge --update --deep --with-bdeps=y --newuse @world
- gerdesj 2y agoI was a Gentoo aficionado for quite a while (20+ years). I'm going back too (rather slowly) There is a lot of very decent technology in Gentoo - it literally describes what all other distros need to do, in some cases. You mention a week of compilation - I remember the gcc 4 thing too!
- bloopernova 2y agoFascinating! I'd like to hear from people who are running this. Is it effective? Worth the setup time?
- usr1106 2y agoInteresting. But if tuning parameters to their best values were easy, shouldn't the kernel just do that in the first place?
- nitinreddy88 2y agoIt depends on workload. This tool generates recommended config for that specific machine workload. App Nodes can have completely different recommendations vs Database Nodes. It will be completely different for Workstation.
- usr1106 2y agoSure, but the kernel could just do the same. Of course the kernel is already too big. Is BPF the right level to make it more modular? Just thinking, I don't think I have the answer.
- onetoo 2y agoThis doesn't necessarily find the best parameters, and it doesn't necessarily do it easily. From my reading, it will converge on a local optimum, and it may take some time to do that. In theory, I don't see why the kernel couldn't have a parameter-auto-tune similar to this. In practice, I think the kernel has to work in so many different domains, it'd be impossible to land on a "globally good enough" set of tuning heuristics. I'm far from a kernel developer, so I'm ready to be corrected here. IMO if we ever see something like this deployed widely, it will be because a popular distribution decided to install it by default.
- RandomThoughts3 2y agoI would reverse the question: if it can be done by a BPF module, why should it be in the kernel? Distributions turning it on by default is another story. Maybe it deserves to be shiped on all the time but that's not the same thing as being part of the kernel.
- jiehong 2y agoIndeed! The kernel might already be too monolithic. This kernel parameters optimisation reminds me of PGO compilation in programs. Yet, perhaps the kernel could come with multiple defaults config files, each being a good base for different workloads: server, embedded, laptop, mobile, database, router, etc.
- robinhoodexe 2y agoIs tuning the TCP buffer size for instance worth it?
- viraptor 2y agoIt depends. At home - probably not. On a fleet of 2000 machines where you want to keep network utilisation close to 100% with maximal throughput, and the non-optional settings translate to a not-trivial value in $ - yes.
- londons_explore 2y agoTCP parameters are a classic example of where an autotuner might bite you in the ass... Imagine your tuner keeps making the congestion control more aggressive, filling network links up to 99.99% to get more data through... But then any other users of the network see super high latency and packet loss and fail because the tuner isn't aware of anything it isn't specifically measuring - and it's just been told to make this one application run as fast as possible.
- withinboredom 2y agoIt literally covers this exact scenario in the readme and explains how it prevents that.
- crest 2y agoIt depends mostly on the bandwidth-delay-product and packet loss you expect on each connection. A there is a vast difference between a local interactive SSH session and downloading a large VM image from across an ocean.
- zymhan 2y agoThere's not much cost to doing it, so yes.
- toast0 2y agoIn my experience running big servers, tuning TCP buffers is definitely worth it, because different kinds of servers have different needs. It doesn't often work miracles, but tuning buffers is low cost, so the potential for a small positive impact is often worth the time to try. If your servers communicate at high datarates with a handful of other servers, some of which are far away, but all of which have large potential throughput, you want big buffers. Big buffers allow you to have a large amount of data in flight to remote systems, which lets you maintain throughput regardless of where your servers are. You'd know to look at making buffers bigger if your throughput to far away servers is poor. If you're providing large numbers of large downloads to public clients that are worldwide from servers in the US only, you probably want smaller buffers. Larger buffers would help with throughput to far away clients, but slow, far away clients will use a lot of buffer space and limit your concurrency. Clients that disappear mid download will tie up buffers until the connection is torn down and it's nice if that's less memory for each instance. You'd know to look at making buffers smaller if you're using more memory than you think is appropriate for network buffers... a prereq is monitoring memory use by type. If you're serving dynamic web pages, you want your tcp buffers to be at least as big as your largest page, so that your dynamic generation never has to block for a slow client. You'd know to look at this if you see a lot of servers blocked on sending to clients, and/or if you see divergent server measured response times for things that should be consistent. This is one case where getting buffer sizes right can enable miracles; Apache pre-fork+mod_PHP can scale amazingly well or amazingly poorly; it scales well when you can use an accept filter so apache doesn't get a socket until the request is ready to be read, and PHP/apache can send the whole response to the tcp buffer without waiting, then closes the socket; letting the kernel deal with it from there. Keep-alive and TLS make this a bit harder, but the basic idea of having enough room to buffer the whole page still fits.
- BSDobelix 2y agoBTW one can use it out of the box with CachyOS. After installation -> CachyOS Hello -> Apps/Tweaks
- bastloing 2y agoIt's great how it grew out of simple packet filtering into tracing and monitoring. It's one of those great tools most should know. Been using it for years.
- orbital-decay 2y agoThe entire modern ML and most of the HPC grew out of very simple programmable shaders in GeForce 3.
- gmuslera 2y agoTwo words: “feedback loop”. That was the first idea that jumped in when thinking in what could go wrong, not because the Linux kernel, or BPF or this program, just for how it is intended to work. There might be no risk of that happening, there may be controls around that, or if they happen they might only converge to an estable state, but still it is something to have in the map.
- marcosdumay 2y ago> or if they happen they might only converge to an stable state That one will always be dependent on the usage patterns. So the auto-tuner can't guarantee it. Also, I imagine the risk of the feedback turning positive is related to the system load (not CPU, but the usage of the resources you are optimizing). If so, it will make your computer less able to manage load. But this can still be useful for optimizing for latency.
- nevon 2y agoI wonder how effective this would be in multi-tenant environments like shared k8s clusters. On the one hand, each application running will have a different purpose and will move around between nodes over time, but on the other hand there are likely broad similarities between most applications.
- mrbluecoat 2y ago> bpftune is designed to be zero configuration; there are no options On behalf of every junior administrator, overworked IT admin, and security-concerned "cattle" wrangler, thank you. Having to learn a thousand+ knobs & dials means most will never be touched. I for one welcome automated assistance in this area, even if the results are imperfect.
- sgarland 2y agoI think it’s still important to know what those dials and knobs do, otherwise (as the currently top-voted comment says) when things break, you’ll be lost.
- zymhan 2y agoThat's exactly why it's such a burden.
- sgarland 2y agoI continue to be amazed and frustrated that people will simultaneously believe that deeply understanding a programming language is a noble pursuit, but that deeply understanding the software that allows their code to run is somehow burdensome. Ops remains extremely important and extremely real. You can abstract it away in exchange for higher costs and less control, but ultimately someone at some level has studied these parameters, and decided what is best for your code.
- xorcist 2y agoKnobs exist for a reason. If there was no reason for it it shouldn't exist. Turning knobs automatically is one of two things: It's either an awfully bad idea, or should be turned into an upstream patch. Speaking from practice, it's usually the former.
- gausswho 2y agoWith this tool I am wary that I'll encounter system issues that are dramatically more difficult to diagnose and troubleshoot because I'll have drifted from a standard distro configuration. And in ways I'm unaware of. Is this a reasonable hesitation?
- pbhjpbhj 2y ago>"bpftune logs to syslog so /var/log/messages will contain details of any tuning carried out." (from OP GitHub readme) The rmem example seems to allay fears that it will make changes one can't reverse.
- admax88qqq 2y agoIt’s not a questions of being able to reverse. It’s a question of being able to diagnose that one of these changes even was the problem and if so which one.
- pbhjpbhj 2y agoI focused primarily on guesswho's "in ways I am unaware of". Your issue appears to be true for any system change. Although, risk will of course vary.
- nehal3m 2y agoIf they can be reversed individually you can simply deduce by rolling back changes one by one, no?
- jstanley 2y agoOnly if you already suspect that this tool caused the problem.
- spenczar5 2y agoSuppose you run a fleet of a thousand machines. They all autotune. They are, lets say, serving cached video, or something. You notice that your aggregate error rate been drifting upwards since using bpftune. It turns out, in reality, there is some complex interaction between the tuning and your routers, or your TOR switches, or whatever - there is feedback that causes oscillations in a tuned value, swinging between too high and too low. Can you see how this is not a matter of simple deduction and rollbacks? This scenario is plausible. Autotuning generally has issues with feedback, since the overall system lacks control theoretic structure. And the premise here is that you use this to tune a large number of machines where individual admin is infeasible.
- almucentaret 2y ago[dead]
- klysm 2y agoThis seems to step into control theory, which I think is somewhat underapplied in software engineering.
- hrdwdmrbl 2y agoWhat is the magnitude of the effect size? The readme provides no information about what kind of results should be expected or might result.
- deleted 2y ago[deleted]
- chipolinapolina 2y ago[dead]