12 ms·
Meta is using the Linux scheduler designed for Valve's Steam Deck on its servers
- 999900000999 10mo agoThat's the magic of open source. Valve can't say ohh noes you need a deluxe enterprise license.
- kstrauser 10mo agoI'm more surprised that the scheduler made for a handheld gaming console is also demonstrably good for Facebook's servers.
- bigyabai 10mo agoI mean, part of it is that Linux's default scheduler is braindead by modern standards: https://en.wikipedia.org/wiki/Completely_Fair_Scheduler https://en.wikipedia.org/wiki/Completely_Fair_Scheduler
- accelbred 10mo agoCFS was replaced by EEVDF, no?
- phdelightful 10mo agoParent's article says > Starting from version 6.6 of the Linux kernel, [CFS] was replaced by the EEVDF scheduler.[citation needed]
- 0x1ch 10mo agoI vaguely remember reading when this occurred. It was very recent no? Last few years for sure. > The Linux kernel began transitioning to EEVDF in version 6.6 (as a new option in 2024), moving away from the earlier Completely Fair Scheduler (CFS) in favor of a version of EEVDF proposed by Peter Zijlstra in 2023 [2-4]. More information regarding CFS can be found in CFS Scheduler.
- jorvi 10mo agoUltimately, CPU schedulers are about choosing which attributes to weigh more heavily. See this[0] diagram from Github. EEVDF isn't a straight upgrade on CFS. Nor is LAVD over either. Just traditionally, Linux schedulers have been rather esoteric to tune and by default they've been optimized for throughput and fairness over everything else. Good for workstations and servers, bad for everyone else. [0]https://tinyurl.com/mw6uw9vh https://tinyurl.com/mw6uw9vh
- 3eb7988a1663 10mo agoPart of that is the assumption that Amazon/Meta/Google all have dedicated engineers who should be doing nothing but tuning performance for 0.0001% efficiency gains. At the scale of millions of servers, those tweaks add up to real dollar savings, and I suspect little of how they run is stock.
- Anon1096 10mo agoThis is really just an example of survivorship bias and the power of Valve's good brand value. Big tech does in fact employ plenty of people working on the kernel to make 0.1% efficiency gains (for the reason you state), it's just not posted on HN. Someone would have found this eventually if not Valve. And the people at FB who worked to integrate Valve's work into the backend and test it and measure the gains are the same people who go looking for these kernel perf improvements all day.
- ranger207 10mo agoA lot of scheduler experimentation has been enabled by sched_ext: https://lwn.net/Articles/922405/ https://lwn.net/Articles/922405/
- giantrobot 10mo agoLatency-aware scheduling is important in a lot of domains. Getting video frames or controller input delivered on a deadline is a similar problem to getting voice or video packets delivered on a deadline. Meanwhile housecleaning processes like log rotation can sort of happen whenever.
- jorvi 10mo agoI mean.. many SteamOS flavors (and Linux distros in general have) have switched to Meta's Kyber IO scheduler to fix microstutter issues.. the knife cuts both ways :)
- bronson 10mo agoKyber is an I/O scheduler. Nothing to do with this article.
- Brian_K_White 10mo agoThe comment was perfectly valid and topical and applicable. It doesn't matter what kind of improvement Meta supplied that everyone else took up. It could have been better cache invalidation or better usb mouse support.
- senfiaj 10mo agoIn this case yes, but on the other hand Red Hat won't publish the RHEL code unless you have the binaries. The GPLv2 license requires you to provide the source code only if you provide the compiled binaries. In theory Meta can apply its own proprietary patches on Linux and don't publish the source code if it runs that patched Linux on its servers only.
- cherryteastain 10mo agoCan't anyone get a RHEL instance on their favorite cloud, dnf install whatever packages they want sources of, email Redhat to demand the sources, and shut down the instance?
- dfedbeef 10mo agoRHEL specifically makes it really annoying to see the source. You get a web view.
- OsrsNeedsf2P 10mo agoHonestly just hearing this makes me want to get all their binaries, request the code, scrape it with OCR and upload it somewhere
- dralley 10mo agoBut that would be silly, because all of the code and binaries is already available via CentOS Stream. There's nothing in RHEL that isn't already public at some point via CentOS Stream. There's nothing special or proprietary about the RHEL code. Access to the code isn't an issue, it's reconstructing an exact replica of RHEL from all of the different package versions that are available to you, which is a huge temporal superset of what is specifically in RHEL.
- Aperocky 10mo agoDon't forget RH is owned by IBM.
- 10mo ago
- sintax 10mo agoWell if you think about it, in this case the license is the 30% cut on every game you purchase on steam.
- HexPhantom 10mo agoExactly. Once the work is upstream and open, it stops being "Valve's thing" and just becomes part of the commons
- mikkupikku 10mo ago> SCX-LAVD has been worked on by Linux consulting firm Igalia under contract for Valve It seems like every time I read about this kind of stuff, it's being done by contractors. I think Proton is similar. Of course that makes it no less awesome, but it makes me wonder about the contractor to employee ratio at Valve. Do they pretty much stick to Steam/game development and contract out most of the rest?
- deleted 10mo ago[deleted]
- jvanderbot 10mo agoThey probably needed some point expertise on this one, as they build out their teams.
- treyd 10mo agoThey seem to be doing it through Igalia, which is a company based on specialized consulting for the Linux ecosystem, as opposed to hiring individual contractors. Your point still stands, but from my perspective this arrangement makes a lot of sense while the Igalia employees have better job security than they would as individual contractors.
- tapoxi 10mo agoValve is actually extremely small, I've heard estimates at around 350-400 people. They're also a flat organization, with all the good and bad that brings, so scaling with contractors is easier than bringing on employees that might want to work on something else instead.
- sneak 10mo ago300 people isn’t “extremely small” for a company. I don’t work with/for companies over 100 people, for example, and those are already quite big.
- hatthew 10mo agothe implied observation is that valve is extremely small relative to what it does and how big most people would expect it to be
- Fiveplus 10mo agoValve is practically singlehandedly dragging the Linux ecosystem forward in areas that nobody else wanted to touch. They needed Windows games to run on Linux so we got massive Proton/Wine advancements. They needed better display output for the deck and we got HDR and VRR support in wayland. They also needed smoother frame pacing and we got a scheduler that Zuck is now using to run data centers. Its funny to think that Meta's server efficiency is being improved because Valve paid Igalia to make Elden Ring stutter less on a portable Linux PC. This is the best kind of open source trickledown.
- MarleTangible 10mo agoOver time they're going to touch things that people were waiting for Microsoft to do for years. I don't have an example in mind at the moment, but it's a lot better to make the changes yourself than wait for OS or console manufacturer to take action.
- benoau 10mo ago"It just works" sleep and hibernate. "Slide left or right" CPU and GPU underclocking.
- pmontra 10mo agoSleep and hibernate don't just work on Windows unless Microsoft work with laptop and boards manufacturers to make Windows play nice with all those drivers. It's inevitable that it's hit and miss on any other OS that manufacturers don't care much about. Apple does nearly everything inside their walls, that's why it just works.
- Insanity 10mo ago“It just works” sadly isn’t true across the Apple Ecosystem anymore. Liquid Glass ruined multitasking UX on my iPad. :( Also my macbook (m4 pro) has random freezes where finder becomes entirely unresponsive. Not sure yet why this happens but thankfully it’s pretty rare.
- alecco 10mo ago[flagged]
- hoppyhoppy2 10mo agoGenerated comments are not allowed on HN.
- TZubiri 10mo agoI feel the intent of these rules are to forbid undisclosed aigen comments. If you ban disclosed usage of AIgen, you will get covert usage of AIgen
- deleted 10mo ago[deleted]
- alecco 10mo ago[flagged]
- littlestymaar 10mo ago
- loeg 10mo ago[flagged]
- hobobaggins 10mo agoPhoronix is blogspam?!
- webdevver 10mo agoyeah thats kinda harsh, phoronix is a good oss news aggregator at the very least, and the PTS is a huge boon for "whats the best bang for buck llvm build box" type of question (which is very useful!)
- deleted 10mo ago[deleted]
- zipy124 10mo agoIt is certainly not, I'm not sure where the commentor gets that view. Most likely because alongside their primary journalistic content they also produce secondary reporting like these short pieces, disseminating niche viewership content to a wider audience. I can see how it might be easy to see it as blog spam given the latter is almost entirely their purview, but it should not be misconstrued as such in this case.
- loeg 10mo agoYes.
- fph 10mo agoLife becomes a lot better the moment you stop considering Youtube videos valid primary sources.
- loeg 10mo agoIt’s a recording of a talk. Feel free to point out other sources but there doesn’t seem like much to object to here.
- redleader55 10mo agoIt's worth mentioning that sched_ext was developed at Meta. The schedulers are developed by several companies who collaborate to develop them, not just Meta or Valve or Italia and the development is done in a shared GitHub repo - https://github.com/sched-ext/scx https://github.com/sched-ext/scx.
- tayo42 10mo agoInteresting to see server workloads take ideas from other areas. I saw recently that some of the k8s specific os do their updates like android devices
- binary132 10mo agoI'm struggling to understand what workloads Meta might be running that are _this_ latency-critical.
- commandersaki 10mo agoAccording to the video linked somewhere in this thread indicates WhatsApp Erlang workers that want sub-ms latency.
- Pr0Ger 10mo agoIt's definitely for ads auctions
- tayo42 10mo agoIf you have 50,000 servers for your service, and you can reduce that by 1 percent, you save 50 servers. Multiply that by maybe $8k per server and you have saved $400k,you just paid for your self for a year. With meta the numbers are probably a bit bigger.
- pixelbeat__ 10mo agoLOL (I used to work for Meta, so appreciate the facetious understatement)
- bongodongobob 10mo agoThat's not how it works though. Budgets are annual. A 1% savings of cpu cycles doesn't show up anywhere, it's a rounding error. They don't have a guy that pulls the servers and sells them ahead of the projection. You bought them for 5 years and they're staying. 5 years from now, that 1% got eaten up by other shit.
- tayo42 10mo agoYou don't buy servers once every 5 years. I've done purchasing every quarter and forecasted a year out. You reduce your services budget for hardware by the amount saved for that year.
- tra3 10mo agoI'm curious how this came to be: > Meta has found that the scheduler can actually adapt and work very well on the hyperscaler's large servers. I'm not at all in the know about this, so it would not even occur to me to test it. Is it the case that if you're optimizing Linux performance you'd just try whatever is available?
- laweijfmvo 10mo agoalmost certainly bottom-up: some eng somewhere read about it, ran a test, saw positive results, and it bubbles up from there. this is still how lots of cool things happen at big companies like Meta.
- erichocean 10mo agoOmarchy should adopt the SCX-LAVD scheduler as its default, it helps conserve power on laptops.
- Sparkyte 10mo agoI've been using Bazzite Desktop for 4 months now and it has been my everything. Windows is just abandonware now even with every update they push. It is clunky and hard to manage.
- aucisson_masque 10mo agoIsn't bazzite a gaming focused distribution ? It seems weird to install it on a PC that does 'my everything'. I wouldn't make excel spreadsheet on the steam deck for instance.
- pawelduda 10mo agoWhy not? It has full desktop mode with Plasma and can be docked like PC
- 0x1ch 10mo agoBazzite is advertised for gamers, however from my understanding it's just Fedora Atomic wrapped up to work well on steamdeck adjacent hardware and gaming is a top priority. You'd still be receiving the same level of quality you would expect from Fedora/RHEL (I would think).
- Sparkyte 10mo agoPrecisely, I like it's commitment to the Fedora Atomic. Fedora is in my opinion the best user experience Linux out there, not just because Linus Torvalds said it was his favorite. Probably not the best server or best to base a console OS on, but as a daily driver consistency is more important. Keeping things in flatpaks makes it easy to manage what is installed too.
- Sparkyte 10mo agoGaming or not, stability is important. An OS that focuses on gaming will typically focus on stability, neither bleeding edge or lag behind in support. Has to update enough to work with certain games and behind enough to not have weird support isues. So Bazzite in my opinion is probably one of the best user experience flavors of Fedora around. Yes you can do more than gaming on Bazzite.
- balls187 10mo agoHow well does Linux handle game streaming? I’m just now getting into it, and now that Windows10 is dead, I want to move my desktop PC over to linux, and end my relationship with Microsoft, formally.
- Kholin 10mo agoIt works will. I've tried used Sunshine as stream server and Moonlight as client to play games on my Steam Deck, my PC installed openSUSE Tumbleweed with KDE Plasma. There may be some key binding issues, but they can be solved with a little setup.
- DebugDruid 10mo agoLooks like open source helped create Silicon Valley, while no IP laws made Shenzhen. Sharing seems to really drive industry growth, so maybe the US and EU should rethink their IP laws?
- deleted 10mo ago[deleted]
- shmerl 10mo agoCan't find scxctl in Debian. Was it never packaged?
- ahartmetz 10mo agoI keep being puzzled by the unwillingness of developers to deal with scheduling issues. Many developers avoid optimization, almost all avoid scheduling. There are some pretty interesting algorithms and data structures in that space, and doing it well almost always improves user experience. Often it even decreases total wall-clock time for a given set of tasks.
- HexPhantom 10mo agoSomething built to shave off latency on a handheld gaming device ends up scaling to hyperscale servers, not because anyone planned it that way, but because the abstraction was done right