8 ms·
Linux 7.3 improves performance when running out of vRAM
- pranav_tech26 2mo ago[flagged]
- imfemambocus 2mo agoThis is a nice blog and it makes sense to me now. As a gamer and linux user myself, I've previously had to do tweaks and go-arounds without really understanding what was going on behind the scenes. :)
- Lunar5227 2mo agoWell written and very informative. I am glad we have these enthusiastic people around for Linux kernel development!
- cubefox 2mo agoEnthusiasm seems proportional to the number of exclamation marks (28)! (:
- bcjdjsndon 2mo ago[flagged]
- fizzbuzzbarbazz 2mo agomeh.
- flaburgan 2mo agoAs proves the fact that this just got merged. :roll_eyes:
- bcjdjsndon 2mo ago[flagged]
- mdp2021 2mo agoIdle jokes have no use here.
- skew-aberration 2mo agoGreat writeup, gpuvis looks particularly interesting and glad the kernel is providing tracepoints for performance events. > Not only does the display hardware like scanned-out images to be in VRAM, it also completely skips past the GPU’s virtual memory architecture and works with physical addresses exclusively. Well there's your problem. Only so smart your memory management can be when you have to pay the cost of doing it manually. Although presumably this only applies to a small fraction of the VRAM?
- NekkoDroid 2mo ago> Although presumably this only applies to a small fraction of the VRAM? They did mention they saw 4GiB of eviction for a single 32MiB scan out image. So while I would call the image allocation small, it seems to cause an avalanche of evictions. Amplified by the fact that each frame has one of these images, though I expect subsequent frames might have a better chance of already fitting into evicted space. What I don't exactly understand is: doesn't it make sense to always reserve the contiguous physical memory for this case and not allow anything else to be put in it?
- jojomodding 2mo agoOr alternatively, instead of evicting it all, can you move the data around in physical memory while updating the page table, so that you can clear a large enough continuous block of physical memory?
- hypfer 2mo agoI'll be the one to ask the obvious question: What does this mean for compute workloads? Specifically, LLM inference. Does it mean anything at all, or is this purely a games-thing?
- skew-aberration 2mo agoI doubt it makes much of a difference, and you can always manually manage what data lives in the GPU when if you 100% have to overcommit. Games have a much larger and more diverse set of objects in the VRAM, and their usage is less predictable, so manual scheduling of the memory is infeasible typically.
- zozbot234 2mo agoIf you have model layers/experts that reside in CPU RAM, it's generally better to do that part of inference on the CPU than pay the cost of shipping them over to the GPU. LLM inference is generally bottlenecked by data-movement, not compute. There may be a limited exception for prefill or perhaps decode of very wide batches, where shuffling the data around may be justified.
- kjuulh 2mo agoGreat article. I find that I learn something every time I read a post about linux kernel work. I guess an LRU with priority would handle VRAM for games pretty decently without going getting too application specific. What about VRAM to Disk specifically NVME, would direct to disk be feasible for large workloads, I know it is used for streaming in assets directly via. PCIE, but i wonder how the performance would be on compute workloads running with NVME as a swap for GPU VRAM.
- PunchyHamster 2mo ago> but i wonder how the performance would be on compute workloads running with NVME as a swap for GPU VRAM. 4x as slow in absolute best case, NVMe drives have 4 PCIe lanes usually
- d3Xt3r 2mo agoGosh, 7.2 literally just dropped with a bunch of awesome of performance/gaming related improvements (large folios, cache-aware scheduling, improved MGLRU reclaiming, Fair GPU Scheduler etc)... and I already can't wait for 7.3 to come out. Meanwhile in the Windows world, users hate updates... Like I genuinely can't think of a single instance that made users exclaim, "oh boy I just can't wait for the next Patch Tuesday!".
- toredash 2mo agoDidn't we eagerly await on updates in the good old days of Windows?
- a012 2mo agoThe only Windows update that made me “eagerly await” is the Windows XP SP2.
- toredash 2mo agoAnd what an update it was. I still remember how Windows ME looked at first, the feeling of something better. That didn't last long
- alt227 2mo agoBeing able to preview media files directly in the preview pane of explorer was mind blowing to me. This feature alone made me love ME, let alone the interface customisation it allowed.
- patrickmcnamara 2mo agoWindows 7 was pretty neat.
- frollogaston 2mo agoMy Mac ran XP SP2 in Boot Camp for an insanely long time. Best version of Windows that will ever exist.
- exceptione 2mo agoGreat article! I share the same hunch as the author does; when allocating memory ultimately the application itself is in the best position to inform the kernel about the desired stickiness to VRAM. The best a kernel can do is guessing. As a side note, it strikes me how much we owe to young trans people for low level performance engineering.
- weird-eye-issue 2mo ago> As a side note, it strikes me how much we owe to young trans people for low level performance engineering. wtf?
- exceptione 2mo agoJust an observation.
- weird-eye-issue 2mo agoWould you have made that observation and pointed it out if they were not trans?
- exceptione 2mo agoI am inclined to rather point out remarkable things than unremarkable things. There seems to be relatively many trans people in this niche. If I see a soil that is unusually rich in iron, I would be more inclined to point that out. If I come across a piece of soil that is as average as the 98% of the surrounding lands I wouldn't be less likely to point out "hey look, this soil is unremarkable average." Also, it was just an aside from me. It is unclear what your problem is exactly, and I am also not sure if a tech forum could provide counsel for that.
- weird-eye-issue 2mo agoMan that's such a terrible analogy. In your analogy of course you wouldn't point out the normal soil because it didn't do anything remarkable. But in this case people did do something remarkable (unrelated to their attributes) but it feels like you only pointed out certain attributes about them because they were trans. In fact your analogy is a great illustration of one of the critiques that a lot of people have with the DEI movement because you are saying that you would point out the soil simply for having specific attributes rather than what it actually achieves. > It is unclear what your problem is exactly, and I am also not sure if a tech forum could provide counsel for that. So now you are making this personal and trying to say that I have some sort of problem? That's very interesting and I think you might be projecting here a little bit
- londons_explore 2mo agoI'm mostly excited for the VRAM savings from apps and games which have loads of unused or super rarely used textures. Game developers often aren't super careful with this stuff, and there could be literal gigabytes of data which isn't used at all.
- FartyMcFarter 2mo agoI worked on a PC/PS4/Xbox One game once. I wish I remembered more details, but at some point when looking at memory usage I found a 2048x2048 (or something like that) texture for something on a character. This texture was never anywhere near occupying the full screen. It got downsized once I told one of the artists, but I wouldn't be surprised if some games are inadvertently shipping with that kind of waste.
- yxhuvud 2mo agoThe release version of City Skylines 2 had enormous issues due to this problem. Some tiny details, like a pile of wood behind a shed, had ridiculous amount of pixels. I do believe that should be pretty much fixed in that title nowadays, however.
- asimovDev 2mo agowasn't it rendering every pedestrian's teeth in detail as well? hilarious oversight
- stuaxo 2mo agoAre there any people out there doing this optimisation post-release for existing games ?
- NekkoDroid 2mo agoI don't know about that specific optimisation, but I remember Helldivers 2 did drop support for loading from HDDs, which allowed them to deduplicate assets bringing the game down from what I remember like 120GB to somewhere round 30GB.
- krisknez 2mo agoI hope there will be an update where when my RAM gets full my PC doesn't freeze and becomes unusable... I remember that Linux and Windows do this in different ways and Windows doesn't have the problem.
- deleted 2mo ago[deleted]
- kalaksi 2mo agoMe too, but you can improve the behavior yourself too. It's been an issue with desktop linux and the default settings for a long time. You could maybe tune OOM killer settings, but another option is to use a package like earlyoom: https://man.archlinux.org/man/earlyoom.1.en https://man.archlinux.org/man/earlyoom.1.en
- ramon156 2mo agoearlyoom is what you're looking for. it'll crash the process instead of hang
- BitPirate 2mo agozswap and systemd-oomd/earlyoom are your friends.
- deleted 2mo ago[deleted]
- ggus 2mo agomacOS too sadly. by the time you realise there's a problem the machine is already comatose and unresponsive all these electron apps and invisible virtual machines (looking at you Claude) really don't help.
- Aperocky 2mo agoOne of the biggest reason I migrated to Codex. I'd like to see each agentic CLI process as a simple building block, and having 100 claude session means I sometimes run out of memory (and I have 256GB of them). Codex never have this problem. The communication is more direct too with GPT5.6 family of LLM. No more belt and suspenders.
- nickjj 2mo agoThis post focuses on performance, but what about crashing due to apps seeing no RAM available to allocate once VRAM is filled up? I remember having this problem when I used an NVIDIA 750 Ti with 2 GB of memory. Just opening a few Firefox or Ghostty terminals (both are GPU accelerated) would result in those apps crashing or not being able to render their contents. Sometimes the compositor would crash or become unstable too (kwin, niri, etc.). I had to reboot every few hours. Complete system instability. This problem only happened with Wayland, when I switched to X windows these problems went away. Since then I switched to an AMD RX 480 (8 GB) GPU and never had an issue here with Wayland, if GPU memory gets full, system memory gets seamlessly used instead. It felt like with the NVIDIA card, it never allocated back to system memory if no VRAM was available. This was only a problem on Linux since that same NVIDIA card was fine in Windows for 10+ years with zero issues related to instability around VRAM allocation. Tons of people reported the same issue on Linux on NVIDIA's forums for years. I wrote a lot more details and recorded demo videos around 9 months ago here https://nickjanetakis.com/blog/gpu-memory-allocation-bugs-with-nvidia-on-linux-and-wayland-adventures https://nickjanetakis.com/blog/gpu-memory-allocation-bugs-wi..., but since then a lot of these things have been resolved from switching over to AMD.
- OsrsNeedsf2P 2mo agoI gave myself 32GB of swap since SSDs are quite fast and the pages sent to disk seem to intelligently be rarely referenced ones. Feels amazing to run 4 VMs and 12 Claude Code instances on my mid tier hardware
- nickjj 2mo agoSwap doesn't help in this case. It's GPU drivers freaking out when they run out of VRAM, but it might only be specific to NVIDIA cards since AMD seems to handle this better. But, if it's driver specific that means it could be fixed at the kernel level I suppose, especially since Windows handles it seamlessly?
- pbhjpbhj 2mo agoCan you explain a little more please. I thought that graphics processing would revert to CPU if the GPU is maxed out and so swap might ultimately help? Is there no general GPU-swap mechanism? If CPU threads are competing for GPU resource does the GPU getting paged-out? I suppose this could be an historic thing, GPUs can have a type of DMA now, and disk access is much faster too, so paging perhaps makes sense when it didn't before?
- Prunkton 2mo agoSuch great work! Happens quiet regularly I hit a game just realizing seconds later I haven't shut down my local LLM yet. At least punishment will be less harsh. Lets see, maybe some games in window mode will just work fine.
- HelloUsername 2mo agoAlso Apple M3 support
- bansiwebix 2mo ago[dead]
- deleted 2mo ago[deleted]
- hnc3yfnu6f 2mo agoWish more people understood this
- j16sdiz 2mo agoIt's VRAM, not vRAM.
- fluffybucktsnek 2mo agoOut of curiosity, may I ask why?
- pezezin 2mo agoI don't know, I guess some rule about the capitalization of acronyms. But I have seen it written as VRAM since the early '90s when it meant dual-ported video RAM. I think this is the first time I see it written as vRAM with a lowercase v.
- burnte 2mo agoGenerally in our IT jargon writing "vRAM" would mean some sort of virtualized RAM thing, while VRAM shows the V is part of the initialism and thus VIDEO RANDOM ACCESS MEMORY. Nothing formal, just informal usage.
- pranav_tech26 2mo ago[flagged]
- dormento 2mo agoA curious thing happens in my Ubuntu 24 system, maybe that's just the normal OoM killer algo but no matter what process is hogging, Firefox is killed. Chrome eating too much memory? Firefox dies. Too many apps open? Firefox dies. Its funny.
- zahlman 2mo ago... Why would you be running Chrome and Firefox simultaneously?
- OkayPhysicist 2mo agoI occasionally have several browsers open when doing web-dev work, to make sure everything looks right on all browsers. There was also a period of time where I got into the habit of opening netflix in Chrome, despite being a Firefox ride-or-die, because Firefox didn't support the DRM that Netflix required to stream higher resolution video, but either that changed or I got sick of the juggling.
- Beijinger 2mo agoYou know what I would like to have? An application that sends a kill / kill -9 signal to a specific app, example Chromes, when things go south.
- bluGill 2mo agoLast I checked the OOM killer was configurable. However the process of configuring it was so convoluted I doubt anyone has successfully done it.
- augusto-moura 2mo agoFor OOM, there's earlyoom [1] and systemd-oomd [2]. They usually work out of the box [1]: https://github.com/rfjakob/earlyoom https://github.com/rfjakob/earlyoom [2]: https://man.archlinux.org/man/systemd-oomd.8 https://man.archlinux.org/man/systemd-oomd.8
- augusto-moura 2mo agoLike xkill? [1] Just be careful where you click [1]: https://en.wikipedia.org/wiki/Xkill https://en.wikipedia.org/wiki/Xkill
- dessimus 2mo agoHow is that sufficiently different from the Halting Problem?
- zahlman 2mo agoThe solution doesn't have to be perfect.
- zamadatix 2mo agoThe halting problem asks you to conclusively answer if a arbitrary program will stop running in the future. This asks to stop running a given program when stats seem bad currently. I don't really see what the relation is supposed to be.
- inventor7777 2mo agoI see all the comments about how Windows and Linux handle OOM, but I suppose I'll also mention macOS. On my M4 Max Mac Studio if I try to load too big of an AI model with protections off, the desktop starts glitching back and forth between the past few hundred frames. It looks bad when it happens but CtrlC still works to kill llama.cpp and if you were using LM Studio, SSH also works. Once you kill the offending process, or if macOS does it for you, the desktop comes back immediately.
- LoganDark 2mo agoOn my M4 Max MacBook Pro, if I run or even load an LLM more than a couple times, even a small one, general desktop performance starts to get really, really bad until a reboot and I don't know why. Genuinely, everything starts feeling super sluggish, even long after the process has terminated -- cmd+tab gets slow, apps constantly hang for a few seconds at a time, the cursor position starts lagging and videos are slow to pause/play -- and the only way I've found to fix it is to reboot the entire machine. Clearing memory, restarting apps etc. doesn't work. Don't get me wrong, the performance and QoS system works really well while the model is still in memory, but after a few cycles of loading and unloading, performance just ends up getting bad in general. Doesn't matter if I'm loading it with llama.cpp, MLX, native Metal, if I start using the GPU for machine learning the machine just gets really upset. This sucks man... macOS is not supposed to need reboots like that, my Intel Mac could run for months on end without slowing down at all and I could push it as hard as I wanted. As far as I can tell, this behavior is new as of either macOS 26 or the 27 beta. Apple really has been betraying their true Unix roots lately.
- radlad 2mo agoOh wow, you might've just explained some symptoms I've run into a few times. Thanks for your comment.
- inventor7777 2mo agoWow! That is wild. I am on 15.7.9 and I have had uptimes in the high 50s with weekly local model usage, I even tried GLM5.2 streaming from SSD and I have not noticed any slowdowns after at all. I guess you are right about 26 and 27. Dang, the more I hear about 26 I am just so glad that I did not update. What version of macOS are you on? 26 or 27? Have you considered doing a custom backup and restoring manually to 15.7?
- fencer12 2mo agoMore Linux goodies
- bsimpson 2mo agoI have a Z1 Extreme that has 16GB shared between the CPU and GPU (what AMD calls the APU). I've noticed that mangohud, the SteamOS utility that shows performance diagnostics, often reports RAM + VRAM numbers that exceed 16GB. I've been curious as to why. I presume it has to do with compression or something.
- burnte 2mo agoI would assume virtual memory.
- SquareWheel 2mo agoThis seems like an impressive improvement, and I'm looking forward to it eventually being upstreamed. Though unfortunately I'm on Nvidia right now, and have been struggling with vram. They don't seem to support any kind of paging at all. I am curious about the bit on virtual memory fragmentation. Would it make sense for the kernel to occasionally defragment that memory in place? I assume that would create a noticeable hitch, but might improve performance otherwise and allow for some applications to just fit in.
- archon810 2mo agoThe Steam Machine should be able to benefit greatly from this, right?
- chiddekel2 2mo ago[dead]