7 ms·
Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe
- tandr 7mo agoSome simpler benchmark table would be great. May I suggest Ollama on base machine, Ollama with T1, Ollama with T1+T2 etc. on midsize and big models to compare token/sec?
- pabs3 7mo agoWould be great to get this into mainline Linux.
- yjftsjthsd-h 7mo agoPreviously: https://news.ycombinator.com/item?id=47384557 https://news.ycombinator.com/item?id=47384557 (Still cool, still would benefit from better benchmarks)
- holoduke 7mo agoThe is extremely slow and not useful in my opinion.
- majorchord 7mo agoI would say it depends entirely on your usecase. I don't think there can be a simple "not useful" generalization that applies to everyone.
- jauntywundrkind 7mo agoMan I wish that was a canned response that could be deployed on demand! Well said. I really appreciate thriftful & resourceful points of view. Exploring what if, looking for use is such a great virtue.
- bigwheels 7mo agoCan you elaborate beyond the shallow/superficial dismissal?
- whywhywhywhy 7mo agoIf it takes seconds in VRAM it can take tens of minutes running the same thing offloaded to RAM if it hasn't been designed to do it.
- daneel_w 7mo agoIt makes the difference between being able to run a lot of machine learning tasks, and not being able at all. Pretty useful.
- sayYayToLife 7mo ago[dead]
- ozgrakkurt 7mo agoIt is about as useful as rtx
- bhewes 7mo agoThis has been fun we can task our nemotron-3-super model to run over night when our desktops are idle. 4070s and 96gb of ram works fine. Slow but does it's job.
- daneel_w 7mo agoRelated, a couple of years ago: https://old.reddit.com/r/Amd/comments/15t0lsm/i_turned_a_95_amd_apu_into_a_16gb_vram_gpu_and_it/ https://old.reddit.com/r/Amd/comments/15t0lsm/i_turned_a_95_... "I turned a $95 AMD APU into a 16GB VRAM GPU and it can run stable diffusion!"
- 3abiton 7mo ago> it can generate a 50 steps 512x512 image around 1 minute and 50 seconds. I have the 4650G APU, and the best way to describe it is: lacking of support. This was even more true 3 yo than now. rocm (is) was absolutely dogshit then, I know this because I tried to do the same when that post was made. You have to compile everything from scratch, get the relevant patches, and even then, xformers which is a library that accelerate diffusion model inferencing was not supported for renoir or rocm back then. Yes, you can generate an image, but it was much slower, and rigged with bugs. You couldn'rt update rocm because it broke compatibility, and it was partly the reason I got into nixos. That being said, those APUs are a power house. Nowadays I can run decent agentic workflows on them (I have 64gb of ddr4 ram, ie APU can suck as much as it needs with the latest linux kernels). Just note, diffusion models are still second class citizens on AMD apus even GPUs. But then again, nothing close right now on the market except for what apple offers.
- nl 7mo agoThe Ryzen AI CPU/GPUs (Ryzan AI 395+ etc) seem to have increasing support - https://lemonade-server.ai/ https://lemonade-server.ai/ now has support for the NPU as well as the combined CPU/GPU (which I guess is a APU but is different to the G series of APUs I think?) But I'm always interested in first hand experiences of how good is it really - I'm pretty cynical about the idea that AMD actually knows what it takes to build good software end-to-end.
- 3abiton 7mo agoI also have one, and indeed support is very much frictionless now compared to a year ago. But again, not thanks to AMD, as initially it was purely community driven. Strix halo was not even supported by ROCm (officially), and we had to deal with therock images, then donato made the toolbox, and then lemonade came through. I am really surprised how AMD approached this. They made big promises, they threw the hardware out, it really is amazing piece of hardware given what you can do with it, but it was left hanging without support for AI stack for months even though it had it in its name. Contrast that with the DGX spark (yes it had and still had bugs in its kernels, but cuda worked on day 1) and you can see the difference. Nvidia is selling an ecosystem, AMD is selling hardware. I really hope AMD focus on the software layer more.
- paultendo 7mo agoCould be a very useful way to do some overnight tasks using spare RAM. Possibly things like LLM-based categorisation, labelling, data cleansing. That's what comes to mind for me anyway.
- MaxikCZ 7mo agoNeat part is every task becomes overnight task when you start offloading to RAM.
- yjtpesesu2 7mo agoHow does this differ from anything llama.cpp offers, regarding offloading layers? The repo consistently refers to "DDR4". Is there a reason DDR5 won't work with this?
- kcb 7mo agoCUDA has had managed memory that pages between VRAM and system RAM for a decade. Problem is doing so is unusably slow for AI purposes. Seems like an unnecessary layer here.
- yjtpesesu2 7mo ago[dead]
- hrmtst93837 7mo ago[flagged]
- svnt 7mo agoThe readme opens with this: > I have an RTX 5070 with 12 GB VRAM and I wanted to run glm-4.7-flash:q8_0, which is a 31.8 GB model. The standard options are: > Offload layers to CPU — works, but drops token/s by 5–10× because CPU RAM has no CUDA coherence. You end up waiting. Use a smaller quantization — you lose quality. At q4_0 the model is noticeably worse on reasoning tasks. > Buy a bigger GPU — not realistic for consumer hardware. A 48 GB card costs more than a complete workstation. > None of those felt right, so I built an alternative: route the overflow memory to DDR4 via DMA-BUF, which gives the GPU direct access to system RAM over PCIe 4.0 without a CPU copy involved. And then limps home with this caveat on the closest thing to a benchmark: > The PCIe 4.0 link (~32 GB/s) is the bottleneck when the model overflows VRAM. The best strategy is to shrink the model until it fits — either with EXL3 quantization or ModelOpt PTQ — and use GreenBoost's DDR4 pool for KV cache only. I think the reason it refers it to DDR4 is because that is how the user explained it to their coding agent. LLMs are great at perpetuating unnecessary specificity.
- moffkalast 7mo ago
- 0xbadcafebee 7mo agoYou can already do this with some GPU drivers: GRUB_CMDLINE_LINUX_DEFAULT="quiet splash amdttm.pages_limit=5242880 ttm.pages_limit=5242880" One downside is your kernel isn't going to reserve that memory away from userland. You will still see all the memory at system level as "free". As the GPU driver starts using it, other apps/the OS will try to use the "free" memory, not knowing how much of it is in use (it may show up as "cache", or not at all). Then OOM killer starts going or programs start crashing, and at some point the OS tips over or GPU driver crashes. You can add loads of swap as a compromise and it works okay, if a bit slow. In any case, loading a gigantic model just to use system RAM is absurdly slow (due to mem bandwidth), like 1-5 t/s, so it's not practical. It'd take a whole day to process one 86k token request. Just pay a cloud provider $0.01 to do it in 10 seconds.
- jmward01 7mo agoThe point is not how fast it is now. The point is that this opens new possibilities that can be built on. Potentially models that are trained with slightly different architectures to optimize to this use case. Possibly others come to improve this path. Possibly HW manufacturers make a few small adjustments that remove bottlenecks. Who knows, the next person may combine CPU compute with this mem sharing to get another token a second. Then the next person does predictive loading into memory to keep that bandwith 100% maxed and usable. Then the next does and the next does. Before you know it there is a real thing there that never existed. This is a great project. I love the possibilities it hints at. Thanks for building it!
- smallnamespace 7mo agoIt’s architecturally not a good approach. System RAM is much slower so you should put data that doesn’t need to be used often on it. That knowledge is at the application layer. Adding a CUDA shim makes system RAM appear like VRAM, which gets things to run, but it will never run very well. The benchmarks at the bottom mention memory tiering and manually controlling where things go, but if your application already does that, then you probably don’t also need a CUDA shim. The application should control the VRAM to system memory transfers with boring normal code.
- sabareesh 7mo agoI wish it provided benchmark comparing Direct RAM offload vs CPU offload vs Full VRAM
- aplomb1026 7mo ago[dead]
- Havoc 7mo ago> The best strategy is to shrink the model until it fits — either with EXL3 quantization or ModelOpt PTQ — and use GreenBoost's DDR4 pool for KV cache only. Does this make sense? I'd have thought the KV is guaranteed to be used 100% of the time while say in a MoE the same can't be said of the weights. Though I suppose if you're shooting for huge context then having that allocation go into ram makes sense specially when its allocated but not used yet
- alexeldeib 7mo agoKV cache is, well, a cache that can fill up and trigger eviction. You require enough space to execute at least 1 fwd pass of 1 request at your context length. KV cache hits reduce TTFT by avoiding prefill. You don’t get to skip decode. MoE is kinda related in terms of lower usage requirements vs a dense model of same total param size, but I think your mental model is a bit off.
- zozbot234 7mo agoKV cache is also eminently swappable if you have fast storage, since it mostly sees small append-only writes per token - it's not rewritten continuously like the activations. (I believe it's even better if you use cached input tokens across requests, since that portion of KV cache can then be recycled and save a single ~KV-cache sized write per request.) Accessing swapped-out cache may be slow, but it's highly preferable to not having that cache amount at all and recomputing from scratch.
- ma2kx 7mo agoThe physical bottleneck to system memory remains. Therefore, I assume that better results are achieved by manually adjusting which layers are offloaded. I would prefer to use system memory to cache different models, focusing on things like embedding, rerankers, and TTS. This is sufficient to run a more complex RAG locally, for example, via Mem0, and then use a larger LLM via the cloud.
- ajaimk 7mo ago[dead]
- nl 7mo agoThis is really interesting engineering, but I agree with the other commentators that the benchmarking makes it hard to understand the contribution various factors are having. The ExLlamaV3 EXL3 2bpw (8 GB, full VRAM) row is an order of magnitude faster than the baseline - but the baseline seems to be the 32GB model running with the KV cache shared to system memory only (I think?) But if a 8GB model gives sufficient quality then it seems like that would have worked without the shared memory thing? I think the useful apples-to-apples benchmark is currently the Ollama + GreenBoost shim (baseline) (2-5 tps) vs ExLlamaV3 + GreenBoost cache (8–20 tps) comparison. It would be really useful to see this compared with the existing llama CPU/memory offload. There is a note at the start ("Offload layers to CPU — works, but drops token/s by 5–10× because CPU RAM has no CUDA coherence") - but it is unclear if that 5-10x token speed drop is compared to running a model completely in GPU or compared to the greenboost approach. I think it is vs GPU, in which case it seems likely the performance is similar to what greenboost is giving but probably much more stable.
- kristianp 7mo agoExLlamaV3 EXL3 2bpw is likely the 30b parameter GLM 4.7 Flash quantised down to 2 bits, the unstated assumption is that you need to check the 2bpw quantisation works well enough for your use case. The reported size of the ModelOpt FP8, 16 GB, sounds wrong to me. If its 8 bits per parameter it is going to be a similar size to the glm-4.7-flash:q8_0. They repeat this a few times in the readme.
- Insanity 7mo agoExtend your VRAM using RAM, then extend your RAM using Swap.
- SV_BubbleTime 7mo agoIf you are doing video models, this is an excellent way to murder your SSD. Do not put swap on an SSD you care about at all.
- Insanity 7mo agoI was writing it somewhat tongue-in-cheek and not as a serious suggestion. But thanks for adding the disclaimer, that's good advice!
- rvz 7mo ago> Do not put swap on an SSD you care about at all. This. Many people rediscovering what the purpose of swap files are, but will still find a way to abuse it without knowing that they are actually destroying their SSD.
- zozbot234 7mo agoYou can of course monitor SMART wearout indicators to check whether this is happening. Casual use of swap for non LLM-use is actually fine since "cold" ephemeral data will be swapped out first and that will never get written to; KV cache is mostly fine since it's similarly append-only so writes are tolerably small; but yes, more general LLM inference totally breaks that limited-writes pattern and will wear out/kill your media.
- duskdozer 7mo agozram swap otoh should be relatively 'free'
- zozbot234 7mo agoLLM working memory is not compressible so ZRAM doesn't buy you anything.
- felipe_aramburu 7mo agoHow does this relate to cuCascade https://github.com/nvidia/cucascade https://github.com/nvidia/cucascade
- armada651 7mo agoDoesn't Windows already do this by default? I can already run models bigger than my GPU VRAM and it will start using up to 50% of my system RAM as "shared memory". This is on a Desktop PC without a shared memory architecture.
- 3836293648 7mo agoI don't think Windows does this, but Ollama does
- nodja 7mo agoNVIDIA's GPU drivers on windows 100% do this https://i.imgur.com/c0a3vUy.png https://i.imgur.com/c0a3vUy.png
- whywhywhywhy 7mo agoIt's the drivers but it was a relatively recent addition, think it was added when either the 30xx or 40xx series shipped and the lower cards had pitiful VRAM so they enabled it by default so they'd work with all games. Most people who know it does this turns it off because it kicks in too early so if you have 24GB it'll offload to RAM and tank your inference speed when you hit around 22GB use. https://nvidia.custhelp.com/app/answers/detail/a_id/5490/~/system-memory-fallback-for-stable-diffusion https://nvidia.custhelp.com/app/answers/detail/a_id/5490/~/s...
- lastdong 7mo agoNicely linked!
- dahart 7mo agoThe Nvidia driver has used system memory fallback for a couple of years now. https://nvidia.custhelp.com/app/answers/detail/a_id/5490/~/system-memory-fallback-for-stable-diffusion https://nvidia.custhelp.com/app/answers/detail/a_id/5490/~/s...
- Yokohiii 7mo ago
- NooneAtAll3 7mo agonvidia failed to provide gpu with actually meaningful amount of vram and instead of improving the actual product, it decided to "solve the problem in software" I expect this greenboost to fall and burn, honestly...
- deleted 7mo ago[deleted]
- brador 7mo agoCould this work on steam deck?
- 152334H 7mo agoNobody mentioning how this project is vibecoded slop? > The code is really bad with completely uneeded parts. The LLM (Qwen 2.5 7B) has hardcoded the i9 14700KF topology, and has variables related to it never used... It's even funnier that the show hardware function always prints the same string. There are even random pip log files. Why did this slop got coverage here? https://www.phoronix.com/forums/forum/linux-graphics-x-org-drivers/open-source-nvidia-linux-nouveau/1619572-open-source-greenboost-driver-aims-to-augment-nvidia-gpus-vram-with-system-ram-nvme-to-handle-larger-llms?p=1619635#post1619635 https://www.phoronix.com/forums/forum/linux-graphics-x-org-d...
- paseante 7mo ago[dead]
- Berazu 7mo agoI wish there was a way to extend RAM/NVMe with GPU VRAM. :(
- nuopnu 7mo agoThere are vram disks, so at least you can use it for the swap.
- bandrami 7mo agoQu'ils mangent de la brioche
- dwroberts 7mo agoThe title here needs changing, this is for nvidia cards but it is not an official project and has nothing to do with them (Feels especially deceptive when there is another top story right with the headline “nvidia nemoclaw” which is an official project)
- wewewedxfgdf 7mo agoWhy don't they just put ram slots on the card so you can augment the fast ram
- VHRanger 7mo agoGDDR7x doesn't come in dimm factor? In general soldered ram seems to get much higher bandwidth than removeable ram. See ryzen AI Max vs 9950x max ram throughputfor example
- wewewedxfgdf 7mo agoWe are talking here about slower ram to augment.
- nic547 7mo agoStrix Halo uses a 256bit memory interface, the normal desktop processors only have a 128bit interface, that's the biggest difference in bandwidth. For more bandwidth you need to go to a Threadripper. Strix Halo seems to use LPDDR with 8000 MT/s, which is a bit faster than the usual 5600 MT/s-6400 MT/s "normal" DDR5-DIMMs (Albeit (expensive) faster ones seem to exist), so there's a slight edge towards soldered memory (not sure about LPCAMM2 and similar tech). GDDR7 is a different league, a 5070 Ti also has a 256bit memory interface, but has 896GB/s bandwidth, compared to strix halo with 256GB/s
- VHRanger 7mo agoIt's really hard to push DDR5 past 6000MT/s on 4+ DIMMs it seems. I had to get everything top spec to fit 4 channels of 6000MT/s on my 9950x (asus proArt motherboard and the top tier trident neo RAM sticks) -- otherwise it's reportedly unstable.
- nic547 7mo ago9950X is dual channel, running 4 DIMMs runs them interleaved, with two DIMMs sharing one physical connection, impacting signal integrity severely. AFAIK this has gotten worse with DDR5 to the point that it's generally recommended to avoid 4 DIMMs unless you really can't get enough RAM otherwise. For maximum bandwidth you need to avoid interleaving. Strix Halo simply has more memory controllers. Threadrippers are also quad channel, and should be able to run 4 DIMMs at rated speeds, but the cheapest Zen 5 Threadripper seems to be almost double the price of a 9950X3D.
- deleted 7mo ago[deleted]
- ninjagoo 7mo agoThis is awesome! Normally, offloading layers to the CPU RAM means that the compute for those layers occurs on the CPU instead of the GPU, generally speaking. The CPU is orders of magnitude slower than the GPU. With this approach the compute occurs on the GPU, with the tradeoff that layers in RAM have to be moved back-and-forth through PCI-DMA. It seems to me that this should offer a speedup vs compute split between GPU and CPU. The amount of speedup will depend on how many layers would have been on CPU compute, minus the reduction due to moving those layers between RAM and the GPU. What's slower? Compute on the CPU or moving data from RAM to GPU through PCI-DMA?
- angry_octet 7mo agoI have a system with an ungodly amount of Optane memory and I'm hoping this will work.
- Rafuino 7mo agoWhat do you have? I've got a 905P and a 900P and am already using these in LM Studio by putting all models there and extending system memory with more scratch space... Not sure if I need to do anything differently with this since LM Studio already enabled it I think
- angry_octet 7mo agoThe DIMM version, PMEM 200 modules. Appears as /dev/dax and you can mmap it. Could also use as a virtual filesystem, or using RAM as a cache for pmem, but sceptical of that. Is the virtual memory system is a good way to expand memory for inference, via having it directly manage a non uniform pool? I haven't seen any research on that.
- aruametello 7mo agoPost traumatic "nvidia TurboCache" disorder triggered. https://en.wikipedia.org/wiki/TurboCache https://en.wikipedia.org/wiki/TurboCache (Not the same thing 1:1, but worth the joke anyway)
- dr_kretyn 7mo agoIs there a similar initiative for AMD?
- bguberfain 7mo ago"A watchdog kernel thread monitors RAM and NVMe pressure and signals userspace before things get dangerous." - which kind of danger this type of solution can have?
- Heer_J 7mo ago[dead]
- yalogin 7mo agoIs there a use case for this today? Feels more like nvidia is priming the software hoping system designers will find ways to use it.
- rnrn 7mo agoWhy is there a new kernel driver here at all? It appears that all it does it allocate system RAM (“DDR4”) and export it as a dmabuf for import to cuda as mapped external memory. Then a userspace shim hijacks APIs to use that if gpu memory is full. cuda already supports allocating mapped system memory, so AFAICT this could be implemented in the userspace shim with no new kernel driver. Also as other commenters have mentioned, redirecting allocations to managed memory would also enable similar oversubscription And the hijack approach only makes sense for making apps have this behavior with no changes, and could be done with minor app changes (e.g. PyTorch has a pluggable allocator interface). App changes also enable intentionally placing specific allocations. My impression is that this is vibe from beginning to end, starting from a design that only makes sense if you are hallucinating
- Melatonic 7mo agoMaybe theres a significant latency advantage to doing it this way? Or, as you said, making everything backwards compatible that is not being regularly updated
- deleted 7mo ago[deleted]