5 ms·

You can already do this with some GPU drivers: GRUB_CMDLINE_LINUX_DEFAULT="quiet splash amdttm.pages_limit=5242880 ttm.pages_limit=5242880" One downside is

You can already do this with some GPU drivers: GRUB_CMDLINE_LINUX_DEFAULT="quiet splash amdttm.pages_limit=5242880 ttm.pages_limit=5242880" One downside is your kernel isn't going to reserve that memory away from userland. You will still see all the memory at system level as "free". As the GPU driver starts using it, other apps/the OS will try to use the "free" memory, not knowing how much of it is in use (it may show up as "cache", or not at all). Then OOM killer starts going or programs start crashing, and at some point the OS tips over or GPU driver crashes. You can add loads of swap as a compromise and it works okay, if a bit slow. In any case, loading a gigantic model just to use system RAM is absurdly slow (due to mem bandwidth), like 1-5 t/s, so it's not practical. It'd take a whole day to process one 86k token request. Just pay a cloud provider $0.01 to do it in 10 seconds.

jmward01 8mo ago
The point is not how fast it is now. The point is that this opens new possibilities that can be built on. Potentially models that are trained with slightly different architectures to optimize to this use case. Possibly others come to improve this path. Possibly HW manufacturers make a few small adjustments that remove bottlenecks. Who knows, the next person may combine CPU compute with this mem sharing to get another token a second. Then the next person does predictive loading into memory to keep that bandwith 100% maxed and usable. Then the next does and the next does. Before you know it there is a real thing there that never existed. This is a great project. I love the possibilities it hints at. Thanks for building it!
- smallnamespace 8mo ago
  It’s architecturally not a good approach. System RAM is much slower so you should put data that doesn’t need to be used often on it. That knowledge is at the application layer. Adding a CUDA shim makes system RAM appear like VRAM, which gets things to run, but it will never run very well. The benchmarks at the bottom mention memory tiering and manually controlling where things go, but if your application already does that, then you probably don’t also need a CUDA shim. The application should control the VRAM to system memory transfers with boring normal code.
  - timnetworks 8mo ago
    Some people are not concerned with having it run the fastest, just having it run at all may be enough.
    - nl 8mo ago
      But then you can use CPU/RAM offload, which already allows you to offload without a kernel module.
    - m-schuetz 8mo ago
      From my experience, accessing system RAM from the GPU is so slow, it might as well count as "does not work". It's orders of magnitudes faster to memcpy large swaths of memory that you are going to use to the GPU, rather than accessing system mem from a kernel which then takes ages to wait for that small block/page of memory, then waits again for the next small page/block of memory, etc. Latency hiding doesnt work anymore if the latency is that large.
      - dahart 8mo ago
        You’re right for some workloads, but not all of them. The same could have been said for disk swap since the beginning though, and people still found it valuable. Disk swapping with spinning drives did used to be multiple orders of magnitude slower than RAM. But it prevented applications or the system from crashing. Using system memory from the GPU isn’t that bad if your compute is high enough and you don’t transfer that much data. There are commercial applications that support it and only see low 2-digit percentage perf impact and not the multiples you might expect. Plus on Windows on Nvidia hardware, the driver will automatically use system memory if you oversubscribe VRAM, and I believe this was introduced to support running Stable Diffusion on smaller GPUs.
  - midnitewarrior 8mo ago
    [dead]
  - jbverschoor 8mo ago
    Not true for unified systems. And for strix halo you need to dedicate the amount which is annoying. You’re basically stating that swapping is also a bad idea. And to take it further, any memory or storage is a bad idea because there’s L1 cache/SRAM which is faster then the rest
lelanthran 8mo ago
> any case, loading a gigantic model just to use system RAM is absurdly slow (due to mem bandwidth), like 1-5 t/s, so it's not practical. It'd take a whole day to process one 86k token reques So don't use it for large requests. Ideal for when you just want to categorise things, for example, "does this task need a shell" or "bucket this email into one of help request, bill due or personal comms".
- zozbot234 8mo ago
  The best use is actually for a layer that "almost fits" into VRAM, such that automated offloading to system RAM will be rare enough that it doesn't impact performance.
  - usrusr 8mo ago
    As in when your secondary memory is fast enough, after the first 10% of the model are processed you can swap their memory with the part for 50% to 60% and when that is done you swap back to have the 0-10% ready in time for the next iteration? Sounds ambitious, for the small improvement in effective capacity. In particular when I start wondering if real life speed differences would be small enough for that 10% increase, or if it would be even smaller. And that's before factoring in power/cooling cost for saturating another interface.
RobotToaster 8mo ago
Would MoE models work better with this approach?
robotswantdata 8mo ago
12 channel ddr5 5600 ECC is around 500gbs which in real world works very well for large MoE
- adrian_b 8mo ago
  You mean 500 GB/s, not Gb/s (actually 537 GB/s). Unfortunately that does not matter. Even in a cheap desktop motherboard the memory bandwidth is higher than of 16-lane PCIe 5.0. Therefore the memory bandwidth available to a discrete GPU is determined by its PCIe slot, not by the system memory. If you install multiple GPUs, in many MBs that will halve the bandwidth of the PCIe slots, for an even lower memory throughput.
  - zargon 8mo ago
    > in many MBs that will halve the bandwidth of the PCIe slots Not on boards that have 12 channels of DDR5. But yeah, squeezing an LLM from RAM through the PCIe bus is silly. I would expect it would be faster to just run a portion of the model on the CPU in llama.cop fashion.
    - Gracana 8mo ago
      It is much faster, yeah. llama.cpp supports swapping between system memory and GPU, but it’s recommended that you don’t use that feature because it’s rarely the right call vs using the CPU to do inference on the model parts in system CPU memory. Edit: the settings is "GGML_CUDA_ENABLE_UNIFIED_MEMORY=1"... useful if you have unified memory, very slow if you do not.
    - lostmsu 7mo ago
      llama.cpp afaik does not run portion of the model on the CPU. --cpu-moe just offloads weights to RAM, but they are still loaded to GPU for compute.
  - robotswantdata 8mo ago
    Talking about dual socket SP5 EPYC with 24 DIMM slots, 128 PCIe 5.0 lanes It’s fast for hybrid inference, if you get the KV and MoE layers tuned between the Blackwell card(s) and offloading. We have a prototype unit and it’s very fast with large MoEs
adrian_b 8mo ago
With discrete GPUs, using system RAM is slow not due to mem bandwidth, but due to PCIe bandwidth, which is the bottleneck. For example, 16x PCIe 4.0: 256 Gb/s, 16x PCIe 5.0: 512 Gb/s, while 2x DDR5-6400 DIMMs: 819 Gb/s. The actual throughput is lower for both PCIe and DDR5, due to communication overhead. On server/workstation motherboards which may have 4, 8 or 12 DIMMs instead of 2, the ratio between memory bandwidth and PCIe bandwidth becomes proportionally higher, so the memory throughput achievable by the GPU becomes a very small fraction of the system memory bandwidth.
- zozbot234 8mo ago
  > slow not due to mem bandwidth, but due to PCIe bandwidth, which is the bottleneck. > On server/workstation motherboards ... the memory throughput [to system RAM] achievable by the GPU becomes a very small fraction of the system memory bandwidth. Yes, this is a critical point. It means that this is only realistically useful for prefill, which is compute- and not memory-bandwidth bound.
  - shdudns 8mo ago
    Sorry, I'm a bit of a noob on llm. What is "prefill"? As opposed to what?
    - natechlin 8mo ago
      Prefill - module computes KV cache over input toks, up to the last token in your input (the 'prompt'), at which point it can then begin - Decode - the model chooses a new token to append to the end of the current token list (i.e. it generates a token), then computes the new tokens KVs. Decode is basically prefill 1 tok -> add 1 tok -> prefill 1 more tok -> .... but in the initial prefill stage it doesn't need to do generation, since you've provided the toks.
      - ghm2199 8mo ago
        And Incidentally prefill would also be how caching,say, a system prompt saves you some $ for API usage with LLM providers. They only compute the kv cache for the new tokens after the system prompt.

jmward01 8mo ago

The point is not how fast it is now. The point is that this opens new possibilities that can be built on. Potentially models that are trained with slightly different architectures to optimize to this use case. Possibly others come to improve this path. Possibly HW manufacturers make a few small adjustments that remove bottlenecks. Who knows, the next person may combine CPU compute with this mem sharing to get another token a second. Then the next person does predictive loading into memory to keep that bandwith 100% maxed and usable. Then the next does and the next does. Before you know it there is a real thing there that never existed. This is a great project. I love the possibilities it hints at. Thanks for building it!

smallnamespace 8mo ago
It’s architecturally not a good approach. System RAM is much slower so you should put data that doesn’t need to be used often on it. That knowledge is at the application layer. Adding a CUDA shim makes system RAM appear like VRAM, which gets things to run, but it will never run very well. The benchmarks at the bottom mention memory tiering and manually controlling where things go, but if your application already does that, then you probably don’t also need a CUDA shim. The application should control the VRAM to system memory transfers with boring normal code.
- timnetworks 8mo ago
  Some people are not concerned with having it run the fastest, just having it run at all may be enough.
  - nl 8mo ago
    But then you can use CPU/RAM offload, which already allows you to offload without a kernel module.
  - m-schuetz 8mo ago
    From my experience, accessing system RAM from the GPU is so slow, it might as well count as "does not work". It's orders of magnitudes faster to memcpy large swaths of memory that you are going to use to the GPU, rather than accessing system mem from a kernel which then takes ages to wait for that small block/page of memory, then waits again for the next small page/block of memory, etc. Latency hiding doesnt work anymore if the latency is that large.
    - dahart 8mo ago
      You’re right for some workloads, but not all of them. The same could have been said for disk swap since the beginning though, and people still found it valuable. Disk swapping with spinning drives did used to be multiple orders of magnitude slower than RAM. But it prevented applications or the system from crashing. Using system memory from the GPU isn’t that bad if your compute is high enough and you don’t transfer that much data. There are commercial applications that support it and only see low 2-digit percentage perf impact and not the multiples you might expect. Plus on Windows on Nvidia hardware, the driver will automatically use system memory if you oversubscribe VRAM, and I believe this was introduced to support running Stable Diffusion on smaller GPUs.
- midnitewarrior 8mo ago
  [dead]
- jbverschoor 8mo ago
  Not true for unified systems. And for strix halo you need to dedicate the amount which is annoying. You’re basically stating that swapping is also a bad idea. And to take it further, any memory or storage is a bad idea because there’s L1 cache/SRAM which is faster then the rest

lelanthran 8mo ago

> any case, loading a gigantic model just to use system RAM is absurdly slow (due to mem bandwidth), like 1-5 t/s, so it's not practical. It'd take a whole day to process one 86k token reques So don't use it for large requests. Ideal for when you just want to categorise things, for example, "does this task need a shell" or "bucket this email into one of help request, bill due or personal comms".

zozbot234 8mo ago
The best use is actually for a layer that "almost fits" into VRAM, such that automated offloading to system RAM will be rare enough that it doesn't impact performance.
- usrusr 8mo ago
  As in when your secondary memory is fast enough, after the first 10% of the model are processed you can swap their memory with the part for 50% to 60% and when that is done you swap back to have the 0-10% ready in time for the next iteration? Sounds ambitious, for the small improvement in effective capacity. In particular when I start wondering if real life speed differences would be small enough for that 10% increase, or if it would be even smaller. And that's before factoring in power/cooling cost for saturating another interface.

RobotToaster 8mo ago

Would MoE models work better with this approach?

robotswantdata 8mo ago

12 channel ddr5 5600 ECC is around 500gbs which in real world works very well for large MoE

adrian_b 8mo ago
You mean 500 GB/s, not Gb/s (actually 537 GB/s). Unfortunately that does not matter. Even in a cheap desktop motherboard the memory bandwidth is higher than of 16-lane PCIe 5.0. Therefore the memory bandwidth available to a discrete GPU is determined by its PCIe slot, not by the system memory. If you install multiple GPUs, in many MBs that will halve the bandwidth of the PCIe slots, for an even lower memory throughput.
- zargon 8mo ago
  > in many MBs that will halve the bandwidth of the PCIe slots Not on boards that have 12 channels of DDR5. But yeah, squeezing an LLM from RAM through the PCIe bus is silly. I would expect it would be faster to just run a portion of the model on the CPU in llama.cop fashion.
  - Gracana 8mo ago
    It is much faster, yeah. llama.cpp supports swapping between system memory and GPU, but it’s recommended that you don’t use that feature because it’s rarely the right call vs using the CPU to do inference on the model parts in system CPU memory. Edit: the settings is "GGML_CUDA_ENABLE_UNIFIED_MEMORY=1"... useful if you have unified memory, very slow if you do not.
  - lostmsu 7mo ago
    llama.cpp afaik does not run portion of the model on the CPU. --cpu-moe just offloads weights to RAM, but they are still loaded to GPU for compute.
- robotswantdata 8mo ago
  Talking about dual socket SP5 EPYC with 24 DIMM slots, 128 PCIe 5.0 lanes It’s fast for hybrid inference, if you get the KV and MoE layers tuned between the Blackwell card(s) and offloading. We have a prototype unit and it’s very fast with large MoEs

adrian_b 8mo ago

With discrete GPUs, using system RAM is slow not due to mem bandwidth, but due to PCIe bandwidth, which is the bottleneck. For example, 16x PCIe 4.0: 256 Gb/s, 16x PCIe 5.0: 512 Gb/s, while 2x DDR5-6400 DIMMs: 819 Gb/s. The actual throughput is lower for both PCIe and DDR5, due to communication overhead. On server/workstation motherboards which may have 4, 8 or 12 DIMMs instead of 2, the ratio between memory bandwidth and PCIe bandwidth becomes proportionally higher, so the memory throughput achievable by the GPU becomes a very small fraction of the system memory bandwidth.

zozbot234 8mo ago
> slow not due to mem bandwidth, but due to PCIe bandwidth, which is the bottleneck. > On server/workstation motherboards ... the memory throughput [to system RAM] achievable by the GPU becomes a very small fraction of the system memory bandwidth. Yes, this is a critical point. It means that this is only realistically useful for prefill, which is compute- and not memory-bandwidth bound.
- shdudns 8mo ago
  Sorry, I'm a bit of a noob on llm. What is "prefill"? As opposed to what?
  - natechlin 8mo ago
    Prefill - module computes KV cache over input toks, up to the last token in your input (the 'prompt'), at which point it can then begin - Decode - the model chooses a new token to append to the end of the current token list (i.e. it generates a token), then computes the new tokens KVs. Decode is basically prefill 1 tok -> add 1 tok -> prefill 1 more tok -> .... but in the initial prefill stage it doesn't need to do generation, since you've provided the toks.
    - ghm2199 8mo ago
      And Incidentally prefill would also be how caching,say, a system prompt saves you some $ for API usage with LLM providers. They only compute the kv cache for the new tokens after the system prompt.