4 ms·

So I'm a bit out my field here but I've been wondering... Is there something that prevents AI software from augmenting VRAM with system RAM? My impression was

So I'm a bit out my field here but I've been wondering... Is there something that prevents AI software from augmenting VRAM with system RAM? My impression was that for instance OpenGL doesn't even have a standard way to get VRAM usage because you're not supposed to care, stuff is just shuffled around in the background for you if VRAM runs out.

Astronaut3315 3y ago
My SDXL workflow was often spilling over my RTX 3080’s 10GB of VRAM into system RAM. I’ve since tweaked my workflow to prevent that, as the slowdowns are dramatic. Tiled VAE encode/decode was a big help.
zirgs 3y ago
The latest nvidia drivers do this automatically, but it's very slow.
osmarks 3y ago
It's not very useful because the point of VRAM is that it's very very high-bandwidth. CPU RAM is slow and the PCIe link to the GPU is even slower.
- dale_glass 3y ago
  What's the usage pattern like? Is all of the VRAM used extensively in AI workloads, or one could hope to augment things a bit with system RAM with little performance impact?
  - Zetobal 3y ago
    The performance impact is not little it's huge.
  - filterfiber 3y ago
    The moment you can't fit the model in VRAM and have to use PCIE the performance impact is MASSIVE. Depending on the type of model/usecase/how much you can fit in VRAM it very well may still be usable, however compared to fitting everything in vram it'll be abysmal for most any model type. From my understanding the entire model must be read per iteration (or very close to all of it. This is why offloading any part of it impacts the performance so much. Diffusion models depending on the sampler and some other stuff can potentially get away with only a few steps (usually 2-20 best case) and are relatively small (3B or less). However LLM's are generally much bigger (they mostly start at 7B, there's a few exceptions depending on the use case), and they need an entire iteration per token which is only a few characters (think 2 to 3). So llm's especially are far more useful if you can keep them in vram.
    - dale_glass 3y ago
      Thanks, I messed around with this stuff a little bit, but have little understanding of what goes on in the guts of it.
dragonwriter 3y ago
> Is there something that prevents AI software from augmenting VRAM with system RAM? As I understand, it adds complication, the points where it is practical without just getting into an horrible churn of shuffling are limited based on model architecture, and (as a consequence) implementations may not always been transferrable between models outside of a narrow family. On the other hand, there are some frameworks with general support which eases this. On the third hand, shuffling stuff back and forth between system RAM and VRAM has a performance cost in the best case. So, yes, and no, and yes. Most (that I've seen) consumer-focused frontends for local models (text generation, image generation, audio generation) do it, to a greater or lesser extent. The base model implementations that are getting thrown out by researchers don't usually focus on that, and its generally implemented by third parties after the fact, sometimes with commonalities that can be applied to other models that are closely-enough related.

Astronaut3315 3y ago

My SDXL workflow was often spilling over my RTX 3080’s 10GB of VRAM into system RAM. I’ve since tweaked my workflow to prevent that, as the slowdowns are dramatic. Tiled VAE encode/decode was a big help.

zirgs 3y ago

The latest nvidia drivers do this automatically, but it's very slow.

osmarks 3y ago

It's not very useful because the point of VRAM is that it's very very high-bandwidth. CPU RAM is slow and the PCIe link to the GPU is even slower.

dale_glass 3y ago
What's the usage pattern like? Is all of the VRAM used extensively in AI workloads, or one could hope to augment things a bit with system RAM with little performance impact?
- Zetobal 3y ago
  The performance impact is not little it's huge.
- filterfiber 3y ago
  The moment you can't fit the model in VRAM and have to use PCIE the performance impact is MASSIVE. Depending on the type of model/usecase/how much you can fit in VRAM it very well may still be usable, however compared to fitting everything in vram it'll be abysmal for most any model type. From my understanding the entire model must be read per iteration (or very close to all of it. This is why offloading any part of it impacts the performance so much. Diffusion models depending on the sampler and some other stuff can potentially get away with only a few steps (usually 2-20 best case) and are relatively small (3B or less). However LLM's are generally much bigger (they mostly start at 7B, there's a few exceptions depending on the use case), and they need an entire iteration per token which is only a few characters (think 2 to 3). So llm's especially are far more useful if you can keep them in vram.
  - dale_glass 3y ago
    Thanks, I messed around with this stuff a little bit, but have little understanding of what goes on in the guts of it.

dragonwriter 3y ago

> Is there something that prevents AI software from augmenting VRAM with system RAM? As I understand, it adds complication, the points where it is practical without just getting into an horrible churn of shuffling are limited based on model architecture, and (as a consequence) implementations may not always been transferrable between models outside of a narrow family. On the other hand, there are some frameworks with general support which eases this. On the third hand, shuffling stuff back and forth between system RAM and VRAM has a performance cost in the best case. So, yes, and no, and yes. Most (that I've seen) consumer-focused frontends for local models (text generation, image generation, audio generation) do it, to a greater or lesser extent. The base model implementations that are getting thrown out by researchers don't usually focus on that, and its generally implemented by third parties after the fact, sometimes with commonalities that can be applied to other models that are closely-enough related.