3 ms·

I know relatively little about the workings of LLMs, but I keep seeing projects like this that run massive MoE models using very modest amounts of RAM, perhaps

by CyLith 2mo ago

I know relatively little about the workings of LLMs, but I keep seeing projects like this that run massive MoE models using very modest amounts of RAM, perhaps excessively so. I wonder, is there a way to make the RAM usage tunable? I have a Macbook with 32 GB of RAM, and it'd be great if I could run the same model but take advantage of the additional RAM to make it run faster.

ianmurrays 2mo ago
I guess you have to know which experts to keep “hot” in ram, which you can’t know beforehand, so there wouldn’t be much gain.
- spockz 2mo ago
  I do wonder if there are some experts that are more likely to be hit. So if the normal optimised setup runs in 12GiB an you have 4GiB extra to spare, you could say “promote the most used X experts to this stable (old gen in GC parlance) region and don’t swap it out. Maybe you could even do something like profiling and remember over multiple sessions (per project/workspace) what the most used agents are and load those up before hand.
zamadatix 2mo ago
That's about the turning point for just using typical quants for me. Larger still and you can just do the full model. Smaller to this degree and you need all sorts of extra tricks to get anything.
fodkodrasz 2mo ago
> I wonder, is there a way to make the RAM usage tunable? In LM Studio I can tune it by selecting different quantation of the model, by selecting how many layers of the neural net to be loaded to GPU (rest stays in main mem, evaluated by the CPU), and by adjusting context window.
leonickson 2mo ago
It's tunable, --cache-gb N on the CLI. In my sweep the speed barely moved between a 1GB and 6GB cache (43% vs 70% hit rate, same tok/s) because right now the bottleneck is GPU dispatch, not the SSD. so more RAM doesnt buy much yet. once the kernel work lands it should start to matter, so on 32GB I would just set 8 and let it age well. Also the hit rates themselves answer the "can you even know which experts stay hot" question, reuse across tokens is very real.

by CyLith 2mo ago

ianmurrays 2mo ago

I guess you have to know which experts to keep “hot” in ram, which you can’t know beforehand, so there wouldn’t be much gain.

spockz 2mo ago
I do wonder if there are some experts that are more likely to be hit. So if the normal optimised setup runs in 12GiB an you have 4GiB extra to spare, you could say “promote the most used X experts to this stable (old gen in GC parlance) region and don’t swap it out. Maybe you could even do something like profiling and remember over multiple sessions (per project/workspace) what the most used agents are and load those up before hand.

zamadatix 2mo ago

That's about the turning point for just using typical quants for me. Larger still and you can just do the full model. Smaller to this degree and you need all sorts of extra tricks to get anything.

fodkodrasz 2mo ago

> I wonder, is there a way to make the RAM usage tunable? In LM Studio I can tune it by selecting different quantation of the model, by selecting how many layers of the neural net to be loaded to GPU (rest stays in main mem, evaluated by the CPU), and by adjusting context window.

leonickson 2mo ago

It's tunable, --cache-gb N on the CLI. In my sweep the speed barely moved between a 1GB and 6GB cache (43% vs 70% hit rate, same tok/s) because right now the bottleneck is GPU dispatch, not the SSD. so more RAM doesnt buy much yet. once the kernel work lands it should start to matter, so on 32GB I would just set 8 and let it age well. Also the hit rates themselves answer the "can you even know which experts stay hot" question, reuse across tokens is very real.