The ideal ram capacity is determined by the biggest available model you want to run. So... ~48GB for llama 70B? Maybe more with batching and very long contexts.

The ideal ram capacity is determined by the biggest available model you want to run. So... ~48GB for llama 70B? Maybe more with batching and very long contexts. RAM bandwidth is basically always going to be a bottleneck. Microsft's proposal to get around this is to just keep everything in SRAM and pipe chips together.