5 ms·
LLM speed is roughly <memory_bandwidth> / <model_size> tok/s. DDR4 tops out about 27Gbs DDR5 can do around 40Gbs So for 70B model at 8 bit quant, you will ge
by tgrowazay 8mo ago
LLM speed is roughly <memory_bandwidth> / <model_size> tok/s.
DDR4 tops out about 27Gbs
DDR5 can do around 40Gbs
So for 70B model at 8 bit quant, you will get around 0.3-0.5 tokens per second using RAM alone.
- vlovich123 8mo agoFaster than the 0.2tok/s this approach manages
- someguy2026 8mo agoDRAM speeds is one thing, but you should also account for the data rate of the PCIe bus (and/or VRAM speed). But yes, holding it "lukewarm" in DRAM rather than on NVMe storage is obviously faster.
- tgrowazay 8mo agoYes. In general systems usually have PCIE version with bandwidth better than RAM of that system. For example a system with DDR4 (27Gbs) usually has at least PCIE4 (32Gbs at 16x). But you can bottleneck that by building a DDR5 (40Gbs) system with PCIE4 card.
- zozbot234 8mo agoShould be active param size, not model size.
- tgrowazay 8mo agoYes, you’re right. LLama 3.1 however is not MoE, so all params are active. For MoE it is tricky, because for each token you only use a subset of params (an “expert”) but you don’t know which one, so you have to keep them all in memory or wait until it loads from slower storage, potentially different for each token.
- uf00lme 8mo agoChannels matter a lot, quad channel ddr4 is going to beat ddr5 in dual channel most of the time.
- wtallis 8mo agoFour channels of DDR4-3200 vs two channels of DDR5-6400 (four subchannels) should come out pretty close. I don't see any reason why the DDR4 configuration would be consistently faster; you might have more bank groups on DDR4, but I'm not sure that would outweigh other factors like the topology and bandwidth of the interconnects between the memory controller and the CPU cores.
- xaskasdf 8mo agoyeah, actually, I'm bottlenecked af since my mobo got pcie3 only :(