4 ms·
Is that the case? If the entire model fits in vram, won't the tps be comparable?
by XCSme 2mo ago
Is that the case?
If the entire model fits in vram, won't the tps be comparable?
- option 2mo agoautoregressive generation is memory bandwidth bounded
- XCSme 2mo agoSo, on a crude calculation, for a 3090 with 936.2 GB/s, a model that has 20 GB of active params would run at 45tps and one with 3GB active params at 300tps? In practice, I don't think I saw over 100tps on a 3090, for a local 20-30B model, be it MoE or not.