4 ms·
and Glimmer has 10x active params of Lightning. Meaning ~ 10 slower on same HW
by rllearneratwork 2mo ago
and Glimmer has 10x active params of Lightning. Meaning ~ 10 slower on same HW
- XCSme 2mo agoIs that the case? If the entire model fits in vram, won't the tps be comparable?
- option 2mo agoautoregressive generation is memory bandwidth bounded
- XCSme 2mo agoSo, on a crude calculation, for a 3090 with 936.2 GB/s, a model that has 20 GB of active params would run at 45tps and one with 3GB active params at 300tps? In practice, I don't think I saw over 100tps on a 3090, for a local 20-30B model, be it MoE or not.