4 ms·
Large context has different perf than large model. Op was likely thinking of running 400b models and finding the compute wasn’t enough to make the memory useful
by lumost 2y ago
Large context has different perf than large model. Op was likely thinking of running 400b models and finding the compute wasn’t enough to make the memory useful.
- imtringued 2y agoIf you don't have tensor parallelism even 8 x A100 is going to struggle with that kind of model.
- novok 2y agoLook on youtube running various models and even a 70b model does a slow 3tok/s