4 ms·
Unusable in what way? Llama 3.3 70b q8 with 100k+ context runs as well as GPT 3.5 did a couple years back, except local on Mac and smarter.
by Terretta 2y ago
Unusable in what way? Llama 3.3 70b q8 with 100k+ context runs as well as GPT 3.5 did a couple years back, except local on Mac and smarter.
- lumost 2y agoLarge context has different perf than large model. Op was likely thinking of running 400b models and finding the compute wasn’t enough to make the memory useful.
- imtringued 2y agoIf you don't have tensor parallelism even 8 x A100 is going to struggle with that kind of model.
- novok 2y agoLook on youtube running various models and even a 70b model does a slow 3tok/s