18 ms·
A 10GB 3080 still beats even an M2 Ultra with 192GB... memory bandwidth is not the only factor. https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference
by ranger_danger 8mo ago
A 10GB 3080 still beats even an M2 Ultra with 192GB... memory bandwidth is not the only factor.
https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inferen...
- seanmcdirmid 8mo agoIf the model is small enough to fit in to 10GB of VRAM the GPU can win. But the bigger models are more useful, so that’s what people fixate on.