3 ms·
What kind of hardware are you using and what is your token generation speed (tok/s)? Every time I've tried to run local models, even on my beefy mac book (128G
by johsole 2mo ago
What kind of hardware are you using and what is your token generation speed (tok/s)?
Every time I've tried to run local models, even on my beefy mac book (128Gb Ram), I've been very disappointed at tok/s speed.
- hermitShell 2mo agoI've been using dual NVIDIA GPU's with Ollama. Even when I push to 30b models and more context window, Ollama manages RAM / VRAM very well. I never got OOM errors, just massive slowdown as the PCI-E bottleneck throttles the GPU's. I've had some large prompts take 30 minutes to process. But I should mention: I am trying to implement workflows and processes akin to CI/CD that run 24/7 in the background. These are not interactive use cases, so I don't care so much about tokens per second.