4 ms·
What do you run it on? And even then, I'm guessing your tokens per second are not great?
by tills13 6mo ago
What do you run it on? And even then, I'm guessing your tokens per second are not great?
- CoolGuySteve 6mo agoI get about 35-40tok/sec on a 3090. It's actually about the same speed when accounting for how much more responsive my system is to Anthropic's saas infrastructure
- tills13 6mo agoI keep forgetting have a 3080 laying around... Gotta figure that out.