3 ms·
Inference speed is still slow in a meaningfully different way. The models are good, but not great, and much slower, which for coding means a 2-3 minute task wit
by datadrivenangel 6mo ago
Inference speed is still slow in a meaningfully different way. The models are good, but not great, and much slower, which for coding means a 2-3 minute task with claude code and opus takes an hour and has a higher chance of being wrong.
- 2ndorderthought 6mo agoIt's only slow if you can't afford to run it properly. A lot of people are getting 70-100 tokens per second on 1 gpu. Not sure what Claude opus or sonnet run at. I know when it goes offline it's 0 tokens per second