3 ms·
It would be nice if there was more information provided on that page. I assume this is just the output token generation speed. Is it using speculative decoding
by alyxya 11mo ago
It would be nice if there was more information provided on that page. I assume this is just the output token generation speed. Is it using speculative decoding to get to 1000 tokens/sec? Is there lossy quantization being used to speed things up? I tend to think the number of tokens per second a model can generate to be relatively low on the list of things I care about, when things like model/inference quality and harness play a much bigger role in how I feel about using a coding agent.
- cschneid 11mo agoYes this is the output speed. Code just flashes onto the page, it's pretty impressive. They've claimed repeatedly in their discord that they don't quantize models. The speed of things does change how you interact with it I think. I had this new GLM model hooked up to opencode as the harness with their $50/mo subscription plan. It was seriously fast to answer questions, although there are still big pauses in workflow when the per-minute request cap is hit. I got a meaningful refactor done, maybe a touch faster than I would have in claude code + sonnet? But my human interaction with it felt like the slow part.
- alyxya 11mo agoThe human interaction part is one of the main limitations to speed, where the more autonomous a model can be, the faster it is for me.