2 ms·
> It's interesting how even 5 tok/s is still much faster than you'd typically type, but feels glacially slow for an agent. Calling the token rate the rate at w
by kqr 5mo ago
> It's interesting how even 5 tok/s is still much faster than you'd typically type, but feels glacially slow for an agent.
Calling the token rate the rate at which they "type" is a bit misleading. They also do virtually all of their more complex reasoning in tokens, so 5 tokens per second is also their thinking speed. And thinking at 5 tokens per second is glacially slow.
This is why faster versions of strong models do so well on reasoning tasks like playing text adventure games[1]. Their output isn't better on a token-for-token basis, but they get so much more thinking in during a given time window, they get more opportunities to find the right conclusion.
[1]: https://entropicthoughts.com/updated-llm-benchmark https://entropicthoughts.com/updated-llm-benchmark
- michibertel 5mo agoHow many tok/s does an average human think?
- madwolf 5mo agoMost of my thinking is non-verbal. I don't think in sentences. I CAN think in sentences and internally rationalize my actions and explain them and sometimes that's beneficial (rubber duck debuggin, sometimes it's good to verbalize and explain something) but usually I don't do it
- kqr 5mo agoThis question gets into information theory way beyond me, but I suspect it depends a lot on the task at hand. Human brains aren't very effective at combining sources of statistical variation, but they're great at other things. I'm personally most impressed by the cerebellum. It is highly trainable, yet if we tried to translate the things it does to maintain locomotion, proprioception, coordination of movement, etc. into tokens would probably result in a high token rate.