3 ms·
And yet so many people on HN are adamant that the more tokens, the better, and it's all just a matter of throwing more money at it, and it's inevitable it will
by hammyhavoc 1y ago
And yet so many people on HN are adamant that the more tokens, the better, and it's all just a matter of throwing more money at it, and it's inevitable it will somehow "get better", because there's "so much money riding on it".
I wonder when the penny will drop?
- incomingpain 1y agoI wonder if my understanding is flawed. I've tested this using lm studio. Lots of dials are involved.
- incomingpain 1y agoI was just testing this a bit more. I grabbed qwen3:4b. Cranked it to the max of 32k tokens. It's fast to be sure, and im struggling to get it to hallucinate; but it is giving me a ton of 'The provided context does not include specifics' Resource-wise its like running 12-16B, but faster. But soon as you expand the 12B to like 10k tokens it's clearly better for barely anymore resources.