3 ms·
Why is token efficiency a concern with free models?
by bmitc 3mo ago
Why is token efficiency a concern with free models?
- cdud3 3mo agoBecause that's the only argument left after Kimi beats Fabel in results, price and autonomy.
- antiloper 3mo agoBecause you're paying for tokens. Especially output tokens.
- dbspin 3mo agoThey're not free to run, Kimi K3 needs to be run on the cloud, and the quantised versions aren't as capable. Unless you happen to have 3 - 5 TB of VRAM and an 8-node cluster of 8× NVIDIA H100s to run the full fat version. Plus the weights are not yet available to download in any case.
- walrus01 3mo agoI agree that quantized versions aren't perfect, but using GLM5.2 as an example, the gap between a BF16 and something like a Q8-K-XL as published by unsloth or a similar Q8 quantization is very minimal. For other "large" LLMs there's a fair number of tests showing that Q8 is about 94% as good at literally half the size in GGUF files on disk, and half the RAM usage. Approx. 1500GB for the BF16 vs 820GB for Q8-K-XL.
- aand16 3mo ago"Very minimal" unless the solution to your current task is in that missing %6 of capability.
- bmitc 3mo agoQwen was mentioned in the comment I replied to and can run locally.