3 ms·
"We also made some smaller UD-1bit quants with UD-IQ1_S being 6.2GB (without MTP) which retain around 72% top-1% accuracy yet being 89% smaller" This is crazy!
by throwa356262 2mo ago
"We also made some smaller UD-1bit quants with UD-IQ1_S being 6.2GB (without MTP) which retain around 72% top-1% accuracy yet being 89% smaller"
This is crazy! But has anyone tried these lower quants on real projects?
- kennywinker 2mo agoNot 1-bit, but I’m getting pretty good results with some light coding using unsloth’s previous 2-bit quant of qwen3.8-27b. With these new quants i may be able to bump up to 3bit, tho it’s already running so slow (15tok/s average for the first 32k of context) that the speed hit might make it not worth the extra smarts
- Aurornis 2mo agoI tried some 1-bit, 2-bit, and bonsai quants against closed eval sets. They were essentially useless for my case. The little errors accumulate and send the whole output off track quickly. If you had some use case with very small output sequences they could be interesting to try. I think dropping down to a 9B-class model would produce better results for most cases.
- Havoc 2mo agoWhat setup are you using to do said private evaluation? Software wise I mean
- andai 2mo agoI wonder if this would help, or if it solves different kinds of errors. Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks https://news.ycombinator.com/item?id=48192383 https://news.ycombinator.com/item?id=48192383