6 ms·
I need help understanding this. I understood that the magic here is the quantization that allows it to use from 50G to 4G and their process retain most of the i
by motbus3 3mo ago
I need help understanding this.
I understood that the magic here is the quantization that allows it to use from 50G to 4G and their process retain most of the intelligence within Pareto limits of gain. And then they proceed to compare with other quantized models as in the level of intelligence per size. It gets to my attention though that the performance in tool calling is mostly affected which is a problem for other small models.
How does this model compare to a recent 4G model? How do we know it retained intelligence from the parent rather then being fine tuned for the benchmarks?
I am not shtng on them or anything. I'd rather find it amazing, BUT given my limited knowledge, I feel the results miss fair comparison plots and the ones might be misleading. Buy I also reckon it might be me the problem.
Anyone care to explain this poor silly fellow some of those points?
- deleted 3mo ago[deleted]
- 0c3ca83 3mo ago[flagged]
- jedbrooke 3mo agofrom what I understand prismml isn’t doing a quant like normal models where you take a model trained at fp16 and then chop off some bits to reduce vram, but rather they’re training the model natively with 1 bit weights. It’s explained more in the article. They’re also doing some other tricks like a fp16 weight per block of 128 1bit weights to get some more data out of 1 bit weights
- parsimo2010 3mo agoThey aren’t training at all. They are quantizing existing models, it’s just that the process is different. The 27B uses Qwen3.6 27B as the base model.
- mehmetkose 3mo ago1-bit weights are not pointers so cpus can process them, storing them takes less space etc. tons of gains