13 ms·
Hey! :) Coincidentally the seeds I always use are 3407, 3408 and 3409 :) 3407 because of https://arxiv.org/abs/2109.08203 https://arxiv.org/abs/2109.08203 I al
by danielhanchen 2y ago
Hey! :) Coincidentally the seeds I always use are 3407, 3408 and 3409 :) 3407 because of https://arxiv.org/abs/2109.08203 https://arxiv.org/abs/2109.08203
I also tried not setting the seeds, but the results are still the same - quantizing all layers seems to make the model forget and repeat everything - I put all examples here: https://docs.unsloth.ai/basics/deepseek-r1-dynamic-1.58-bit#non-dynamic-quantization-trial-output https://docs.unsloth.ai/basics/deepseek-r1-dynamic-1.58-bit#...
- iamnotagenius 2y agowould be great to have dynamic quants of V3-non-R1 version, as for some tasks it is good enough. Also would be very interesting to see degradation with dynamic quants on small/medium size MoEs, such as older Deepseek models, Mixtrals, IBM tiny Granite MoE. Would be fun if Granite 1b MoE will still be functioning at 1.58bit.
- danielhanchen 2y agoOh yes multiple people have asked me about this - I'll see what I can do :)