4 ms·
Oh I would always use Q2_K_XL :) It uses our dynamic methodology to quantize certain layers in different bits ie 2, 3, 4, 5, 6, 8 bits - the more important the
by danielhanchen 1y ago
Oh I would always use Q2_K_XL :) It uses our dynamic methodology to quantize certain layers in different bits ie 2, 3, 4, 5, 6, 8 bits - the more important the layer is, the higher the bitrate
- Squeeze2664 1y agoHow do you determine the importance of a layer in this case?
- kkzz99 1y agoAfaik they have a test bench that they use and take the activation data from that.
- danielhanchen 1y agoYes we have around 1 to 3 million tokens of high quality self verified data that we use to calibrate models!
- smallerize 1y agohttps://unsloth.ai/blog/dynamic-v2 https://unsloth.ai/blog/dynamic-v2
- danielhanchen 1y agoYes also https://unsloth.ai/blog/deepseekr1-dynamic https://unsloth.ai/blog/deepseekr1-dynamic, https://unsloth.ai/blog/dynamic-4bit https://unsloth.ai/blog/dynamic-4bit, https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs