4 ms·
This is so good. Is it kinda similar in concept to QLoRA training?
by freakynit 5mo ago
This is so good. Is it kinda similar in concept to QLoRA training?
- zqiu 5mo agoHi, thank you so much :) Yes, you are very sharp, to make truely it scalable, we have to finetune quantized LLMs directly, similar ideas as QLoRA. The algorithmic innovation is based on our research about orthogonal finetuning OFT and QOFT, which has demonstrated better training stability when finetuning quantized base models in such a scale.
- freakynit 5mo agoThank you for the information, creating this and releasing the research behind it. Truly appreciate.