4 ms·
I prefer the not from scratch, but from configuration approach by Axolotl. Aolotl supports fine-tuning mistral, llama-2, with lots of the latest techniques - sa
by jamesblonde 3y ago
I prefer the not from scratch, but from configuration approach by Axolotl. Aolotl supports fine-tuning mistral, llama-2, with lots of the latest techniques - sample packing, flash attention, xformers.
I concentrate on collecting and curating the fine-tuning data, do "data-centric" fine-tuning - not learning LoRA from scratch.
- wfalcon 3y agothis is also what our (Lightning AI) lit-gpt library does. https://github.com/Lightning-AI/lit-gpt https://github.com/Lightning-AI/lit-gpt
- jamesblonde 3y agoThanks, hadn't seen this.