4 ms·
> did you also try using weighted loss with Axolotl This is really smart, I didn't think about this! Will add it to my list of things to try, great idea! > Do
by dmakian 3y ago
> did you also try using weighted loss with Axolotl
This is really smart, I didn't think about this! Will add it to my list of things to try, great idea!
> Domain adaptation over subreddits/forums before finetuning may help as well.
I was thinking about this too (along with transcribing draft youtube videos), I'd definitely be curious how much this helps.
- float-trip 3y agoRelated comment from gwern: https://news.ycombinator.com/item?id=38438859 https://news.ycombinator.com/item?id=38438859. Can't find the docs now - I think they were the old GPT 3 ones - but they suggested a low value somewhere around 0.01 and 0.1. Also - why qlora rather than a full finetune? Using LambdaLabs, it'd cost roughly the same as your quote. Cheaper I think if you're willing to gamble with fp8: https://github.com/mosaicml/llm-foundry/tree/main/scripts/train/benchmarking https://github.com/mosaicml/llm-foundry/tree/main/scripts/tr.... And fewer hyperparameters to tune as well