3 ms·
The article actually sneaks in a footnote that answers this (https://www.answer.ai/posts/2024-03-06-fsdp-qlora.html#fn1 https://www.answer.ai/posts/2024-03-06-f
by jph00 3y ago
The article actually sneaks in a footnote that answers this (https://www.answer.ai/posts/2024-03-06-fsdp-qlora.html#fn1 https://www.answer.ai/posts/2024-03-06-fsdp-qlora.html#fn1): "Throughout this article “training” can refer to either pre-training, or fine-tuning".
(Generally, we've told students at fast.ai since 2017 that they should almost never be starting from random weights -- most of the time it's best to start with a pretrained model and fine-tune that, even if it's from a somewhat different domain to the problem you're working on.)
- Tomte 3y agoHave you changed your mind on „The End of Finetuning“ (https://www.latent.space/p/fastai https://www.latent.space/p/fastai ) or did I simply misunderstand that? Oh, and thanks for quirky stuff like your APL video!
- jph00 3y agoThe title of that podcast isn't something I actually said (IIRC). I commented in that interview that I feel we should not consider pre-training and fine-tuning to be as separate as we do now.
- Tomte 3y agoSo you‘re generally in favor of mixing training data without separating them in phases, but when I use pretrained weights (as you recommend instead of random weights) I generally do not have access to whatever the neural net was pretrained with by someone else, so I have to make do with my finetuning data, yes? Thank you!
- pama 3y agoYes.
- swyx 3y ago"The right way to fine-tune language models... is to actually throw away the idea of fine-tuning. There's no such thing. There's only continued pre-training." :) i hope i didnt pervert your intent too too much for clickbait or something, i thought it was the spirit of what you said