3 ms·
You most definitely can, the main difference is that only partial ~2% of the parameters get updated during training. Say you start from a model like llama-70B w
by keremturgutlu 3y ago
You most definitely can, the main difference is that only partial ~2% of the parameters get updated during training. Say you start from a model like llama-70B which already knows english and has some world knowledge based on its pretraining dataset. It might not be ideal for drastic domain shifts, such as adapting a model to learn new languages (which might require a new tokenizer and model embeddings) but still might be possible to some extent.
- qsi 3y agoThank you for clarifying. I have been wanting to dip my toes into LLMs at home but obviously I have a steep learning curve ahead of me, and would need considerably beefier hardware!
- chasd00 3y agoIt’s steep but manageable, absolutely go for it. The more people who understand the tech the better.