3 ms·
Open-source LLM projects have largely solved this using Low-Rank Adaptation of Large Language Models (LoRA): https://arxiv.org/abs/2106.09685 https://arxiv.org/
by jiggawatts 4y ago
Open-source LLM projects have largely solved this using Low-Rank Adaptation of Large Language Models (LoRA): https://arxiv.org/abs/2106.09685 https://arxiv.org/abs/2106.09685
Apparently an RTX 4090 running overnight is sufficient to produce a fine-tuned model that can spit out new Harry Potter stories, or whatever...
- nico 4y agoHow much would they cost aprox per training? What would the quality of the model be, compared to what Karpathy uses on his video “let’s build gpt from scratch”? In that video, he builds a decoder-only transformer model, that learns Shakespeare from 1MB of data and trains in 15min
- jiggawatts 4y agoThese just fine-tune existing models, so the cost is single digit dollars. Whatever it costs in electricity to run a desktop GPU for a day or a few days.