5 ms·
> and fine‑tune them for your tasks This is the part that raised my eyebrows. Finetuning 70B is not just hard, its literally impossible without renting a very
by brucethemoose2 3y ago
> and fine‑tune them for your tasks
This is the part that raised my eyebrows.
Finetuning 70B is not just hard, its literally impossible without renting a very expensive cloud instance or buying a PC the price of a house, no matter how long you are willing to wait. I would absolutely contribute to a "llama training horde"
- Zetobal 3y agoAn H100 is maybe a car but not nearly close to a house...
- KomoD 3y agoMaybe not in your area, but it's very doable in other places, like where I live.
- ShamelessC 3y agoYou expect me to believe there are other places than where I live?!
- nextaccountic 3y agoIs a single H100 enough?
- brucethemoose2 3y ago80GB is enough, yeah. I'm not sure what exact LORA/quantization settings would be ideal, but check out https://github.com/OpenAccess-AI-Collective/axolotl#config https://github.com/OpenAccess-AI-Collective/axolotl#config
- ioedward 3y ago8 H100s would have enough VRAM to finetune a 70B model.
- AaronFriel 3y agoThat's true for conventional fine-tuning, but is it the case for parameter efficient fine tuning and qLORA? My understanding is that for a N billion parameter model, fine tuning can occur with a slightly-less-than-N gigabyte of VRAM GPU. For that 70B parameter model: an A100?
- zacmps 3y agoI think you'd need 2 80GB A100's for unquantised.
- brucethemoose2 3y ago2x 40/48GB GPUs would be the cheapest. But that's still a very expensive system, especially if you don't have a beefy workstation with 2x PCIe slots just lying around.
- deleted 3y ago[deleted]
- LoganDark 3y agoEven mATX boards tend to come with two (full-length) PCIe slots, and that's easy sub-$1k territory. Not exactly a beefy workstation. Source: have a $200 board in my computer right now with two full-length PCIe slots.
- 7speter 3y agoWhats more difficult is trying to cool gpus with 24-48gb of RAM… they all seem to be passively cooled
- LoganDark 3y agoGood point, I think most of them are designed for a high-airflow server chassis, with airflow in a direction that a desktop case wouldn't necessarily facilitate (parallel to the card).
- akomtu 3y agoWhat prevents parallel LLM training? If you read book 1 first and then book 2, the resulting update in your knowledge will be the same if you read the books in the reverse order. It seems reasonable to assume that LLM is trained on each book independently, the two deltas in the LLM weights can be just added up.
- contravariant 3y agoIn ordinary gradient descent the order does matter, since the position changes in between. I think stochastic gradient descent does sum a couple of gradients together sometimes, but I'm not sure what the trade-offs are and if LLMs do so as well.
- whimsicalism 3y agoBy the “delta in the LLM weights”, I am assuming you mean the gradients. You are effectively describing large batch training (data parallelism) which is part of the way you can scale up but there are quickly diminishing returns to large batch sizes.
- ctoth 3y agoThis is not at all intuitive to me. It doesn't make sense in a human perspective, as each book changes you. Consider the trivial case of a series, where nothing will make sense if you haven't read the prior books (not that I think they feed it the book corpus in order maybe they should!), but even in a more philosophical sort of way, each book changes you. and the person who reads Harry Potter first and The Iliad second will have a different experience of each. Then, with large language models, we have the concept of grokking something. If grokking happens in the middle of book 1, it is a different model which is reading book 2 and of course the inverse applies.
- eachro 3y agoI'm not sure this is true. For instance, consider reading textbooks for linear algebra and functional analysis out of order. You might still grok the functional analysis if you read it first but you'd be better served by reading the linear algebra one first.
- 3y ago
- malwrar 3y agoImpossible? It’s just a bunch of math, you don’t need to keep the entire network in memory the whole time.
- brucethemoose2 3y agoWell, any scheme where weights are dynamically loaded/unloaded from memory enough to fit on a 48GB GPU are so slow that training is basically impractical. Your 70B model would be obsolete by the time the finetuning is done. Some inference frameworks came up with schemes for just this, and it was horrifically slow.
- pavelstoev 3y agoYou can finetune 40B falcon on 4 x A10 with compiler optimization technology from CentML. No changes to the model.
- YetAnotherNick 3y agoFinetuning in a distributed way with questionable network would be lot more energy/cost inefficient than doing it with a single node or a well connected cluster. Also, you can finetune 70b model for million tokens for $2 in lambda cloud or <$10 in replicate.