5 ms·

So, as I understand it, this is for finetuning a preexisting llm? So not actually training one from scratch. I guess that would be too much to ask for. Nonethel

• Tepix 3y ago
  Training a 70b model from scratch uses 80,000 GPU hours (4,6 years if you have two of those GPUs). The electricity would cost more than 10,000€ in Germany, just for the GPUs.
• jph00 3y ago
  For now, it's for finetuning. The issue of to what degree it might be possible to train a model from scratch using QLoRA is still an open question. The relora paper showed that it can work in some situations, but attempts to scale it up were unsuccessful. The recent DoRA paper perhaps might allow a "re-DoRA" approach to work. If so, that could be combined with quantization to do "re-QDoRA"!
  • hantusk 3y ago
    Digging into the low rank structure of the gradients, instead of the weights seems like a promising direction for training from scratch with less memory requirements: https://twitter.com/AnimaAnandkumar/status/1765613815146893348 https://twitter.com/AnimaAnandkumar/status/17656138151468933...
    • hantusk 3y ago
      Simo linked some older papers with this same idea: https://twitter.com/cloneofsimo/status/1765796493955674286 https://twitter.com/cloneofsimo/status/1765796493955674286
  • qsi 3y ago
    The headline and introduction on the linked page say "You can now train a 70b language model at home. We’re releasing an open source system, based on FSDP and QLoRA, that can train a 70b model on two 24GB GPUs." How does "fine tuning" differ from "training?" Reading the linked article I had assumed I could create my own trained LLM at home with two 24GB GPUs.
    • jph00 3y ago
      The article actually sneaks in a footnote that answers this (https://www.answer.ai/posts/2024-03-06-fsdp-qlora.html#fn1 https://www.answer.ai/posts/2024-03-06-fsdp-qlora.html#fn1): "Throughout this article “training” can refer to either pre-training, or fine-tuning". (Generally, we've told students at fast.ai since 2017 that they should almost never be starting from random weights -- most of the time it's best to start with a pretrained model and fine-tune that, even if it's from a somewhat different domain to the problem you're working on.)
      • Tomte 3y ago
        Have you changed your mind on „The End of Finetuning“ (https://www.latent.space/p/fastai https://www.latent.space/p/fastai ) or did I simply misunderstand that? Oh, and thanks for quirky stuff like your APL video!
        • jph00 3y ago
          The title of that podcast isn't something I actually said (IIRC). I commented in that interview that I feel we should not consider pre-training and fine-tuning to be as separate as we do now.
          • Tomte 3y ago
            So you‘re generally in favor of mixing training data without separating them in phases, but when I use pretrained weights (as you recommend instead of random weights) I generally do not have access to whatever the neural net was pretrained with by someone else, so I have to make do with my finetuning data, yes? Thank you!
            • pama 3y ago
              Yes.
          • swyx 3y ago
            "The right way to fine-tune language models... is to actually throw away the idea of fine-tuning. There's no such thing. There's only continued pre-training." :) i hope i didnt pervert your intent too too much for clickbait or something, i thought it was the spirit of what you said
• buildbot 3y ago
  Lit-GPT is what I have been using to pretrain models at home: https://github.com/Lightning-AI/litgpt https://github.com/Lightning-AI/litgpt Using the openwebtext example, I can train a 700M param model to 2.6 loss in a few days on dual 4090s. Pretty awesome!