15 ms·

Fine tune a 70B language model at home

• lbj 3y ago
  Can't believe they didn't name this Qolor
• tbenst 3y ago
  Very interesting but hard to interpret until the performance numbers / benchmarks are available. I can already fine-tune a 70B language model at home using CPU + RAM, but it would be so slow as to be almost totally impractical (~20x slower than GPU). It would be great to see a comparison to eg 8 x A100 (available for $32/hr on AWS on-demand) and also CPU + RAM. Presumably it’s somewhere in between, but hard to predict where!
• int_19h 3y ago
  This is great, but one thing I really hoped would come sooner is fast training on Metal. As things are, you can get an M1/M2 Ultra (~800 Gb/s memory bandwidth; for comparison, RTX 4090 is ~1050 Gb/s) Mac Studio with 128Gb RAM for ~$3500. For large model inference, this is already way more affordable than stacking GPUs while being "fast enough", but training solutions are basically non-existent. I do wonder why; it feels like a low-hanging fruit.
  • sqreept 3y ago
    M1, M2, M3 still have very low number of GPU cores. Apple should release some better hardware to take advantage of their recently released MLX library.
    • sbinnee 3y ago
      At this moment it looks clear to me that Apple won’t go that way. It’s enough for them to focus on inference and actual application not the heavy training part. They have been probably training models on a cluster with non Apple silicon and make them available for their chips only for inference.
      • ttul 3y ago
        Not to mention entirely outsourcing training workloads to specialist firms. Apple does a lot of secretive outsourcing of things you might think they would or should do in-house. This contrasts with Google and Meta who seem to like keeping everything in-house.
    • kergonath 3y ago
      It’s true that their GPUs are slower than Nvidia’s. But keep in mind that cores are really different and cannot be compared across architectures. You want more Gflops, not necessarily more cores.
    • int_19h 3y ago
      They do, but for inference at least, it's memory bandwidth that is the primary limiting factor for home LLMs right now, not raw compute.
      • sroussey 3y ago
        Wonder if the apple silicon ultra series will start using HBM3(e) on desktop in the future.
  • buildbot 3y ago
    Compute limited - an m2 ultra has 27 tflops, a 4090 80+
    • erichocean 3y ago
      Memory limited - an m2 ultra has >150GiB, a 4090 24GiB
      • lawlessone 3y ago
        So why is nobody doing this?
        • ttul 3y ago
          My personal experience with Apple Silicon and machine learning in comparison with Nvidia is that the libraries are often missing various features on Apple, leading to headaches when trying to use various libraries and tools. Apple is working to bridge the gap and I am excited for the gap to be closed because the memory bandwidth on big M2 and M3 machines is monstrous.
          • lawlessone 3y ago
            sounds similar to how people have described game dev for Mac. The hardware is there. It just isn't supported.
            • imhoguy 3y ago
              Apple could single-handedly kill consumer dGPU market if they have released proper low-level APIs for their M1/2/3. I feel they have some huge coming out in the pipe to tumble the "AI" market.
              • deleted 3y ago
                [deleted]
    • 3y ago
• delegate 3y ago
  Maybe I've missed it in the article - but how long would a full training run take on 2 consumer GPUs (local or rented) ? Ballpark - hours, days... ?
  • gardnr 3y ago
    The author is discussing fine-tuning a base model. How long it takes really depends on the dataset, the method, and the hyperparameters. DPO, for example, can achieve some great results with a fraction of the steps of other methods. Just like with unsloth or axolotl, the people that use this will have to make compromises that give results in a reasonable amount of time.
• eurekin 3y ago
  This might be the most interesting constructive approach in "Open Source" LLMs I've seen. Grounded, reasonable and inviting to replicate! I wish academia took that as a standard. Great job!
  • carlossouza 3y ago
    Answer.ai is truly open AI. :)
    • rvz 3y ago
      That's what was said about OpenAI, Mistral, before the VCs and investors came in. After that, the larger flagship AI models were then closed up again and used as an server only offering.
      • ericd 3y ago
        I doubt it, Jeremy’s been walking the walk for quite a while now, when it comes to opening up access to AI, especially with his excellent, free fast.ai course, it seems pretty clear that his primary motivations are in helping others. (If you’re in this thread, Jeremy, thanks for fast.ai, it helped me immensely in getting started in training models).
  • 20wenty 3y ago
    For the most part this post was easy to read, and I could feel the collective excitement of the team. I came away feeling like I'd learned something and ready to try it myself. The only time the post gets a little fuzzy is "...store the quantized parameters in a selectable data type, where that storage data type is the same data type as the “computation type” of the mode". I assume "selectable datatype" is the float size of the quantization?
    • Yenrabbit 3y ago
      We've got a technical post with all the juicy details coming next week. But that bit refers to packing the 4-bit weights into a type FSDP is happy to shard (like float16 or float32) which matches the other non-quantized bits of the model. This way FSDP will happily wrap and shard all the parameters as if they were just normal floats.
• hathym 3y ago
  Imagine the potential of a Folding@Home-inspired project for AI development. What kind of powerful model could a community of gamers and GPU owners create.
  • nilsherzig 3y ago
    Wouldn't bandwidth be a big problem? And I think models like llama already require (ten)thousands of gpus to train in a reasonable timeframe
• llmzero 3y ago
  I liked that you link to renting a dual 24GPU for 0.60cents/hour, but how long could it takes to fine tune a 70b model using your system (4 bits for weights)? If I were a consumer I would be interested in the final price of fine tuning, for example a table with model size, training size, cost of training, and expected loss of quality with this technology. One obvious question: Can you apply your technology with the recent (-1,0,1) encoding?, I think you will answers that the (-1,0,1) model is not available and you can't try it, but my question is whether once/if that model is available answer.ai will be able to use the same technology that this post to fine tune a big model in two very small GPUs, and then I should ask for a new table with cost/benefits analysis. Edited: I should add that I find this kind of work very useful for enhancing individual users like me to be able to compete in the applications of LLM market, this is great work and along the lines of the book "from zero to one" (not that I like or dislike the author) to solve the kind of problem that nobody is trying to solve. Edited: Now that I have a total of 23 points in HN, I will change my password to some random one, just to cure my desire to look for votes, and try to make some work, and again some tomorrow create a new presence in HN.
  • jph00 3y ago
    As mentioned in the post, benchmarking results are coming in a later post. But in short: you can train an epoch of Alpaca in 24 hours or so, which is enough to get very significant change in model behavior.
  • danielhanchen 3y ago
    On how long, finetuning is influenced by your dataset size (more = slower), sequence length since attention is O(N^2), data movement etc and most important is how many steps you want to take. For QLoRA, some runs can do a few hundred steps which can complete in minutes to 1 hour. Too many can overfit. So being able to fit it on consumer GPUs can be very cost effective. On the 1.58bit paper, from what I understand, this requires a total retraining from scratch. Hopefully the researchers will open source their weights :) On the technicals, weights are encoded in (-1, 0, 1), whilst QLoRA uses a 4bit dynamic mapping of 16 numbers. The only change required would be the torch.matmul(X, W) step, where it'll be torch.bitlinear_matmul(X, W). Before with QLoRA, one has to do torch.matmul(X, dequantize(W)). So one has to implement torch.bitlinear_matmul. The backward is torch.bitlinear_matmul(dY, W.T).
• Kelteseth 3y ago
  Any plans on supporting AMD? In Germany, the price of an 7900XTX is HALF of a NV 4090...
  • slices 3y ago
    take a look at recent posts from https://twitter.com/__tinygrad__ https://twitter.com/__tinygrad__ re: the state of AMD for AI work
• vouaobrasil 3y ago
  It would be great if we were a bit more respectful towards our natural resources. Using so much energy to play with language models is a waste of resources.
  • firtoz 3y ago
    The point in the article is to use less resources. So, yes?
    • vouaobrasil 3y ago
      People think that by making a system use less resources, the entire use of it on a societal level will be reduced. Unfortuantely, we must watch out for more efficiency making more poeple use it, and potentially increasing absolute quantity of energy being used.
      • MacsHeadroom 3y ago
        Energy usage is good, actually. Energy scarcity and dirty energy are bad. But energy usage is good. We should strive to use and produce orders of magnitude more (clean) energy.
  • nraford 3y ago
    It would be great if I had a bathtub full of ice cream as well, and if we all lived in a world overflowing with love, respect and joy for all living things. Until then, I'm happy that these kinds of incredible tools are (and increasingly will be) be in more of our hands for close to free. Upwards and onwards!
    • vouaobrasil 3y ago
      Seems like with every passing year, we are going downwards, not upwards. Perhaps it only seems the other way around to those with the greatest addictions to technology, who will justify any development to satisfy their cravings.
• chompychop 3y ago
  Does this support multimodal language models (E.g.: LLaVA)?
• pama 3y ago
  Thank you for the repo and write up. What tools (if any) did you use for performance tuning once you achieved the main goal of being able to finetune the model?
• zerop 3y ago
  Question - Can I use this to retrain an LLM (70B) weights on my own data? I am using RAG as of now for asking questions on my text, but always wonder if I could retrain an LLM on my own text. Thoughts?
  • AnthusAI 3y ago
    Fine tuning is generally not the best way to teach an LLM new knowledge. RAG is still more appropriate. Fine tuning is generally more effective for controlling the format of the responses but it's not going to teach the model a lot of new concepts. The model can learn how to handle new vocabulary through fine tuning but it's not a great way to teach the model new facts. Giving it access to a knowledge base is a better way to do that.
• jl6 3y ago
  Besides being a great result, the quality and clarity of the technical writing here is excellent.
• iandanforth 3y ago
  This is great, however there were many opportunities to use the word 'nibble' in this post and they were all missed.
• chasd00 3y ago
  What’s the best way for people to contribute to AI open source? I can’t produce things like this for many reasons so how can I and others like me do our part to keep SOTA AI open?
  • deleted 3y ago
    [deleted]
  • sophrocyne 3y ago
    There is a ton you can do to help SOTA AI remain open. Join the community building the tools - Help with UI/UX, documentation, keeping up with the latest, and evangelizing whatever method the team building it has devised to keep it sustained. Being part of the community itself is more valuable than you realize.
    • SamPatt 3y ago
      Where are you finding this community?
  • ativzzz 3y ago
    Off the top of my head - try to implement techniques that are doable on home hardware like the one described in OP (requires some $$$ investment) and give feedback or contribute to documentation / guides - learn about different techniques and do educational writeups or documentation (like https://vickiboykis.com/2024/02/28/gguf-the-long-way-around/ https://vickiboykis.com/2024/02/28/gguf-the-long-way-around/) - build a tool / library that wrap academic techinques and expose them more easily to end users (like A1111 or comfyUI for stable diffusion) Anything that can translate the high end research down to something a moderately technical user can use or learn from is a gigantic win
  • hamilyon2 3y ago
    I am random software engineer, but from what I learned high-quality open source data sets seems to be enabler. There is shortage of golden datasets for training and evaluation in every popular and niche area you can imagine.
• sieszpak 3y ago
  4x 3080???
  • jph00 3y ago
    That should work great, although we haven't tried it. Let us know what you find!
  • deleted 3y ago
    [deleted]
  • kristianp 3y ago
    If you're buying new, 3x 4070 Super Ti 16GB might be better than 4x 3080 12GB: 4070 super ti 16GB, fp32 44 tflops, Mem 672 GB/s 3080 12gb, fp32 30 tflops, Mem 760 GB/s 3080 Ti 12GB fp32 34 tflops, Mem 912 GB/s For 50% more gddr, a 4070 Super Ti costs less than 50% more than a base 3080. There's a tradeoff between FLOPS and memory bandwidth though, but using 3 cards instead of 4 should makeup for that. https://www.techpowerup.com/gpu-specs/geforce-rtx-3080-12-gb.c3834 https://www.techpowerup.com/gpu-specs/geforce-rtx-3080-12-gb... https://www.techpowerup.com/gpu-specs/geforce-rtx-3080-ti.c3735 https://www.techpowerup.com/gpu-specs/geforce-rtx-3080-ti.c3... https://www.techpowerup.com/gpu-specs/geforce-rtx-4070-ti-super.c4187 https://www.techpowerup.com/gpu-specs/geforce-rtx-4070-ti-su...
• openquery 3y ago
  This article is very well written and super informative. One thing I didn't understand is: > At Answer.AI our north star is making useful AI more accessible. $150,000 to create your own high-quality personalized model definitely doesn’t count as accessible! Renting an A100 on RunPod is ~$1.89 / hour. So you'd need ~80,000 A100 hours to train a useful AI model?
  • humansareok1 3y ago
    In the post it explicitly says you can train on 2 3090 level cards which are significantly cheaper and the headline literally says "Finetune" Not "Pretrain"
• jncfhnb 3y ago
  So… why do people want to fine tune LLMs at home? It seems very unlikely to provide value. * you’re probably not going to succeed at injecting new knowledge in a way that feels satisfyingly top of mind to the bot * you’re probably not going to create such a meaningfully new style that it warrants a Lora like in images What’s an example use case?
  • Solvency 3y ago
    Illicit fan fiction. Whether it's image or text models. It's ALWAYS illicit fan fiction.
    • CharlesW 3y ago
      Consensual sex between Gilligan and the Professor is not a crime.
    • jncfhnb 3y ago
      I mean I’ve seen the expressive niches on image models of civitai, but do you really need custom fine tuned LLMs for text fanfiction? Like sure, you need something that is not the friendly question answerer; but do you really need such a broad population as in images to suit your needs? I’m guessing no?
  • tracerbulletx 3y ago
    Hmm why would someone on a forum called hacker news want to tinker and explore an exciting new technology. Who can say? One of life’s great mysteries really.
    • jncfhnb 3y ago
      I’m curious what they’re trying to do because I’m curious and I don’t see it. You’re the one being a dismissive dick here.
      • SamPatt 3y ago
        >So… why do people want to fine tune LLMs at home? It seems very unlikely to provide value. Asking the first question is fine, but your follow-up comment sounds more dismissive than curious. That's probably why the snarky response.
• curl-up 3y ago
  Does anyone have sources, or experience, about fine tuning primarily to teach the model some factual data, especially when it comes to later "higher level" question answering. For example, giving the model a bunch of text (academic papers and such) about 19th century writers, then asking things like "Who were the main influences on writer X"? Obviously simple RAG-like approaches don't work, as such information is rarely available in the text as-is, and needs to be "extrapolated" to some extent. Long context models might work (just dumping everything into the prompt), but are way too expensive for my needs.
  • armcat 3y ago
    RAG approaches should work quite well for the examples you mentioned. It's a matter of how you approach the retrieval part - you can opt for a larger recall on retrieval, and leverage the large context window for the LLM to figure out the answer. Even if it's not "as-is", semantically if it's in there, it should be able to find it. Other things to try out is how you approach the question "expansion" part, for example using Hypothetical Document Embeddings (HyDE); or how you approach the filtering-out part, e.g. using "System 2 Attention", https://arxiv.org/abs/2311.11829 https://arxiv.org/abs/2311.11829.
    • curl-up 3y ago
      I tried most of such techniques, but the point is that this information really isn't in there directly, and to perform the question expansion, the model needs to know about the domain already. For example, imagine that one paper is about how author X was French, in early 19th-c.and how they were one of the first ones to write about topic T. Another paper is about how author Y was inspired by the early 19th-c. French writers writing about T. However, this second article does not mention X at all. Asking about "who were the main influences on X" would not give you the second article. Of course, I could run "multiple-hop" RAG-like process, where the model keeps asking questions itself and so on in a loop, but this becomes extremely clumsy, and the models (even GPT-4) tend to get out of hand. It is also extremely slow, of course.
      • 3y ago
• jiwidi 3y ago
  > home > two 24GB GPUs. geez
• Nouser76 3y ago
  Is there any framework/system that distributes the work across multiple GPUs on different computers over a network (LAN or WAN)? I'm not concerned much about latency or generation time, but would love to train or load up huge models and send jobs to run overnight.
  • jph00 3y ago
    Yes, you can run FSDP/QLoRA over multi-node. There's a slurm script in the repo showing how to do it.
• ericd 3y ago
  This is the best news I’ve seen all month. I think one of the great near-term dangers of AI is the bulk of the economic benefit going mainly to relatively few companies. That risk seems substantially reduced if they have to compete with a great variety of models.
• Tostino 3y ago
  Nice, i've been hoping this would be possible for a while. I'll have to do a new fine-tune of Inkbot on top of one of the 70b models. What are the max context lengths / batch sizes you can train at with this method for 2x24gb? What about 4x24gb?
• samstave 3y ago
  OK this is going to come out as moronic because I dont have the proper vocab to phrase it: -- Is it possible to 'array' tokenized wokloads across providers of GPU? I want to farm-out my 'compute' across [things] More importantly can there be a marketplace for GPU resources that I can effectively point my local job at?
  • jph00 3y ago
    Yes you can use multiple computers in parallel with this script. And there's lots of GPU providers out there (the article links to an aggregator of providers and prices).
• Havoc 3y ago
  This is great. I don't think local will be competitive in future IN GENERAL...but if I have a specific use case and I have a specific training dataset...local with specific training will murder the big commercial models.
• staticman2 3y ago
  If I wanted to use this software to finetune a 70b model on two 3090s to write fiction, what is the maximum sequence length that would be practical? I'm at the dataset collection stage, but I'm not sure whether to aim for bigger or smaller sequence lengths at the moment.
• JoelJacobson 3y ago
  Do the two 4090 GPUs need to be on the same machine, or is it possible to somehow use two separate machines, each with its own 4090, and link them somehow via e.g. InfiniBand?
• Tepix 3y ago
  Congratulations, fantastic contribution to open source AI. Why does the website headline say "train" instead of "finetune"?
• OOPMan 3y ago
  Ah yes, 24gb top-of-the-line GPUs, I happen to have a whole warehouse full. /s
  • jph00 3y ago
    Fair -- 24GB GPUs are still quite expensive. >10x cheaper than 80GB GPUs though! We already have something we're testing that's gotten 70b training down to dual 16GB GPUs, and we're working it making it even smaller...
• Donberry 3y ago
  [dead]
• erwincoumans 3y ago
  NVlink + two 3090 gives 48GB relatively easy (appears as unified memory). I only skimmed the article briefly, was it considered?
  • jph00 3y ago
    NVlink does not appear as unified memory. But NVlink + two 3090 does work great with FSDP/QLoRA (I have just such a machine at home!)
• seratibp 3y ago
  Is it worth it though when the base model isn't even smart enough?
• deleted 3y ago
  [deleted]
• artninja1988 3y ago
  So, as I understand it, this is for finetuning a preexisting llm? So not actually training one from scratch. I guess that would be too much to ask for. Nonetheless, cheers to Jeremy and the gang for the work.
  • Tepix 3y ago
    Training a 70b model from scratch uses 80,000 GPU hours (4,6 years if you have two of those GPUs). The electricity would cost more than 10,000€ in Germany, just for the GPUs.
  • jph00 3y ago
    For now, it's for finetuning. The issue of to what degree it might be possible to train a model from scratch using QLoRA is still an open question. The relora paper showed that it can work in some situations, but attempts to scale it up were unsuccessful. The recent DoRA paper perhaps might allow a "re-DoRA" approach to work. If so, that could be combined with quantization to do "re-QDoRA"!
    • hantusk 3y ago
      Digging into the low rank structure of the gradients, instead of the weights seems like a promising direction for training from scratch with less memory requirements: https://twitter.com/AnimaAnandkumar/status/1765613815146893348 https://twitter.com/AnimaAnandkumar/status/17656138151468933...
      • hantusk 3y ago
        Simo linked some older papers with this same idea: https://twitter.com/cloneofsimo/status/1765796493955674286 https://twitter.com/cloneofsimo/status/1765796493955674286
    • qsi 3y ago
      The headline and introduction on the linked page say "You can now train a 70b language model at home. We’re releasing an open source system, based on FSDP and QLoRA, that can train a 70b model on two 24GB GPUs." How does "fine tuning" differ from "training?" Reading the linked article I had assumed I could create my own trained LLM at home with two 24GB GPUs.
• ricopags 3y ago
  This is such exciting news! Huge thanks to you for your continued work in making sense of AI. I wonder if the recent Bitnet 1.58 paper [the use of ternary bits in lieu of fp/int] might be an advancement that could further reduce the computation required for inference?
  • jph00 3y ago
    Yes, along with the many other <4 bit quant methods recently developed -- there's been a wonderful boom in low-bit quant methods in the last 6 months, and we've got our own ideas for taking them further too. Along with QLoRA/FSDP, we're likely to see big advances in model training this year on consumer hardware.
• buildbot 3y ago
  Nice, I tried to use QLoRA+FSDP in the past with litgpt and obviously at that time it did not work. This is very useful!
• jph00 3y ago
  One thing I forgot to mention in the post which I think is kinda cool: at the NeurIPS Efficiency Challenge this year, where Tim Dettmers and I both did keynotes, every single top-ranked entry used QLoRA! The challenge was to create the most accurate model on a single GPU in 24 hours. I think that is a great example of how important and useful QLoRA is. Maybe we should run a dual-GPU challenge next time not that multi-GPU is working...
  • TaylorAlexander 3y ago
    Congrats on the release. On Twitter someone asked you if you would provide any risk assessment reflection, and you replied that the risk would be similar to the release of a new model of pen or pencil. That reply, while cute, isn’t accurate. A new pencil does not mean new capabilities for humanity, while the whole point of your release is that it does afford new capabilities and therefore new risks. The woman, a professor working on the societal impacts of AI, asked a straightforward question in apparent good faith [1]. Your reply did not seem to me to be in good faith. Can you explain the apparent disconnect? I’m less concerned as to whether or not you would release an assessment and more concerned at the dismissive attitude, especially towards a female professor studying societal impacts of AI, a fellow researcher. [1] https://x.com/dr_atoosa/status/1765874818283385071 https://x.com/dr_atoosa/status/1765874818283385071
    • jph00 3y ago
      I believe my answer is accurate. I don't know on what basis you claim otherwise. This is my work, and I'm well placed to make the assessment. My response is based on reading a great many papers and doing many experiments over the last few years, and I am confident of it. By way of background: I studied the philosophy of ethics at university, I co-authored a book chapter on AI/ML ethics, my wife and I quit our jobs and worked for free for years entirely focused on trying to help society adapt to and benefit from AI, I wrote the actual article we're commenting on, and the article is about LLM fine-tuning -- a field I to some extent created by developing the ULMFiT algorithm. The person in question is, IIRC, at Governance AI, a group whose work I spent months studying in depth -- work which I believe is more likely to cause harm to society than to benefit it, as I explained here: https://www.fast.ai/posts/2023-11-07-dislightenment.html https://www.fast.ai/posts/2023-11-07-dislightenment.html
• pella 3y ago
  > the ability to use multiple GPUs with QLoRA training. Thorough article! Question: What's your opinion on: - How viable will NVIDIA's consumer cards be in the long run? - Besides https://tinygrad.org https://tinygrad.org, what other cost-effective future alternatives could there be?
  • bugglebeetle 3y ago
    Unsloth (mentioned in the Answer.AI post) is planning multi-GPU support in a future release.
• carbocation 3y ago
  I wonder whether LoRAs could be useful for U-Net training. Especially thinking of CNN-based U-Net models with pre-trained encoders (but randomly initialized decoders). At least, it seems possible that normal weight updates on the decoder and LoRA training on the encoder could improve efficiency.
  • jph00 3y ago
    Diffusion unet has an "extended" version nowadays that applies to the resnet part as well as the cross-attention: https://github.com/cloneofsimo/lora https://github.com/cloneofsimo/lora
• m3kw9 3y ago
  If they can continuously train it, it could be better than a large context as this is how a AI OS would need to work when you have constant updates to your files
  • padolsey 3y ago
    I don’t think you’d be fine-tuning a whole model in such cases. That seems over the top, no? I assume you’d get sufficiently far with big context windows, vector search, RAG. Etc.
    • jph00 3y ago
      It's an interesting question. I'm not sure we really know yet what the right mix of RAG and fine tuning is. IMO small-scale fine tuning might be under-appreciated.
• keeptrying 3y ago
  If you are gonna be doing stuff like this I’m damn excited for answer.ai! It’ll be the first time we’ll have someone who knows AI create leverage to open source it. Way to go!
  • chasd00 3y ago
    > It’ll be the first time we’ll have someone who knows AI create leverage to open source it. It can’t be overstated how important this is. Thank you again.
• g42gregory 3y ago
  This is brilliant. Thank you for doing his!
• yalok 3y ago
  Have you guys looked at using sparsification? It would probably require true re-training of the foundation model, to go at high sparse ratios (say 90% weights excluded), which could be done once on expensive GPU - but fine tuning such sparse models would require less RAM hopefully. The trick with getting more benefit from sparse approach is to do block sparse (iirc, Tim Dettmers used to work on this as well, a few years ago), but large block size (say 16x16) would require much longer retraining to recover for the lost accuracy…
  • jph00 3y ago
    Yes, sparsification is another useful approach for higher efficiency, although block sparse kernels are pretty complex to work with -- especially when combined with quantization and LoRA! Most of the sparsity papers I've seen use "structured" sparsity; i.e removing layers, attention heads, and features. But the upside from this seems somewhat limited so far.
    • yalok 3y ago
      I'm not sure about structured sparsity, but for the weights sparsity in my experience going to around 50-70% of excluded weights (even with block sparsity - say 4x4) did not cause any noticeable degradation to training & quality at all (original paper on sparsity from LeCun suggests much higher sparsity ratios - like 90% - but for DNNs I didn't find those attainable if accuracy is important) The block sparsity can really help with saving RAM - because you only need to keep a short array of indexes for the excluded weights. The trouble is the kernel mult functions become complex, so it's a bit of a trade-off between RAM and GPU cycles.
  • AhtiK 3y ago
    Has anyone seen an implementation of 'SpQR: A Sparse-Quantized Representation,' published in June 2023 by Tim Dettmers et al.? https://arxiv.org/abs/2306.03078 https://arxiv.org/abs/2306.03078
    • AhtiK 3y ago
      Found it from https://github.com/Vahe1994/SpQR https://github.com/Vahe1994/SpQR Was somehow expecting it to be at https://github.com/TimDettmers/bitsandbytes https://github.com/TimDettmers/bitsandbytes. My bad.
• jamesblonde 3y ago
  This is a fantastic breakthrough for those of us who fine-tune LLMs on limited hardware budgets. I was curious about the choice of FSDP over DeepSpeed. I have been using Axolotl for fine-tuning, and FSDP has been broken there, whilst DeepSpeed is rock solid. Why FSDP over DeepSpeed jph00?
  • jph00 3y ago
    DeepSpeed has more features than FSDP, but it's much more complex to hack on -- FSDP is written directly in python using calls to the PyTorch library, whereas DeepSpeed is 20% C++ and 10% CUDA (according to the GitHub stats). We've found that FSDP works just as well for our needs, and we appreciated the increased "hackability". (Axolotl is terrific BTW. I hadn't heard of problems with it with FSDP before -- I'll see if that's something we can help with.)
    • jph00 3y ago
      Good news -- axolotl has just merged support for FSDP/QLoRA training, thanks to a rapid collaboration between the axolotl and Answr.AI teams!
    • bradfox2 3y ago
      There's a long gh issues thread with technium struggling with Mistral 7 and loss spikes. Easy to find googling.
      • jph00 3y ago
        Yes I'm familiar with Teknium's Mistral issues, which were resolved some time ago. IIRC they weren't related to FSDP.
• itsgrimetime 3y ago
  Would be cool to build an “LLM@home” project like folding@home or SETI@home (rip), where tons of folks could donate their GPUs and train something huge and FOSS. I don’t know enough about how these models are trained though. Could it be chunked up and distributed in that way, then stitched/merged back together?
  • woctordho 3y ago
    https://github.com/learning-at-home/hivemind https://github.com/learning-at-home/hivemind is also relevant
  • humansareok1 3y ago
    Always figured it would be too slow. Distributed training on clusters is usually done with 1+ gb/s interconnects.
  • miohtama 3y ago
    Golem has been building this since 2017 https://www.golem.network/ https://www.golem.network/ They also have on option to get paid in crypto for your GPU power. The challenge is that the AIsoftware architectures are not made "to run over Internet."
  • fho 3y ago
    https://stablehorde.net/ https://stablehorde.net/ comes somewhat close.