13 ms·
Show HN: 80% faster, 50% less memory, 0% loss of accuracy Llama finetuning
Hi HN! I'm just sharing a project I've been working on during the LLM Efficiency Challenge - you can now finetune Llama with QLoRA 5x faster than Huggingface's original implementation on your own local GPU. Some highlights:
1. Manual autograd engine - hand derived backprop steps.
2. QLoRA / LoRA 80% faster, 50% less memory.
3. All kernels written in OpenAI's Triton language.
4. 0% loss in accuracy - no approximation methods - all exact.
5. No change of hardware necessary. Supports NVIDIA GPUs since 2018+. CUDA 7.5+.
6. Flash Attention support via Xformers.
7. Supports 4bit and 16bit LoRA finetuning.
8. Train Slim Orca fully locally in 260 hours from 1301 hours (5x faster).
9. Open source version trains 5x faster or you can check out Unsloth Pro and Max codepaths for 30x faster training!
https://www.reddit.com/r/LocalLLaMA/comments/188197j/80_faster_50_less_memory_0_accuracy_loss_llama/ https://www.reddit.com/r/LocalLLaMA/comments/188197j/80_fast... has more info about Unsloth!
Hopefully you can try it out! Wrote a blog post at https://unsloth.ai/introducing https://unsloth.ai/introducing if you want to learn more about our manual hand derived backprop or Triton kernels and stuff! Thanks once again!
- danielhanchen 3y agoFor those interested - just released a new blog post on all our optimizations. There are also 59 fully reproducible benchmarks! https://unsloth.ai/blog/mistral-benchmark https://unsloth.ai/blog/mistral-benchmark
- deleted 3y ago[deleted]
- kristopolous 3y agoIt'd be great to have a chronicle of all these efforts. I lost track of the variations quite a long time ago. It'd be quite a lift unless we're just willing to just accept the self reported metrics as golden. And even then, they're always qualified by hardware and usage scope. Making it good enough to be useful is the hard part. CI/CD pipeline with a bunch of machine configurations and benchmarks along with a reasonable way to communicate them... If anyone's up for it you'd legitimately become indispensable.
- danielhanchen 3y agoHey there! Yes that's exactly what I thought! I was writing up a blog post at https://colab.research.google.com/drive/1AOuhMVILE06mD-Go7-RuN47Dg35t1wuK?usp=sharing https://colab.research.google.com/drive/1AOuhMVILE06mD-Go7-R... which step by step shows every change I did plus gave timings / mem savings. I'll post it once its completed if you're interested!
- kristopolous 3y agoThanks. There's been a few substantial and laudable efforts which are much appreciated but what I'm suggesting is an actual continuous infrastructure, like how those benchmarking sites have software for people to run on their machines that phone home so that people who make new benchmarks or new variations can submit them and refine the results. For instance, are any of your prompting tests in say, Korean? What about winograd schema challenges in languages other than English? Japanese for instance, comes with its own unique set of context ambiguities that do not appear in English. I'm sure dozens of languages are similar. It'd be nice to have user contributable tests to cover the breadth of use cases here. A great optimization that moves a score let's say from 95% -> 5% on "winograd-persian" may be fine or may be a show stopper, depends on what you care about. That's why it's gotta be normalized, future-proof, and crowdsourced.
- danielhanchen 3y agoOhhh interesting! So like ML Perf or Faster CPython https://github.com/faster-cpython/benchmarking-public https://github.com/faster-cpython/benchmarking-public?
- kristopolous 3y agoSure. This thing I propose might already exist in a nascent form. I haven't checked in a few months and the way things are moving now...
- 3y ago
- apsec112 3y agoI haven't run the code, but... how is this even possible? I've done PyTorch profiling on QLoRA Llama-2-70B fine tunes, and the runtime is dominated by the large matrix multiplies in the MLP layers, plus a bit for attention. Under the hood, this repo calls the same torch.matmul() for MLP and flash_attn_func() for attention as HuggingFace does. So how can it be that much faster? They have a few Triton kernels, but there appears to be no Triton on MLP or attention and that's most of the bottleneck.
- TheGeminon 3y agoThere is a more detailed explanation at https://unsloth.ai/introducing https://unsloth.ai/introducing
- apsec112 3y agoThat... doesn't really explain how they can get such a high number? Standard FLOP efficiency on fine-tuning big models is like 30-40%. How can you get 750%?
- danielhanchen 3y agoHey! Great question! That's what I'm confused about as well! So in GPUs the goal is to saturate the GPU with matrix multiplies instead of data movement. I'll write a more detailed blog but approximately: 1. Flash Attention v2 reduces the time taken by 17% or so 2. RoPE Triton kernels: -7.1% 3. RMS Layernorm in Triton: -3.1% 4. Cross Entropy in Triton: -1% 5. Manual autograd for MLP: -4% 6. Manual QKV autograd: -2% 7. Manual O autograd: -2% 8. Smart cache evictions and reduced data duplications etc: -30% 9. And other tricks in the Max and Pro versions makes it 30x faster You can see it's just tricks in each step, which accumulate together to make to go faster. I'll write up a blog post to detail it all in the future!!!
- demosthanos 3y ago> And other tricks in the Max and Pro versions makes it 30x faster This feels like the collecting underpants meme. Phase 1: Get to the same performance as other methods. Phase 2: ???. Phase 3: Now you're at 750%! You may or may not actually have succeeded at what you claim to, but you're not being very persuasive. I realize that you're trying to turn these tricks into a profit and revealing them would destroy that possibility, but you're going to have a really hard time persuading people to pay for a product that does something that enormous teams of PhDs at BigTech haven't been able to pull off on the basis of "trust me".
- foota 3y agoThere are a number of papers on optimizing xor chains for cauchy reed Solomon computation, it sounds like a superficially similar problem?
- danielhanchen 3y agoHey! Oh noo we don't do any XOR chain optimizations or Cauchy Reed Solomon computations - I had to wikipedia it and learn about them - mroe reading to do! It's mainly trying to reduce data movement, maximize FLOP utilization via matrix multiplies, and reducing FLOPs via manually deriving backpropagation equations + more!
- foota 3y agoI realize they're different, but it seems like there are some similarities. In both, you have some chain of operations you need to do, and you want to minimize their cost and optimize for cache benefits etc., This is a pretty good overview paper: https://arxiv.org/pdf/2108.02692.pdf https://arxiv.org/pdf/2108.02692.pdf (they claim to be better than the rest -- I haven't evaluated their claims in practice yet) The techniques are mostly about ordering and common subexpression elimination for XOR operations and choosing "better" matrices to do the computation with. CRS codes can be used with many different matrices, but it turns out some are better than others for effiency (XOR operations are encoded by 1s in the CRS matrix, so if you choose a CRS matrix with fewer of them while meeting the requirements, you can do less operations and get the same results).
- danielhanchen 3y agoOHHH yes yes - sorry you're correct - apologies I'm not too familiar with this - thanks for the paper I'll read it through!! Yep so the OSS version does have the reordering of operations - I'll write a blog post about it in the coming days!! We also have other tricks up our sleeve to make it go even faster!!
- foota 3y ago
- graphe 3y agoSomewhat related: is it worth it to use a P100 or P40? I was gonna get one but seems like pascal isn't being supported by more and more projects.
- danielhanchen 3y agoP100 - oh my so Xformers for Flash Attention I think does support it, but Triton supports Compute Capability 7.0+, whilst a P100 is 6.0 :( So technically the code can run, but I'll have to edit it to remove the Triton changes.
- graphe 3y agoThank you! I generally don't see as much support for them and needs to use old versions of it was supported. This trend is happening more and more, it's unfortunate but it seems like we need to use a newer GPU now.
- danielhanchen 3y agoYe unfortunately :( I tried my best to support all GPUs from 2018 onwards - for eg sadly Flash Attention v2 is only Ampere+, whilst Xformers is nicely Tesla T4+
- TuringNYC 3y agowow i wish i could do this with all my M1 pro max neural cores
- danielhanchen 3y agoM1 support is probably coming in the future if there is enough support - could you make an Issue at https://github.com/unslothai/unsloth https://github.com/unslothai/unsloth - that would be much appreciated! I think there are a few Redditors from /r/localllama who also requested this, but for now first priority is getting Mistral support!!
- xrd 3y agoOK, this is my favorite .ai domain. And, the logo snaps!
- danielhanchen 3y agoThanks!! My brother came up with the name!! "Un-sloth" where sloth is the slow cute animal, and it also means lazy and slow, and "un" reverses it!
- juunpp 3y agoSSL_ERROR_NO_CYPHER_OVERLAP when checking your site.
- danielhanchen 3y agoDo you mean http://www.unsloth.ai/ http://www.unsloth.ai/ doesn't work? My brother is working on it as we speak! Edit: it seems like http://unsloth.ai http://unsloth.ai works - we're talking with our web hosting people to fix it
- deleted 3y ago[deleted]
- gentleman11 3y agoIsnt llama proprietary? Why not dine tune one of the truly open models instead?
- danielhanchen 3y agoLlama is generally OSS, except for some gating to large companies - but as a first try I made it work for Llama since the architecture is replicated in other models like Mistral or Yi.
- selfhoster11 3y agoThere's a tremendous open source ecosystem in place for Llama already. This is a huge deal for beginners and model integrators alike. Not to mention that other than beyond 7B models, your options drastically taper off. Mistral and most open base model projects only have models available up to 7 billion parameters or so, which is quite tiny if you are used to the relative ease of using un-finetuned GPT-4 to carry out your tasks of choice. So what options are there? Falcon 40B and MPT-30B - sure, the weights license is all right, but many in the community have reservations about those models' underperformance, as you can get much more bang for your buck with newer models, in an equal number of weights in a newer base model. Subjectively speaking, it could be a waste of time. Falcon 180B and Yi 34B weights are both issued under non-free licenses, just like Llama 2. Is Llama 2 proprietary? For the vast majority of people, for the vast majority of purposes, no. I'm not a lawyer, but I think that Meta would be quite unlikely to do more than cut off your HuggingFace access to the repo where new models will be distributed.
- danielhanchen 3y agoThanks to Meta for open sourcing Llama!!! Ye sadly the HFF leaderboard doesn't have a high opinion for Falcon. MPT's long context via Alibi did work, just less so when compared to RoPE scaling. All thanks to Llama - the LLM community is now vibrant and alive!
- yunohn 3y agoThis seems very interesting, but I’m very confused why you have gated the /max/imum speedup version for enterprise-only? It would make more sense to have only the Free and Paid plans differing in performance, and Enterprise getting support/etc.
- danielhanchen 3y agoGood point - we thought about this - we're still figuring out the pricing as we go along - so all suggestions are welcome! I'm all very new to this, so I'm literally making stuff as I go along!! Apologies!
- yunohn 3y agoNo worries, just my 2c! Given the current state of AI performance, I’d imagine that those without 100s of GPUs but looking to maximise the performance of what they do have, would be a great demographic for the Paid plan.
- danielhanchen 3y agoInteresting! I did have some chats from interested people who have say 2 or 4 or 8 GPUs who are willing to somehow buy the Pro plan - the issue is now how do we price it, do we make a training platform? etc
- hirako2000 3y agoOffer a (pricy) license to existing training platform providers. It will still make save a lot on infra costs.
- danielhanchen 3y agoWhat do you think about ourselves offering a platform?
- 0x6c6f6c 3y agoI see this mentions 2018+ GPUs, but I'm curious why this wouldn't work on say a 1080Ti. My cursory look at the hardware specs shows this has support for CUDA8+ and this states 7.5 Would someone be able to tell me more about this?
- zargon 3y ago1080 Ti has CUDA capability version 6.1.
- danielhanchen 3y agoSorry about 1080 Ti - sadly Triton and Xformers support Cuda 7.0, so unless if OpenAI and Meta makes it support Cuda 6.0, we sadly don't support it. The main reason is because from Turing, Tensor Cores got provided, and so the matrix matmuls are all different in Tensor Cores
- 3abiton 3y agoAny source recommendations on howto find out the minimum hw requirementsto train LLM with Xbit and YB parameters?
- danielhanchen 3y agoGreat question - I'm actually not sure - I'll probably write up some minimum requirements - I know QLoRA's paper https://arxiv.org/pdf/2305.14314.pdf https://arxiv.org/pdf/2305.14314.pdf has an approximate calculation of the weights size, but not the gradient updates
- squigz 3y agoWhat is the motivation behind open sourcing code at all if you're not going to open source it all? I don't mean this to be an asshole, as I truly want to understand the motivation.
- sbierwagen 3y agoShareware. See the open source preview, pay a million dollars for the full version.
- danielhanchen 3y agoI wouldn't characterize it exactly as "shareware", since the OSS version is fully functional, trains 5x faster on a single GPU. The paid version just makes it train 30x faster on multi GPU platforms - it's more of an open core approach.
- ShamelessC 3y agoThis is, I believe, part of the Y Combinator playbook. You want to get as much growth as possible using any means you can think of then flip the pricing (or tracking) on as soon as it peaks. Alternatively you can sell your company or go public. The company here does seem less sinister however - don't think they've accepted VC investment yet? Could be wrong.
- danielhanchen 3y agoNo we don't have any funding! I'm grateful to my family to supporting us till now. We're open to VC investment now, except we truly believe in bootstrapping it - we have tonnes of other products in the pipeline like a Recession predictor, a Data Science Consultant, our own trained chatbot via DPO and our fully cleaned datasets and more!
- ShamelessC 3y agoRefreshing to see, especially on this forum! Best of luck.
- 29athrowaway 3y ago[flagged]
- danielhanchen 3y agoYes that does sound like a scary dystopian future :( My bro and I don't have any evil intentions, so our first priority is to release stuff carefully and safely!
- neilmovva 3y agopromising results, excited to try it out! question on the perf benchmarks: why do all the results with 2 GPUs & DDP take longer than the single GPU case? Both benchmarks do the same amount of work, one training epoch, so this negative scaling is surprising.
- danielhanchen 3y agoSo there's 2 main reasons: 1. DDP itself has an overhead since it has to synchronize gradients at each training step since GPU0 and GPU1 has to give gradients to GPU0. 2. Huggingface seems to not be optimized well for DDP mainly due to inefficient data movement - we fixed that - interestingly - even on 1 GPU it's faster.
- neilmovva 3y agoI agree that synchronization causes overhead, so 2x GPUs won't achieve the ideal 0.5x total runtime. But here, taking your Alpaca benchmark as an example, we are seeing 2x GPUs get 3.6x runtime with Huggingface, or 1.15x with Unsloth Max. In other words, every benchmark, in either HF or Unsloth, is slower in absolute terms when going from 1 to 2 GPUs. That makes me think something is wrong with the test. Could you share your benchmark code?
- danielhanchen 3y agoYou can refer to QLoRA's official finetuning notebook https://colab.research.google.com/drive/1VoYNfYDKcKRQRor98Zbf2-9VQTtGJ24k?usp=sharing https://colab.research.google.com/drive/1VoYNfYDKcKRQRor98Zb... for your reference!! Obviously I can't provide the code we have, but if you use the same datasets and the same settings (bsz = 2, ga = 4, max_grad_norm = 0.3, num_epochs = 1, seed = 3407, max_seq_len = 2048) you should be able to replicate it.
- aerioux 3y agoIf this technology is generalizable to more LLM architectures, y'all can start messaging venture capitalists with a demo and they can help on the rest (pricing models, customers, etc).
- danielhanchen 3y agoWorking on making it work on all archs!! Probably some sort of automatic dispatcher and automatic autograd engine!
- nikolayasdf123 3y agomonetising software tooling and libraries is notoriously hard. and amount of heat you getting here is huge. hope you find a way!
- danielhanchen 3y agoThanks!! I agree fully - I'm still trying to work it out with my brother on how we can effectively monetize it - probably products is the way to go like how OpenAI does it? But thanks a lot again!
- rightbyte 3y agoSell the patches to Zuckerberg so he can FOSS it to piss MS off.
- danielhanchen 3y agoLoll - I guess now the question is at what price? :)
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- peytoncasper 3y agoCan I suggest that you ignore all the criticism about the pricing on this thread and immediately find a sales rep or SE that has worked at an early stage DB company and begin cold calling high end customers with thousands of GPUs. B2B deals at 200-300k+ are your best bet at selling this IMO.
- danielhanchen 3y agoOh thanks for the cool tips! Super good points - I'll have to ask around sadly I'm just the engineering guy loll :(
- peytoncasper 3y agoWould be happy to answer questions as you have them.
- danielhanchen 3y agoThanks a bunch a lot for the kind support!! If I have any questions I'll definitely ask - again much appreciated!!
- naveen99 3y agoHow does this compare to PyTorch labs optimizations for Sam and llama2 ? https://github.com/pytorch-labs/segment-anything-fast https://github.com/pytorch-labs/segment-anything-fast https://github.com/pytorch-labs/gpt-fast https://github.com/pytorch-labs/gpt-fast
- danielhanchen 3y agoHey! That's for inference - our code is for training :) We do plan faster inference in the future!! Saw GPT Fast by Chillee - it is blazingly fast!