5 ms·
I haven't run the code, but... how is this even possible? I've done PyTorch profiling on QLoRA Llama-2-70B fine tunes, and the runtime is dominated by the large
by apsec112 3y ago
I haven't run the code, but... how is this even possible? I've done PyTorch profiling on QLoRA Llama-2-70B fine tunes, and the runtime is dominated by the large matrix multiplies in the MLP layers, plus a bit for attention. Under the hood, this repo calls the same torch.matmul() for MLP and flash_attn_func() for attention as HuggingFace does. So how can it be that much faster? They have a few Triton kernels, but there appears to be no Triton on MLP or attention and that's most of the bottleneck.
- TheGeminon 3y agoThere is a more detailed explanation at https://unsloth.ai/introducing https://unsloth.ai/introducing
- apsec112 3y agoThat... doesn't really explain how they can get such a high number? Standard FLOP efficiency on fine-tuning big models is like 30-40%. How can you get 750%?
- danielhanchen 3y agoHey! Great question! That's what I'm confused about as well! So in GPUs the goal is to saturate the GPU with matrix multiplies instead of data movement. I'll write a more detailed blog but approximately: 1. Flash Attention v2 reduces the time taken by 17% or so 2. RoPE Triton kernels: -7.1% 3. RMS Layernorm in Triton: -3.1% 4. Cross Entropy in Triton: -1% 5. Manual autograd for MLP: -4% 6. Manual QKV autograd: -2% 7. Manual O autograd: -2% 8. Smart cache evictions and reduced data duplications etc: -30% 9. And other tricks in the Max and Pro versions makes it 30x faster You can see it's just tricks in each step, which accumulate together to make to go faster. I'll write up a blog post to detail it all in the future!!!
- demosthanos 3y ago> And other tricks in the Max and Pro versions makes it 30x faster This feels like the collecting underpants meme. Phase 1: Get to the same performance as other methods. Phase 2: ???. Phase 3: Now you're at 750%! You may or may not actually have succeeded at what you claim to, but you're not being very persuasive. I realize that you're trying to turn these tricks into a profit and revealing them would destroy that possibility, but you're going to have a really hard time persuading people to pay for a product that does something that enormous teams of PhDs at BigTech haven't been able to pull off on the basis of "trust me".
- danielhanchen 3y agoI agree fully - what do you suggest then? OSS the entire code base and using AGPL3? I tried that with https://github.com/danielhanchen/hyperlearn https://github.com/danielhanchen/hyperlearn to no avail - we couldn't even monetize it at all, so I just OSSed everything. I listed all the research articles and methods in Hyperlearn which in the end were gobbled up by other packages. We still have to cover life expenses and stuff sadly as a startup. Do you have any suggestions how we could go about this? We thought maybe an actual training / inference platform, and not even OSSing any code, but we decided against this, so we OSSed some code. Any suggestions are welcome!
- wsxiaoys 3y agoWow, this is a great topic. I don't really have specific suggestions, but I'd like to contribute some thoughts on the matter. Monetizing anything isn't inherently problematic; the challenge lies in defining what should be paid for and what should be offered for free. In the realm of open-source products and SaaS, the common practice is to provide free self-hosting options while charging for cloud hosting or enterprise-specific features, such as access control and authentication integrations. However, the landscape becomes significantly more challenging for LLMOps (assuming you are still focusing on training as a major aspect of your business, which can be categorized as LLMOps). Historically, there haven't been many success stories in this area (with exceptions like wand.ai, which focusing on tracking experiments). I believe this difficulty arises from the largely ad-hoc nature of training and fine-tuning processes, making standardization a challenge, coupled with the infrequency of these tasks. That being said, training/finetuning is a valuable technique. However, transforming it into a company that offers products is really challenging. Successful examples in this realm typically depend heavily on solution customization or consulting-oriented business models.
- deleted 3y ago[deleted]
- WhitneyLand 3y agoThey say it’s due a custom optimized version of autograd which is sort of a key calculus component. They also mention simple things like function inlining or memory optimizations. It seems plausible these things could be optimized. Whatever advantage they have I don’t see how they would be able to to keep it for long as part of their closed source “pro” version. If it’s low hanging fruit the open source equivalents are bound to snipe them before long.
- danielhanchen 3y agoThere's more as well!! I agree hypothetically in a few years people will catch up - we're well aware of that! But we made this into a startup, and so this was just one product we were releasing! We're going to be making many AI products in the coming weeks!
- WhitneyLand 3y agoThat sounds really great, we need all the advancement we can get in this field. Best of luck and all success to you!
- danielhanchen 3y agoThanks a bunch!! Super appreciate it!
- bradfox2 3y agoThese are significant claims locked behind a paid 'pro' version. Red flags.
- danielhanchen 3y agoSorry about that - I'm super new to pricing and stuff so it might seem off since I'm literally making the plans with my bro as we go along. If you don't believe the timings, I was the author of Hyperlearn https://github.com/danielhanchen/hyperlearn https://github.com/danielhanchen/hyperlearn which makes ML faster - I also listed the papers which cite the algos. I also used to work at NVIDIA making TSNE 2000x faster on GPUs and some other algos like Randomized SVD, sparse matrix multiplies etc. If you have any suggestions on a more appropriate pricing strategy - I'm all ears!! I really don't know much about pricing and the open core model, so I'm making stuff up literally.
- bradfox2 3y agoAre the 30x claims comparing full update training vs qlora/4bit weights?
- danielhanchen 3y agoSorry about the confusion - the 30X is comparing QLora with QLora - all benchmarking is QLoRa bsz=2, ga=4
- MacsHeadroom 3y agoPro says single GPU some places and multi-GPU others. I really hope it is multi-GPU because basically every enthusiast making finetunes is using 2-4 3090s locally or renting 8x GPU boxes on Vast for finetuning. If multi-GPU is enterprise only you will miss out on basically the largest customer segment as far as I can tell.
- danielhanchen 3y agoApologies on the confusion - I'm still trying to flesh stuff out on the differentating factors with my bro - I'll update it once we're fully sure - sorry again!