3 ms·

Are you referring to the finetuning part? The multiple bug fixes are separate from the finetuning sections - Unsloth itself makes finetuning 2x faster and use

Are you referring to the finetuning part? The multiple bug fixes are separate from the finetuning sections - Unsloth itself makes finetuning 2x faster and use 70% less memory - the bug fixes are totally detached from finetuning - ie you can take the fixed version we uploaded at https://huggingface.co/unsloth/phi-4 https://huggingface.co/unsloth/phi-4, and use it in any framework or inference engine. Apologies I'm confused on the comment sorry. If you're questioning the credibility of the bug fixes - we fixed 8 bugs in Gemma https://x.com/danielhanchen/status/1765446273661075609 https://x.com/danielhanchen/status/1765446273661075609, multiple bugs in Llama, Mistral, Qwen, a gradient accumulation bug https://x.com/danielhanchen/status/1846235913443262891 https://x.com/danielhanchen/status/1846235913443262891 and much more

grumpopotamus 2y ago
2x faster than what?
- danielhanchen 2y ago
  Oh 2x faster and uses >70% less memory than Hugging Face + Flash Attention 2! I did a CUDA / GPU Mode talk about it here: https://www.youtube.com/watch?v=hfb_AIhDYnA https://www.youtube.com/watch?v=hfb_AIhDYnA Also to the PyTorch team here: https://www.youtube.com/watch?v=MQwryfkydc0 https://www.youtube.com/watch?v=MQwryfkydc0 and the PyTorch Conference here: https://www.youtube.com/watch?v=PdtKkc5jB4g https://www.youtube.com/watch?v=PdtKkc5jB4g
  - kouteiheika 2y ago
    > Oh 2x faster and uses >70% less memory than Hugging Face + Flash Attention 2! Is this doing the same type of fine-tuning, or are you comparing full bf16 fine-tuning in HF with 4-bit QLoRA in Unsloth (in which case it's not really an apples-to-apples comparison)? If it's the latter then do you have a comparison of the former?
    - danielhanchen 2y ago
      Oh I compared 4bit QLoRA HF+FA2 with Unsloth 4bit QLoRA. 16bit LoRA have similar boosts in performance! Full bf16 full finentuning is not yet supported, but it'll come out soon!

grumpopotamus 2y ago

2x faster than what?

danielhanchen 2y ago
Oh 2x faster and uses >70% less memory than Hugging Face + Flash Attention 2! I did a CUDA / GPU Mode talk about it here: https://www.youtube.com/watch?v=hfb_AIhDYnA https://www.youtube.com/watch?v=hfb_AIhDYnA Also to the PyTorch team here: https://www.youtube.com/watch?v=MQwryfkydc0 https://www.youtube.com/watch?v=MQwryfkydc0 and the PyTorch Conference here: https://www.youtube.com/watch?v=PdtKkc5jB4g https://www.youtube.com/watch?v=PdtKkc5jB4g
- kouteiheika 2y ago
  > Oh 2x faster and uses >70% less memory than Hugging Face + Flash Attention 2! Is this doing the same type of fine-tuning, or are you comparing full bf16 fine-tuning in HF with 4-bit QLoRA in Unsloth (in which case it's not really an apples-to-apples comparison)? If it's the latter then do you have a comparison of the former?
  - danielhanchen 2y ago
    Oh I compared 4bit QLoRA HF+FA2 with Unsloth 4bit QLoRA. 16bit LoRA have similar boosts in performance! Full bf16 full finentuning is not yet supported, but it'll come out soon!