3 ms·
I'm pretty sure anyone finetuning Lllama now on a regular basis is using https://github.com/unslothai/unsloth https://github.com/unslothai/unsloth so comparison
by nl 2y ago
I'm pretty sure anyone finetuning Lllama now on a regular basis is using https://github.com/unslothai/unsloth https://github.com/unslothai/unsloth so comparisons should be against that. The open source version is ~2x faster than default implementations. NVidia only, although the kernels are in Triton so might be portable.
- syntaxing 2y agoI remember seeing them on HN when the first started! I never understood what’s the price you pay, how did they get such a big speed up and less memory usage?
- randomcatuser 2y agoThere's previous comments, apparently the founder did a lot of math re-deriving things from scratch :) https://news.ycombinator.com/item?id=39672070 https://news.ycombinator.com/item?id=39672070 https://unsloth.ai/blog/gemma-bugs https://unsloth.ai/blog/gemma-bugs
- mistrial9 2y agonice work in gemma-bugs -- compared to plenty of research work that is a km deep in real math, this tech note is a just few python tweaks. But finding those and doing it? apparently this is useful and they did it. Easy to read (almost child-like) writeup.. thx for pointing to this.
- segmondy 2y agoThey main author used to worth Nvidia. There's a free plan, and you can pay to get multiple GPU support.
- pilooch 2y agoIndeed, a lora finetune of llama 3.1 8B works on a single 24GB GPU and takes from a few hours to a few days depending on the dataset size.