3 ms·
Hard to read on mobile, but if you don’t mind me asking, what’s the catch? Is there a penalty to training faster and less VRAM?
by syntaxing 2y ago
Hard to read on mobile, but if you don’t mind me asking, what’s the catch? Is there a penalty to training faster and less VRAM?
- danielhanchen 2y agoNo catch at all!! There's 0 approximations, so everything is exact! We just have a custom backprop engine, rewrite everything into OpenAI's Triton language, and do all the differentiations and maths ourselves :) Unsloth can fit 4x longer context windows than HF + Flash Attention 2 as well with our latest long context update, so 30% less VRAM use, at the expense of slightly +1.9% overhead.