3 ms·
Yeah I am using the default training script with int8 quantisation. It uses peft with lora but this still requires 26gb
by fbodz 4y ago
Yeah I am using the default training script with int8 quantisation. It uses peft with lora but this still requires 26gb
- freeqaz 4y agoIs it possible to offload some layers to CPU and still train in a reasonable amount of time?
- generalizations 4y agoThere’s also that pruning tool that was on hn in the last couple weeks. It seemed to work really well on the larger models, and could reduce size by 30-50%
- int_19h 4y agoI'm not sure about this model specifically, but training with 4-bit quantization has been a thing with LLaMA for a while now, although the setup involves manual hacks of various libraries.