4 ms·
~7.5GB - it'll be the same as running inference with a full context. That's for 4-bit quantization, the 8-bit quantization uses more RAM than my 3080 has...
by eulers_secret 3y ago
~7.5GB - it'll be the same as running inference with a full context. That's for 4-bit quantization, the 8-bit quantization uses more RAM than my 3080 has...
- seydor 3y agoI wonder how much it would take to train the 4b 13B
- MacsHeadroom 3y agoAbout 15GB training it in the webui. If you use https://github.com/johnsmith0031/alpaca_lora_4bit https://github.com/johnsmith0031/alpaca_lora_4bit then 30B only needs 24GB, and works on a single 3090 or $200 P40.