3 ms·
13B in 4bit runs on your average 8GB VRAM consumer video card at fast enough speeds. The GP's info is outdated. Apparently they don't realize how quickly LLaMA
by MacsHeadroom 4y ago
13B in 4bit runs on your average 8GB VRAM consumer video card at fast enough speeds.
The GP's info is outdated. Apparently they don't realize how quickly LLaMA is being developed on. It's just like the early days of SD.
- smy20011 4y agoMaybe out-dated by 1 day LOL.
- zamadatix 4y agollama.cpp runs on the CPU anyways, the whole conversation is on a false premise. The only hard thing about running it on consumer hardware is the quantization step (where the largest RAM requirements come in) and that's not even a technical problem since you could just redistribute the quantized version if it weren't for the license of the model.