4 ms·
How much VRAM is needed for the 32B distillation?
by OCHackr 2y ago
How much VRAM is needed for the 32B distillation?
- jadbox 2y agoDepends on compression, I think 24gb can hold a 32B at around 3b-4b compression.
- deleted 2y ago[deleted]
- brandall10 2y agoDepends on the quant used and the context size. On a 24gb card you should be able to load about a 5 bit if you keep the context small. In general, if you're using 8bit which is virtually lossless, any dense model will require roughly the same amount as the number of params w/ a small context, and a bit more as you increase context.
- buyucu 2y agoI had no problems running the 32b at q4 quantization with 24GB of ram.