3 ms·
Not an expert but I'm pretty sure it has to do with how much VRAM you need in your GPU in order to process them efficiently. Last time I was reading about the
by riskable 3y ago
Not an expert but I'm pretty sure it has to do with how much VRAM you need in your GPU in order to process them efficiently. Last time I was reading about the sizes someone mentioned that 8B was just barely too big for their GPU.
- aidenn0 3y agoSeems like 6B would still be useful if I want to run it on my GPU without exiting firefox.
- londons_explore 3y agoKinda lame that applications can't be told "yo, your gpu buffer has now been moved back to RAM".
- selcuka 3y agoWith Apple Silicon there is no separation between GPU RAM and CPU RAM, so the whole model can be loaded into RAM and processed by GPU (Metal).
- brucethemoose2 3y agoThis is not really true anymore, as the "consumer GPU" backends have very flexible quantization. Llama.cpp has like a dozen steps between 2 and 6 bit, and exLlamav2 will literally do an arbitrary decimal bpw. It sort of matters with bigger models trying to squeeze into a server GPU, with the (currently) inflexible vLLM 4-bit quantization. I think its just a standard set by Llama.