3 ms·
5950x is a CPU model. Integer-quantized models are generally run with CPU inference. For the larger models the problem then becomes generation time per token.
by sbierwagen 4y ago
5950x is a CPU model. Integer-quantized models are generally run with CPU inference. For the larger models the problem then becomes generation time per token.
- int_19h 4y agoQuantized models are used aplenty with GPUs as well - 4-bit quantization is the only way you can squeeze llama-30b into 24Gb of VRAM (i.e. RTX 3090 or 4090). In fact, I would say that, at this point, most people running LLaMA locally are likely using 4-bit quantization regardless of model size and hardware, just to get the most out of the latter.
- whimsicalism 4y agoMost people running llama locally are doing CPU inference, period.
- barbariangrunge 4y agoIf your desktop had 256gb of ram, could you train a far larger model? Some motherboards support that