3 ms·
A RTX 3090 is more than enough for 7B LLMs. With 4-bit quantization, you can run inference with an even larger LLM using a 24GB GPU. If you're using remote API
by dmezzetti 2y ago
A RTX 3090 is more than enough for 7B LLMs. With 4-bit quantization, you can run inference with an even larger LLM using a 24GB GPU.
If you're using remote API services, you might be able to just use a CPU.