3 ms·
If you're planning to do local GenAI, the biggest factor is VRAM, not just GPU power. Even smaller LLMs (e.g., 13B-30B) require a lot of VRAM to run efficiently
by memorydial 2y ago
If you're planning to do local GenAI, the biggest factor is VRAM, not just GPU power. Even smaller LLMs (e.g., 13B-30B) require a lot of VRAM to run efficiently, and 8GB is borderline for anything beyond toy use. If you’re serious about local GenAI, a second-hand RTX 3090 (24GB VRAM) or a 4070 Ti (16GB VRAM) would be a much better investment than an NPU, which currently isn't widely supported for local inference.