2 ms·
Yes and I can fit 70b in system RAM on my linux box. But parent was talking about discrete GPUs, and shared memory architectures like Intel's integrated GPUs or
by do_not_redeem 2y ago
Yes and I can fit 70b in system RAM on my linux box. But parent was talking about discrete GPUs, and shared memory architectures like Intel's integrated GPUs or Apple's unified memory will never perform as well as dedicated GPU memory.
- zozbot234 2y ago> shared memory architectures like Intel's integrated GPUs or Apple's unified memory can't compare to dedicated GPU memory. It depends. For the initial prompt+context processing, iGPUs (and NPUs) are still a win because you're compute bound at that stage. For token generation, CPU is typically faster than iGPU (or even plausibly NPU) for local quantized models because it can make better use of all available memory bandwidth, with no wasteful padding reducing your usable throughput.
- alwayslikethis 2y agoIt is slower, but I would say fairly usable for just LLM inference (still much faster than you can read). I'd say it's more of a matter of control. You can be reasonably sure that your linux box is free from ordinary forms of telemetry, which is hard to do with a Mac.