7 ms·
As the article highlights the problem is really twofold. You need enough VRAM to load the model at all but there also needs to be enough bandwidth that accessin
by zamadatix 1y ago
As the article highlights the problem is really twofold. You need enough VRAM to load the model at all but there also needs to be enough bandwidth that accessing all of that memory is fast enough to be worthwhile. It'd be "easy" to slap 2 TB of "slow" DDR5 onto a GPU but it wouldn't perform much better than a high core count CPU running LLMs with the same memory.