3 ms·
Reading this gave me flashbacks to the 80s, when tinkerers tried to move utilities into the upper- and extended memory area to free up precious conventional mem
by esafak 1y ago
Reading this gave me flashbacks to the 80s, when tinkerers tried to move utilities into the upper- and extended memory area to free up precious conventional memory, 640KB of which we were told ought to have been "enough for anyone". All this because we were saddled with a 16-bit OS. This is not an LLM problem -- 32GB of memory is peanuts in 2025 -- this is an Intel and AMD problem.
- zamadatix 1y agoAs the article highlights the problem is really twofold. You need enough VRAM to load the model at all but there also needs to be enough bandwidth that accessing all of that memory is fast enough to be worthwhile. It'd be "easy" to slap 2 TB of "slow" DDR5 onto a GPU but it wouldn't perform much better than a high core count CPU running LLMs with the same memory.