3 ms·
Here's my report running several different models on a dual Xeon with 256 GB of DDR4 and no GPU. https://gist.github.com/hparadiz/f3596d00a62d8ebb2dadcc46ee582
by hparadiz 3mo ago
Here's my report running several different models on a dual Xeon with 256 GB of DDR4 and no GPU.
https://gist.github.com/hparadiz/f3596d00a62d8ebb2dadcc46ee5822c7 https://gist.github.com/hparadiz/f3596d00a62d8ebb2dadcc46ee5...
- neomindryan 3mo agoThank you for sharing!
- puzzlingcaptcha 3mo agoHave you tried with a single CPU to get rid of the NUMA penalty? I understand this likely means halving the memory but I am interested in how much of a difference it makes
- trollbridge 3mo agoI have (192GB machine with two CPUs), pretty much does the trick. It just runs some small models used for embedding, etc. and has those on one CPU / memory node and all the Docker containers on the other one.c
- ryandrake 3mo agoI have a dual xeon also, same as OP: Ivy Bridge + 128GB DRAM, and was never really able to get decent LLM performance out of it. So I ended up biting the bullet and adding a "budget tier" A4000 20GB GPU. Too bad all my DRAM is wasted now--not sure if there is a way to take advantage of lots of DRAM once you move over to having inference happening on the GPU.
- puzzlingcaptcha 3mo agoHave you tried putting the KV cache on the GPU and running inference from RAM? From what I gather, prompt processing is particularly painful using RAM alone.
- trollbridge 3mo agoI haven't, since the machine has almost no GPU to speak of. It's an old machine, although in theory it would make sense to pop an R9700 in there and load a bunch of models into RAM and swap them in and out.
- edg5000 3mo agoGreat speeds. So a small model on a lot of slow ram is fast. Now I wonder how larger models run on that thing.