4 ms·
It’s time for me to upgrade the main server in my home lab and I’m thinking about which machine should I have. What kind of hardware you’d need to run the 397B
by rbanffy 2mo ago
It’s time for me to upgrade the main server in my home lab and I’m thinking about which machine should I have.
What kind of hardware you’d need to run the 397B one at an acceptable speed?
- colingauvin 2mo ago3 DGX Sparks with 400 Gb interconnects in a loop.
- rbanffy 2mo agoI was looking for something more general-purpose, such as a decommissioned quad-socket x86 with 80 or more AVX512 cores and a terabyte of RAM, which I suspect will be a strong limiting factor. The possibility of adding some older Nvidia accelerators is interesting as well even though they won’t be able to hold the whole model in HBM and will need to stream it over PCIe 4, which sucks. That way, when not running inference, the machine can easily host VMs for other experiments and general housekeeping functions.