3 ms·
MoE is pretty enabling after you've spent all the extra $$$$ to stuff your server CPU memory channels with ram so it's possible to run at all. But it's still sp
by superkuh 2y ago
MoE is pretty enabling after you've spent all the extra $$$$ to stuff your server CPU memory channels with ram so it's possible to run at all. But it's still spending a lot of money which makes this a lot less novel or interesting than "just on 1~2 Arc A770" implies. Especially for the marginal performance that even 8-12 channels of CPU memory bandwidth gets you.
- utopcell 2y agoIs this amount of RAM really that expensive? 6x 64GiB DDR4 DIMMs are < $1,000.
- utopcell 2y agoActually, 384GiB is already <$400 [1]. [1] https://www.amazon.com/NEMIX-RAM-DDR4-2666MHz-PC4-21300-Reduced/dp/B0BJFN1JYS https://www.amazon.com/NEMIX-RAM-DDR4-2666MHz-PC4-21300-Redu...
- superkuh 2y agoA slow-end DDR4 speed older generation Xeon system is unlikely to be used by Intel for this benchmark. It's far more likely they used an expensive DDR5 modern Xeon with as many memory channels as they could. Single user LLM inference is memory bandwidth bottlenecked. I just can't see Intel using old/deprecated hardware. And if someone not Intel were to build a Xeon DDR4 system it wouldn't reach the DDR5 tokens/s speeds reported here. The reason they used a Xeon is memory channels. Non-server CPUs only have 2 but modern Xeons have 8 to 12 depending on generation/type. And the Xeons with the most are the most $$$$ and it ends up cheaper to just get a GPU or dedicated accelerator.