3 ms·

r/LocalLLaMA: 5 X A100 Setup Complete

• qwertox 3y ago
  > 5 x A100 40GB running at 450w each Dedicated 4 port PCIE Switch PCIE extenders going to 4 units Other unit attached via sff8654 4i port ( the small socket next to fan ) 1.5M SFF8654 8i cables going to PCIE Retimer > The GPU setup has its own separate power supply. Whole thing runs around 200w whilst idling ( about £1.20 elec cost per day ). Added benefit that the setup allows for hot plug PCIE which means only need to power if want to use, and don’t need to reboot. > P2P RDMA enabled allowing all GPUs to directly communicate with each other. > So far biggest stress test has been Goliath at 8bit GGUF, which weirdly outperforms EXL2 6bit model. Not sure if GGUF is making better use of p2p transfers but I did max out the build config options when compiling ( increase batch size, x, y ). 8 bit GGUF gave ~12 tokens a second and Exl2 10 tokens/s.