3 ms·
I haven't found a good case/risers/etc I really like. Most the miner stuff wasn't made for PCIe 16x.
by postalrat 2y ago
I haven't found a good case/risers/etc I really like. Most the miner stuff wasn't made for PCIe 16x.
- manmal 2y agoIs that a problem? According to this, the GPUs don’t communicate that much once the weights are loaded: https://github.com/turboderp/exllama/discussions/16#discussioncomment-6245573 https://github.com/turboderp/exllama/discussions/16#discussi... > So at FP16 precision that's a grand total of 16 kB you're transmitting over the PCIe bus, once per token. If you multiply by, say, 20 tokens per second, then you're still only using like 0.1% of your PCIe bandwidth. Intra GPU memory bandwidth is very important, but I‘ve seen lots of people use just a x4 lane and they didn’t complain much.