4 ms·

> However today, with the latest, surprisingly good reasoning models like QwQ-32B using up thousands or tens of thousands of tokens in their replies, performanc

> However today, with the latest, surprisingly good reasoning models like QwQ-32B using up thousands or tens of thousands of tokens in their replies, performance is getting more important than previously and these systems (Macs and even RTX 3090s) might fall out of favor, because waiting for a finished reply will take several minutes or even tens of minutes. 3090 is the last gen to have proper nvlink support, which is supported for LLM inference in some frameworks. Would 1x 5090 be faster than 2x 3090?

whalesalad 2y ago
1x5090 not enough vram
reginald78 2y ago
I was browsing r/localllm recently and was surprised that some people were purchasing two 3060 12GB and using them in tandem somehow. I actually didn't think this would work at all without nvlink.
- magicalhippo 2y ago
  You just copy data over PCIe, however it is slower than nvlink. There are several ways to utilize multiple GPUs, the main contenders as I understand it is pipelined and so-called tensor parallelism. Think of it as slicing a loaf of bread in regular slices vs along its longer axis. The former can have higher peak throughput while the latter can have lower latency, though it depends on the details[1]. [1]: https://blog.squeezebits.com/vllm-vs-tensorrtllm-9-parallelism-strategies-36310 https://blog.squeezebits.com/vllm-vs-tensorrtllm-9-paralleli...
  - ai-christianson 2y ago
    > You just copy data over PCIe AFAIK this only happens directly over PCIe if using hacked drivers with p2p enabled (I think tinygrad/tinybox provided these drivers initially.) Otherwise, data goes through system bus/CPU first. `nvidia-smi topo -m` will show how the GPUs are connected.
- ai-christianson 2y ago
  > I actually didn't think this would work at all without nvlink. It will work, but with less memory bandwidth. If using hacked drivers with p2p enabled, it will depend on the PCIe topology. Otherwise, the data will take a longer path. Depending on the model, it may not be that big of a hit in performance. Personally, I have seen higher perf on nvlinked GPUs and models that can fit on 2x 3090 24gbs.
  - reginald78 2y ago
    Yes, no one seemed to be bragging about much of a performance boost. I think they were happy being able to run models (at any speed) that needed 24GB without having to buy a 3090.

whalesalad 2y ago

1x5090 not enough vram

reginald78 2y ago

I was browsing r/localllm recently and was surprised that some people were purchasing two 3060 12GB and using them in tandem somehow. I actually didn't think this would work at all without nvlink.

magicalhippo 2y ago
You just copy data over PCIe, however it is slower than nvlink. There are several ways to utilize multiple GPUs, the main contenders as I understand it is pipelined and so-called tensor parallelism. Think of it as slicing a loaf of bread in regular slices vs along its longer axis. The former can have higher peak throughput while the latter can have lower latency, though it depends on the details[1]. [1]: https://blog.squeezebits.com/vllm-vs-tensorrtllm-9-parallelism-strategies-36310 https://blog.squeezebits.com/vllm-vs-tensorrtllm-9-paralleli...
- ai-christianson 2y ago
  > You just copy data over PCIe AFAIK this only happens directly over PCIe if using hacked drivers with p2p enabled (I think tinygrad/tinybox provided these drivers initially.) Otherwise, data goes through system bus/CPU first. `nvidia-smi topo -m` will show how the GPUs are connected.
ai-christianson 2y ago
> I actually didn't think this would work at all without nvlink. It will work, but with less memory bandwidth. If using hacked drivers with p2p enabled, it will depend on the PCIe topology. Otherwise, the data will take a longer path. Depending on the model, it may not be that big of a hit in performance. Personally, I have seen higher perf on nvlinked GPUs and models that can fit on 2x 3090 24gbs.
- reginald78 2y ago
  Yes, no one seemed to be bragging about much of a performance boost. I think they were happy being able to run models (at any speed) that needed 24GB without having to buy a 3090.