5 ms·

It could completely depend on what size model your are training the topology, it may not make a difference for you, but for reference- See here [0] about just

It could completely depend on what size model your are training the topology, it may not make a difference for you, but for reference- See here [0] about just the difference between having NV-link between cards or not, and the 23% increase in training speed, and the note that the peak bandwidth between 2x 3090s with the link is a peak of 112.5 GB/sec. [0]: https://huggingface.co/docs/transformers/v4.31.0/en/perf_hardware#nvlink https://huggingface.co/docs/transformers/v4.31.0/en/perf_har... Now look at PCIe 3.0 speeds, which would be what any two cards talking to each other would need to use thru your risers- only 15.754 GB/s on x16 and only 7.877 GB/s if you are on a x8 riser. For some non-ML things that I use GPU's for (CFD), the interconnect / memory access bandwidth is the bottleneck, and the simulation time literally scales near linearly with the PCIe bandwidth I have between cpu lanes and the cards.

jacquesm 3y ago
Hm, that makes me wonder whether vast.ai does some kind of testing prior to dropping particular workloads on a machine so that this can be done without configuration. But there doesn't seem to be a financial incentive to suppliers of GPU capacity to provide for good interconnects.
- mk_stjames 3y ago
  vast.ai does denote GPU's as either PCIe or SXM (SXM for the A100's and now H100's up there). The SXM bandwidth between GPU's is the full n-way NV-link, the best you can get; They also have a little stat that lists 'per-GPU' bandwidth in GB/s and what PCIe version and speed is being used. So they must run some tests beforehand to gauge this. When I look on there now it varys setup to setup. I see people running quad 4090's on PCIe4.0 x16 with 24GB/s bandwidth between them, some people running x8, some on PCIe3.0 with 11 GB/s, even someone with quad 3090's but all on PCIe 2.0 x1 slots with the bandwidth reading 0.3GB/s!!! (likely an old mining rig with those x1 slots)
  - jacquesm 3y ago
    Do they get more money per instance or is that simply a matter of seeing more utilization?
    - mk_stjames 3y ago
      I've never rented on there so I'm not sure. It seems like the prices are set by the sellers but with some guidance from their performance tool, likely? The have a overall "DLperf" deep learning performance score which is an performance metric and it seems like if you want to get users you set your price per hour at something competitive in line with the market given that metric. For instance, the guy with the quad 3090s on x1 slots actually has an hourly price set HIGHER than everyone else... even with horrible DLPerf score. No one is using that, ever.
      - jacquesm 3y ago
        Interesting... I should try some workloads to see what it can do in practice. Beats having a whole bunch of fans in my room and it seems to be cheaper than the other cloud GPU providers.

jacquesm 3y ago

Hm, that makes me wonder whether vast.ai does some kind of testing prior to dropping particular workloads on a machine so that this can be done without configuration. But there doesn't seem to be a financial incentive to suppliers of GPU capacity to provide for good interconnects.

mk_stjames 3y ago
vast.ai does denote GPU's as either PCIe or SXM (SXM for the A100's and now H100's up there). The SXM bandwidth between GPU's is the full n-way NV-link, the best you can get; They also have a little stat that lists 'per-GPU' bandwidth in GB/s and what PCIe version and speed is being used. So they must run some tests beforehand to gauge this. When I look on there now it varys setup to setup. I see people running quad 4090's on PCIe4.0 x16 with 24GB/s bandwidth between them, some people running x8, some on PCIe3.0 with 11 GB/s, even someone with quad 3090's but all on PCIe 2.0 x1 slots with the bandwidth reading 0.3GB/s!!! (likely an old mining rig with those x1 slots)
- jacquesm 3y ago
  Do they get more money per instance or is that simply a matter of seeing more utilization?
  - mk_stjames 3y ago
    I've never rented on there so I'm not sure. It seems like the prices are set by the sellers but with some guidance from their performance tool, likely? The have a overall "DLperf" deep learning performance score which is an performance metric and it seems like if you want to get users you set your price per hour at something competitive in line with the market given that metric. For instance, the guy with the quad 3090s on x1 slots actually has an hourly price set HIGHER than everyone else... even with horrible DLPerf score. No one is using that, ever.
    - jacquesm 3y ago
      Interesting... I should try some workloads to see what it can do in practice. Beats having a whole bunch of fans in my room and it seems to be cheaper than the other cloud GPU providers.