2 ms·
Are PCIe lanes really that important for ML? I've seen reports that x4 isn't any different from x16 for a single card.
by wruza 3y ago
Are PCIe lanes really that important for ML? I've seen reports that x4 isn't any different from x16 for a single card.
- borissk 3y agoIf the card onboard memory is not enough, the card can use the PC RAM over the PCIe connection - and then the PCIe bandwidth is very important. But the main reason for less lanes is to cut costs.
- wruza 3y agoYes, but it starts to crawl on 16x anyway. My firsthand experience with LLMs is that there's little difference between VRAM-to-RAM spills and running on just RAM. Same for LoRA training. When I accidentally disturbed too much VRAM by e.g. parallel upscales, it went from 1.3it/s to 30-40s/it and never recovered. NVidia even added a new setting in their control panel to disable CUDA RAM fallback to work around unintentional slowdowns in lowvram cards, afaiu.