4 ms·
Wondering whether most major inference libraries support storage-direct or if the listed providers are cheaping out on storage latency. Several seconds to load
by touisteur 3y ago
Wondering whether most major inference libraries support storage-direct or if the listed providers are cheaping out on storage latency. Several seconds to load a 100MB model when pcie 4 is 256Gb/s I'd have expected an order of magnitude less - at least from my experience with gpudirect and real-time processing of up to 200Gb/s network streams with datacenter gpus (not performing inference but I'm talking about actual streaming to gpu memory here). PCIe-5 and H100 (the only one in the line-up that has PCIe-5...) should also improve on that.
- touisteur 3y agoAdditional note, the L4 inferencing board is also still on PCIe-4 (like the L40, very sadly), so NVIDIA doesn't seem to foresee the need for more bandwidth on inferencing workloads in the next 3 years (until they maybe get a H30 out, with the binned not-fully-functional H100s? Maybe? Hopefully?) and/or will make you pay for a full H100.