4 ms·
In the tests I’ve done, retail GPUs are much faster than data center GPUs.
by andrewstuart 3y ago
In the tests I’ve done, retail GPUs are much faster than data center GPUs.
- the-dude 3y agoWould you be so kind to post these results? How much faster, which models etc.
- pitched 3y agoI found some charts here showing A100 roughly doubling a 3090: https://bizon-tech.com/blog/best-gpu-for-deep-learning-rtx-3090-vs-rtx-3080-vs-titan-rtx-vs-rtx-2080-ti https://bizon-tech.com/blog/best-gpu-for-deep-learning-rtx-3... With the 4090 being a fair amount more powerful, it might give the a100 a challenge for about 10% of the cost. That’s still about 10k hours of renting one with this service though.
- YetAnotherNick 3y agoSure if you are using tiny net(by today's standard) and small batch sizes, A100 is just twice as fast compared to 3090 due to overhead of framework and memory movement. A100 has 312 FP16 TFlops[1], compared to 40 and 82.6 of 3090 Ti and 4090 respectively[2], and has 3x memory. Also it has 10x inter GPU communication bandwidth. [1]: https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/a100/pdf/nvidia-a100-datasheet-us-nvidia-1758950-r4-web.pdf https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Cent... [2]: https://images.nvidia.com/aem-dam/Solutions/Data-Center/l4/nvidia-ada-gpu-architecture-whitepaper-v2.1.pdf https://images.nvidia.com/aem-dam/Solutions/Data-Center/l4/n...
- lostmsu 3y agoI believe you are wrong. The relevant metric is tensor flops and 4090 has 330. I believe 312 you quoted for A100 is tensor flops.
- oblivuslimited 3y agoWe use the same graphic buses, such as SXM4 and PCI Express 4.0 x 16, as you would find in your system. Theoretically, there should be no difference in performance or compatibility at all. If you have conducted your tests on shared instances and have observed such differences, then it's understandable. However, I want to emphasize once again that we provide dedicated virtual machines, which means that the resources are exclusively allocated to each user and not shared.
- josefx 3y ago> Theoretically, there should be no difference in performance or compatibility at all. At least in the past there was a tendency by NVIDIA to run their professional GPUs at a lower frequency, offering reliability and correctness guarantees as tradeoff. Meanwhile most of the retail/gaming GPUs came overclocked and almost outright guaranteeing that they would start to crash and fail if you tried to run them 24/7.
- etherael 3y agoEven so, an A100 is not actually faster than an RTX 4090, right? So basically the reason you would want to use something like this instead is because you need the VRAM?
- oblivuslimited 3y agoYes, 0x008 makes a valid point regarding this topic. Data center GPUs and RTX Series use different technologies and optimizations for different purposes. Our intention is not to discourage anyone from purchasing their own GPUs. We offer our GPU cloud services as an alternative for those who may not have the resources or prefer the convenience and flexibility of renting GPUs on-demand. We apologize if the discussion took a different direction, and we appreciate your understanding.
- upbeat_general 3y agoThat’s not the case. It is significantly faster, at least for deep learning workloads. Check out Lambda Lab’s benchmarks.
- 0x008 3y agoThey are, but data center GPU (ampere and above) are twice as good in performance per watt.
- moralestapia 3y agoAside from being green and that, why would a consumer care about a leased processor's performance per watt?
- late2part 3y agoGiven a relative amount of benefit, with a ratio of Watts/Benefit, a lower ratio of Watts/Benefit should cost less to the consumer. So the consumer would care because the service is lower cost.
- deleted 3y ago[deleted]
- moralestapia 3y agoI'm truly confused by your argument and @pjlegato's ... It's more expensive to rent a GPU than to buy it. This whole comment thread started because GP wrote: >Remember retail GPUs are much more powerful, much cheaper and much more available than cloud GPUs. So, still, why would a consumer care about performance per watt?
- rat9988 3y agoBecause you'll have to include its price when you buy it, but not when you rent it. And how the economics plays out then is not obvious without maths.
- moralestapia 3y agoI can assure you it's cheaper to buy. Otherwise they wouldn't be leasing it. (ofc assuming you're not going to use it for like a day)