5 ms·
Thank you for sharing your thoughts. Acquiring the latest technology, such as an A100 GPU, individually from a store can indeed be quite expensive, with prices
by oblivuslimited 3y ago
Thank you for sharing your thoughts.
Acquiring the latest technology, such as an A100 GPU, individually from a store can indeed be quite expensive, with prices around $10,000 per unit. Additionally, setting up and maintaining a home lab to scale from one GPU to thousands can be a significant investment in terms of both cost and resources.
In contrast, our platform provides access to the latest GPU technology without the need for users to individually purchase and manage the hardware. We offer the scalability to deploy multiple GPUs as needed and scale back down when required, making it a more cost-effective and flexible solution compared to setting up and maintaining a personal lab.
Furthermore, it's important to note that our GPUs are directly dedicated to each virtual machine, ensuring that the power and performance are not compromised by sharing resources. This ensures that our GPUs provide the full capability and performance expected, making them just as powerful as individual units.
- andrewstuart 3y agoIn the tests I’ve done, retail GPUs are much faster than data center GPUs.
- the-dude 3y agoWould you be so kind to post these results? How much faster, which models etc.
- pitched 3y agoI found some charts here showing A100 roughly doubling a 3090: https://bizon-tech.com/blog/best-gpu-for-deep-learning-rtx-3090-vs-rtx-3080-vs-titan-rtx-vs-rtx-2080-ti https://bizon-tech.com/blog/best-gpu-for-deep-learning-rtx-3... With the 4090 being a fair amount more powerful, it might give the a100 a challenge for about 10% of the cost. That’s still about 10k hours of renting one with this service though.
- YetAnotherNick 3y agoSure if you are using tiny net(by today's standard) and small batch sizes, A100 is just twice as fast compared to 3090 due to overhead of framework and memory movement. A100 has 312 FP16 TFlops[1], compared to 40 and 82.6 of 3090 Ti and 4090 respectively[2], and has 3x memory. Also it has 10x inter GPU communication bandwidth. [1]: https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/a100/pdf/nvidia-a100-datasheet-us-nvidia-1758950-r4-web.pdf https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Cent... [2]: https://images.nvidia.com/aem-dam/Solutions/Data-Center/l4/nvidia-ada-gpu-architecture-whitepaper-v2.1.pdf https://images.nvidia.com/aem-dam/Solutions/Data-Center/l4/n...
- lostmsu 3y agoI believe you are wrong. The relevant metric is tensor flops and 4090 has 330. I believe 312 you quoted for A100 is tensor flops.
- oblivuslimited 3y agoWe use the same graphic buses, such as SXM4 and PCI Express 4.0 x 16, as you would find in your system. Theoretically, there should be no difference in performance or compatibility at all. If you have conducted your tests on shared instances and have observed such differences, then it's understandable. However, I want to emphasize once again that we provide dedicated virtual machines, which means that the resources are exclusively allocated to each user and not shared.
- josefx 3y ago> Theoretically, there should be no difference in performance or compatibility at all. At least in the past there was a tendency by NVIDIA to run their professional GPUs at a lower frequency, offering reliability and correctness guarantees as tradeoff. Meanwhile most of the retail/gaming GPUs came overclocked and almost outright guaranteeing that they would start to crash and fail if you tried to run them 24/7.
- etherael 3y agoEven so, an A100 is not actually faster than an RTX 4090, right? So basically the reason you would want to use something like this instead is because you need the VRAM?
- oblivuslimited 3y agoYes, 0x008 makes a valid point regarding this topic. Data center GPUs and RTX Series use different technologies and optimizations for different purposes. Our intention is not to discourage anyone from purchasing their own GPUs. We offer our GPU cloud services as an alternative for those who may not have the resources or prefer the convenience and flexibility of renting GPUs on-demand. We apologize if the discussion took a different direction, and we appreciate your understanding.
- upbeat_general 3y agoThat’s not the case. It is significantly faster, at least for deep learning workloads. Check out Lambda Lab’s benchmarks.
- 0x008 3y agoThey are, but data center GPU (ampere and above) are twice as good in performance per watt.
- moralestapia 3y agoAside from being green and that, why would a consumer care about a leased processor's performance per watt?
- late2part 3y agoGiven a relative amount of benefit, with a ratio of Watts/Benefit, a lower ratio of Watts/Benefit should cost less to the consumer. So the consumer would care because the service is lower cost.
- deleted 3y ago[deleted]
- moralestapia 3y agoI'm truly confused by your argument and @pjlegato's ... It's more expensive to rent a GPU than to buy it. This whole comment thread started because GP wrote: >Remember retail GPUs are much more powerful, much cheaper and much more available than cloud GPUs. So, still, why would a consumer care about performance per watt?
- rat9988 3y agoBecause you'll have to include its price when you buy it, but not when you rent it. And how the economics plays out then is not obvious without maths.
- moralestapia 3y agoI can assure you it's cheaper to buy. Otherwise they wouldn't be leasing it. (ofc assuming you're not going to use it for like a day)
- deleted 3y ago[deleted]
- dtx1 3y agoEating your own dogfood I see: https://imgur.com/IXPWOSk https://imgur.com/IXPWOSk
- oblivuslimited 3y agoAs someone whose native language is not English, I find it helpful to use AI to grammatically correct and improve my texts. There's no shame in asking for help or using tools to improve your language skills at all.
- dtx1 3y agoSame here and don't interpret it as an accusation, more of an "huh, that's interesting", mostly cause I had the suspicion that this was partly written by ChatGPT and was able to detect it, I'm not entirely sure what made me realize it.
- deleted 3y ago[deleted]
- dougSF70 3y agoI have just bought 4 servers with 10 GPU (Titan X Pascal) and 384 GB of RAM for the price of a macbook. I found a solar powered CoLo facility to put them in. The economics of owning vs renting from AWS are worth it. My server set up is the same price as running one model on AWS.
- crabbone 3y agoIf memory serves, NVidia's drivers can only be used for Tesla / Ampere family, if you are using them in a datacenter. Titan / GeForce are not allowed. So, there's that (but maybe I'm missing something). Now, "running" may also mean different things. For example, you may want to do things s.a. performance diagnostics (in order to understand if your code uses resources efficiently), and then you'd need stuff like NVML, DCGM and co. Consumer-grade hardware might not be supported, or might not in principle support diagnostics collection / instrumentation. Or, if you have multiple GPU-dependent workloads that cannot saturate your resources -- you might think of MIG as being a way to address that... and, again, consumer-grade GPUs won't help you here... I'm not saying you shouldn't try self-hosting. I'm actually all for it. But, you also need to be mindful of the pros and cons. NVidia must have some reason to pitch the datacenter family of GPUs to, well, datacenters. They aren't just blowing up prices. Also, to give some sense of comparison: Titan X ~= 3.5K CUDA cores. V100 ~= 5K CUDA cores. A100 ~= 7K CUDA cores. H100 ~= 18.5K CUDA cores. It probably doesn't translate directly into H100 being six times as fast as Titan X, but, I hear that these GPU workloads might be lengthy...
- dougSF70 3y agoI bought these from a Chinese internet firm. They dismantled their data center and I am buying it.
- dougSF70 3y agoAlso, Datacenter GPUs only have passive cooling, presumably to allow for more Cuda cores. I get that what I have is older tech but for the cost of one H100 GPU I can have 60 of these servers (600 GPUs) plus some change to pay for CoLo fees.