4 ms·
Lots of people seem to think they have no choice but to rent GPUs from cloud service providers. Remember retail GPUs are much more powerful, much cheaper and m
by andrewstuart 3y ago
Lots of people seem to think they have no choice but to rent GPUs from cloud service providers.
Remember retail GPUs are much more powerful, much cheaper and much more available than cloud GPUs.
Just go to the store and buy one.
- oblivuslimited 3y agoThank you for sharing your thoughts. Acquiring the latest technology, such as an A100 GPU, individually from a store can indeed be quite expensive, with prices around $10,000 per unit. Additionally, setting up and maintaining a home lab to scale from one GPU to thousands can be a significant investment in terms of both cost and resources. In contrast, our platform provides access to the latest GPU technology without the need for users to individually purchase and manage the hardware. We offer the scalability to deploy multiple GPUs as needed and scale back down when required, making it a more cost-effective and flexible solution compared to setting up and maintaining a personal lab. Furthermore, it's important to note that our GPUs are directly dedicated to each virtual machine, ensuring that the power and performance are not compromised by sharing resources. This ensures that our GPUs provide the full capability and performance expected, making them just as powerful as individual units.
- andrewstuart 3y agoIn the tests I’ve done, retail GPUs are much faster than data center GPUs.
- the-dude 3y agoWould you be so kind to post these results? How much faster, which models etc.
- pitched 3y agoI found some charts here showing A100 roughly doubling a 3090: https://bizon-tech.com/blog/best-gpu-for-deep-learning-rtx-3090-vs-rtx-3080-vs-titan-rtx-vs-rtx-2080-ti https://bizon-tech.com/blog/best-gpu-for-deep-learning-rtx-3... With the 4090 being a fair amount more powerful, it might give the a100 a challenge for about 10% of the cost. That’s still about 10k hours of renting one with this service though.
- YetAnotherNick 3y agoSure if you are using tiny net(by today's standard) and small batch sizes, A100 is just twice as fast compared to 3090 due to overhead of framework and memory movement. A100 has 312 FP16 TFlops[1], compared to 40 and 82.6 of 3090 Ti and 4090 respectively[2], and has 3x memory. Also it has 10x inter GPU communication bandwidth. [1]: https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/a100/pdf/nvidia-a100-datasheet-us-nvidia-1758950-r4-web.pdf https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Cent... [2]: https://images.nvidia.com/aem-dam/Solutions/Data-Center/l4/nvidia-ada-gpu-architecture-whitepaper-v2.1.pdf https://images.nvidia.com/aem-dam/Solutions/Data-Center/l4/n...
- lostmsu 3y agoI believe you are wrong. The relevant metric is tensor flops and 4090 has 330. I believe 312 you quoted for A100 is tensor flops.
- oblivuslimited 3y agoWe use the same graphic buses, such as SXM4 and PCI Express 4.0 x 16, as you would find in your system. Theoretically, there should be no difference in performance or compatibility at all. If you have conducted your tests on shared instances and have observed such differences, then it's understandable. However, I want to emphasize once again that we provide dedicated virtual machines, which means that the resources are exclusively allocated to each user and not shared.
- josefx 3y ago> Theoretically, there should be no difference in performance or compatibility at all. At least in the past there was a tendency by NVIDIA to run their professional GPUs at a lower frequency, offering reliability and correctness guarantees as tradeoff. Meanwhile most of the retail/gaming GPUs came overclocked and almost outright guaranteeing that they would start to crash and fail if you tried to run them 24/7.
- etherael 3y agoEven so, an A100 is not actually faster than an RTX 4090, right? So basically the reason you would want to use something like this instead is because you need the VRAM?
- oblivuslimited 3y agoYes, 0x008 makes a valid point regarding this topic. Data center GPUs and RTX Series use different technologies and optimizations for different purposes. Our intention is not to discourage anyone from purchasing their own GPUs. We offer our GPU cloud services as an alternative for those who may not have the resources or prefer the convenience and flexibility of renting GPUs on-demand. We apologize if the discussion took a different direction, and we appreciate your understanding.
- upbeat_general 3y agoThat’s not the case. It is significantly faster, at least for deep learning workloads. Check out Lambda Lab’s benchmarks.
- 0x008 3y agoThey are, but data center GPU (ampere and above) are twice as good in performance per watt.
- moralestapia 3y agoAside from being green and that, why would a consumer care about a leased processor's performance per watt?
- late2part 3y agoGiven a relative amount of benefit, with a ratio of Watts/Benefit, a lower ratio of Watts/Benefit should cost less to the consumer. So the consumer would care because the service is lower cost.
- deleted 3y ago[deleted]
- moralestapia 3y agoI'm truly confused by your argument and @pjlegato's ... It's more expensive to rent a GPU than to buy it. This whole comment thread started because GP wrote: >Remember retail GPUs are much more powerful, much cheaper and much more available than cloud GPUs. So, still, why would a consumer care about performance per watt?
- rat9988 3y agoBecause you'll have to include its price when you buy it, but not when you rent it. And how the economics plays out then is not obvious without maths.
- moralestapia 3y agoI can assure you it's cheaper to buy. Otherwise they wouldn't be leasing it. (ofc assuming you're not going to use it for like a day)
- deleted 3y ago[deleted]
- dtx1 3y agoEating your own dogfood I see: https://imgur.com/IXPWOSk https://imgur.com/IXPWOSk
- oblivuslimited 3y agoAs someone whose native language is not English, I find it helpful to use AI to grammatically correct and improve my texts. There's no shame in asking for help or using tools to improve your language skills at all.
- dtx1 3y agoSame here and don't interpret it as an accusation, more of an "huh, that's interesting", mostly cause I had the suspicion that this was partly written by ChatGPT and was able to detect it, I'm not entirely sure what made me realize it.
- deleted 3y ago[deleted]
- dougSF70 3y agoI have just bought 4 servers with 10 GPU (Titan X Pascal) and 384 GB of RAM for the price of a macbook. I found a solar powered CoLo facility to put them in. The economics of owning vs renting from AWS are worth it. My server set up is the same price as running one model on AWS.
- crabbone 3y agoIf memory serves, NVidia's drivers can only be used for Tesla / Ampere family, if you are using them in a datacenter. Titan / GeForce are not allowed. So, there's that (but maybe I'm missing something). Now, "running" may also mean different things. For example, you may want to do things s.a. performance diagnostics (in order to understand if your code uses resources efficiently), and then you'd need stuff like NVML, DCGM and co. Consumer-grade hardware might not be supported, or might not in principle support diagnostics collection / instrumentation. Or, if you have multiple GPU-dependent workloads that cannot saturate your resources -- you might think of MIG as being a way to address that... and, again, consumer-grade GPUs won't help you here... I'm not saying you shouldn't try self-hosting. I'm actually all for it. But, you also need to be mindful of the pros and cons. NVidia must have some reason to pitch the datacenter family of GPUs to, well, datacenters. They aren't just blowing up prices. Also, to give some sense of comparison: Titan X ~= 3.5K CUDA cores. V100 ~= 5K CUDA cores. A100 ~= 7K CUDA cores. H100 ~= 18.5K CUDA cores. It probably doesn't translate directly into H100 being six times as fast as Titan X, but, I hear that these GPU workloads might be lengthy...
- dougSF70 3y agoI bought these from a Chinese internet firm. They dismantled their data center and I am buying it.
- dougSF70 3y agoAlso, Datacenter GPUs only have passive cooling, presumably to allow for more Cuda cores. I get that what I have is older tech but for the cost of one H100 GPU I can have 60 of these servers (600 GPUs) plus some change to pay for CoLo fees.
- zamnos 3y agoThen I'd only have one. Instead, I can borrow a dozen of them and use them in parallel and I'll return them when I'm done. I don't know how many I'll need, or for how long, and cloud enables my bad habit of not planning.
- immibis 3y agoPresuming you don't want an A100 for just a few hours.
- cookieperson 3y agoThis is the actual answer. Why rent when you can own. The beefy GPUs of today will still carry their weight years into the future. If you take it even remotely seriously you'll save money and have resale value
- circuit10 3y agoBecause you’ll probably be only be using it for a small fraction of the time you have it, so sharing can be much more cost effective depending on how much you use it
- meinheld111 3y agoTrue. As with most rental models, there is a usage threshold from wich on it’s more cost effective.
- cookieperson 3y agoI dropped a couple hundred bucks on a GPU years ago. It's done a lot of work for me and has years to go. True if you don't plan on using it no point in buying it. Same goes for literally anything
- crabbone 3y ago> Why rent when you can own. This is a very silly question... the answer is the same as with virtually anything else: real estate, transportation, wedding dresses... I mean, you need to compare the price that will ultimately depend on the kind of thing you are doing... that's all. Sometimes it makes sense to rent, other times it makes sense to buy. There's no one right answer.
- protortyp 3y agoI agree. Even just a bunch of RTX 3060s go a long way[1]. Also, in the case of my startup we can use our local storage vs uploading dozens of terabytes of data to the cloud and deal with data privacy issues. [1] https://timdettmers.com/2023/01/16/which-gpu-for-deep-learning https://timdettmers.com/2023/01/16/which-gpu-for-deep-learni...
- NickHoff 3y ago> retail GPUs are much more powerful I agree with your basic point about building vs renting, but retail GPUs aren't simply more powerful. Retail GPUs are just as fast but have a lot less VRAM and memory bandwidth. The RTX4090 has 24 GB of VRAM with 1 TB/s of bandwidth. The A100 has 80 GB at 2 TB/s. For some tasks, the memory and bandwidth are the strongest constraints.
- seanw444 3y agoAI being a primary example, and that's probably the thing most people are renting these for nowadays anyways.
- modeless 3y agoRetail GPUs do not have the maxed out VRAM that is desirable for machine learning. And it's cheaper to run them, but it's not free.