4 ms·

Many excellent points and the one of interest is power usage - it'snot just the cost of buying the kit, but also powering, maintaining (admin) aspects. So the

by Zenst 6y ago

Many excellent points and the one of interest is power usage - it'snot just the cost of buying the kit, but also powering, maintaining (admin) aspects. So the Bottom line of ownership is far bigger and as you say, you really need to get 100% utilisation to capitalise in that level of outlay. Hence the on-demand cloud option on many levels becomes cost effective, even if it does seem not cheap. Sure it is not cheap, but overall, it can and does work out way cheaper for many use cases. Makes you wonder what kind of power costs you would incur running one 100% utilisation. Certainly even with best prices, be looking at several thousand a year I would of thought, that's not even factoring in provision. Which would mean 3 phase power for that type of load and then you have to balance out the phases. So many little details that become more an issue when you start getting to datacenter level power usage. Then UPS load/capacity costs/planning, networking. So whilst the costs of these units are high, the other costs that add up, sure do add up fast.

boulos 6y ago
Oh, I didn’t mean to suggest that the powering of the system was unreasonable. For example, Hetzner will charge you about $250 per rack per month and between 15 and 30 cents (USD roughly) per kWh [1]. I assume they include the cooling and so on, but that means you’re basically looking at a ~4 kW draw for the usual 8xA100 system with some cores and RAM. So budget just under $1/hr for the power and $1000/month all in (depending on location). Most colo power in the US is closer to the Finland calculation than the Germany one. I’d say keeping a spare system in case of failure is probably a bigger deal than the $10k/yr to house it :). [1] https://www.hetzner.com/colocation https://www.hetzner.com/colocation
jjoonathan 6y ago
Eh, the Titan V I bought last year broke even vs AWS inside of a month ($2000 vs $2234). Obviously there are factors going in to that -- I could live without paying the Tesla tax (didn't need to virtualize, didn't need the vram, did need the fp64), I bought used, I didn't have a problem keeping it fed, I didn't need to burst, etc, but my point is that for some GPU workloads the cloud GPUs are really expensive and the break-even utilization is far south of 100%, more like 5%.
- codemk8 6y ago
  The GeForce makes huge difference here.
- echelon 6y ago
  I'm currently renting LambdaLabs V100 instances at 100% utilization for training https://vo.codes https://vo.codes It's really expensive, and I think I should lean into buying hardware at this point. I want to build a high end GPU rig, but was wondering how easy the setup was. I've only built "consumer" systems before (2x 1080Ti). Is there any appreciable difference? Do you have a single card? Multiple? What motherboard do you use? Do you have any takeaways or resources you can share?
  - jjoonathan 6y ago
    I only have one card. I swapped the Titan V with my 1x1080 in my old, cheap motherboard and it just worked. I had to hook up the water cooling, but I did that because it came with a block, not because I optimized the thermal design. To verify motherboard compatibility, I looked up in the nvidia specs how many PCIe lanes and at what speed I should expect and confirmed in HWinfo that they were active in that configuration -- much like I'd look at a network interface to make sure my 10/5/2.5GbE hadn't turned into 1GbE on account of gremlins in the wires. I'm not using this for machine learning, so you might want to talk to someone who is before pulling the trigger. In particular, my need for fp64 made the choice of Titan V completely trivial, whereas in ML you might have to engage brain cells to pick a card or make a wait/buy determination.
  - tw04 6y ago
    Probably depends on your budget and if you care about support. Building it yourself isn't a big deal at all, but pretty much every major compute vendor will have a GPU platform as well. That's ignoring nvidia themselves who are the easy button and probably your best bet if you think you'll need support up the software stack: https://www.nvidia.com/en-us/data-center/dgx-station/ https://www.nvidia.com/en-us/data-center/dgx-station/
- boulos 6y ago
  This comes up in nearly every thread, but NVIDIA’s EULA doesn’t permit using their drivers on “consumer” parts in “datacenters”. So you didn’t include the main reason for the Tesla tax: compliance :). The “consumer” parts are certainly popular ML workstations, and rightly so.

by Zenst 6y ago

boulos 6y ago

Oh, I didn’t mean to suggest that the powering of the system was unreasonable. For example, Hetzner will charge you about $250 per rack per month and between 15 and 30 cents (USD roughly) per kWh [1]. I assume they include the cooling and so on, but that means you’re basically looking at a ~4 kW draw for the usual 8xA100 system with some cores and RAM. So budget just under $1/hr for the power and $1000/month all in (depending on location). Most colo power in the US is closer to the Finland calculation than the Germany one. I’d say keeping a spare system in case of failure is probably a bigger deal than the $10k/yr to house it :). [1] https://www.hetzner.com/colocation https://www.hetzner.com/colocation

jjoonathan 6y ago

Eh, the Titan V I bought last year broke even vs AWS inside of a month ($2000 vs $2234). Obviously there are factors going in to that -- I could live without paying the Tesla tax (didn't need to virtualize, didn't need the vram, did need the fp64), I bought used, I didn't have a problem keeping it fed, I didn't need to burst, etc, but my point is that for some GPU workloads the cloud GPUs are really expensive and the break-even utilization is far south of 100%, more like 5%.

codemk8 6y ago
The GeForce makes huge difference here.
echelon 6y ago
I'm currently renting LambdaLabs V100 instances at 100% utilization for training https://vo.codes https://vo.codes It's really expensive, and I think I should lean into buying hardware at this point. I want to build a high end GPU rig, but was wondering how easy the setup was. I've only built "consumer" systems before (2x 1080Ti). Is there any appreciable difference? Do you have a single card? Multiple? What motherboard do you use? Do you have any takeaways or resources you can share?
- jjoonathan 6y ago
  I only have one card. I swapped the Titan V with my 1x1080 in my old, cheap motherboard and it just worked. I had to hook up the water cooling, but I did that because it came with a block, not because I optimized the thermal design. To verify motherboard compatibility, I looked up in the nvidia specs how many PCIe lanes and at what speed I should expect and confirmed in HWinfo that they were active in that configuration -- much like I'd look at a network interface to make sure my 10/5/2.5GbE hadn't turned into 1GbE on account of gremlins in the wires. I'm not using this for machine learning, so you might want to talk to someone who is before pulling the trigger. In particular, my need for fp64 made the choice of Titan V completely trivial, whereas in ML you might have to engage brain cells to pick a card or make a wait/buy determination.
- tw04 6y ago
  Probably depends on your budget and if you care about support. Building it yourself isn't a big deal at all, but pretty much every major compute vendor will have a GPU platform as well. That's ignoring nvidia themselves who are the easy button and probably your best bet if you think you'll need support up the software stack: https://www.nvidia.com/en-us/data-center/dgx-station/ https://www.nvidia.com/en-us/data-center/dgx-station/
boulos 6y ago
This comes up in nearly every thread, but NVIDIA’s EULA doesn’t permit using their drivers on “consumer” parts in “datacenters”. So you didn’t include the main reason for the Tesla tax: compliance :). The “consumer” parts are certainly popular ML workstations, and rightly so.