4 ms·

On a previous job we had a huge GPU server on prem because we wanted to use it for deep learning. It would have been much, much cheaper to rent gpu in the clou

On a previous job we had a huge GPU server on prem because we wanted to use it for deep learning. It would have been much, much cheaper to rent gpu in the cloud and when we knew _something_ about our workload we could buy an on prem and maybe rent extra if needed. But we weren’t cleared for cloud. Regarding in-house expertise: I think most companies would like to have in house expertise but they can’t hire the expertise they need so they use consultants.

dnautics 5y ago
that's really surprising to me, because I used to run a GPU cloud that was cheaper than amazon, and the margins were insane.
- Enginerrrd 5y ago
  It all comes down to utilization as a percentage of time. If the workloads are infrequent enough, GPU cloud providers on going to crush the onsite equipment. However... I was also sort of surprised, because last time I priced this out, you can have pretty low utilization rates of the equipment and still come out ahead.
  - dnautics 5y ago
    the other thing to consider though is that if you are under pressure by management to not utilize GPU cloud, you might do fewer experiments than if you had at least a GPU workstation under the desk. (that, presumably, you can share with other workmates)
- wiz21c 5y ago
  I've looked for some GPU on the web and it's super expensive to rent. I was expecting to rent that for say 50$ per month (considering I can rent some webspace for less then 10$ a month) but everything was more expensive like 600$/month (I look for entry level GPU, used just a bit, I'm a student) ... Did I look correctly ?
  - dnautics 5y ago
    if you want an always-on gpu for $50/mo? That's unreasonable. Consider how much "building from scratch" even a cheap "minimal" GPU unit costs (let's say 3k GPU + 1k supporting PC), whereas a cheap "building from scratch minimal webspace" is a fraction of a single CPU unit, so let's say $100-ish/unit dollars, to be really generous. A rented non-fractional GPU should be easily minimally 40x the cost of a website. Don't expect these ratios to be authoritative; there are obviously different economies of scale and many many details I papered over, but this should just give you an idea of the back of the envelope "relative costs"; just that 5x is way out of the range you should have guessed.
ttymck 5y ago
I appreciate your point about "they can't hire the expertise they need" but I feel it's missing the "because": because those hires are so expensive? Because they only exist as consultants? Why can't they hire the expertise?
- baq 5y ago
  because nobody wants to be hired for the position. it isn't even about the money, there's simply no candidates - and any candidates competent enough are already consultants and/or freelancers.
  - lotsofpulp 5y ago
    I.e. those hires are expensive. I am sure there would be candidates with an offer of $1M/year.
    - baq 5y ago
      of course, everyone has a number - but also, for that money you'll just hire a consultant and will probably come out ahead.
- Suchos 5y ago
  For company I am working at, it is acceptable to pay more for even long term consultanst, than to pay market price for experienced developers. The reason explained was, that it is different budget line - HR does not want to "overpay" for emplyees, so consultancy is the only option.
- wodenokoto 5y ago
  I would so that about 3/4 places I've consulted, I wouldn't want to work full time. Mostly because they simply didn't know how to run an IT office, and so they couldn't build a culture where good IT people want to work. When they finally get a strong candidate, they end up overloading the poor hire, because finally, it looks like they can get their infrastructure fixed or dream project off the ground.

dnautics 5y ago

that's really surprising to me, because I used to run a GPU cloud that was cheaper than amazon, and the margins were insane.

Enginerrrd 5y ago
It all comes down to utilization as a percentage of time. If the workloads are infrequent enough, GPU cloud providers on going to crush the onsite equipment. However... I was also sort of surprised, because last time I priced this out, you can have pretty low utilization rates of the equipment and still come out ahead.
- dnautics 5y ago
  the other thing to consider though is that if you are under pressure by management to not utilize GPU cloud, you might do fewer experiments than if you had at least a GPU workstation under the desk. (that, presumably, you can share with other workmates)
wiz21c 5y ago
I've looked for some GPU on the web and it's super expensive to rent. I was expecting to rent that for say 50$ per month (considering I can rent some webspace for less then 10$ a month) but everything was more expensive like 600$/month (I look for entry level GPU, used just a bit, I'm a student) ... Did I look correctly ?
- dnautics 5y ago
  if you want an always-on gpu for $50/mo? That's unreasonable. Consider how much "building from scratch" even a cheap "minimal" GPU unit costs (let's say 3k GPU + 1k supporting PC), whereas a cheap "building from scratch minimal webspace" is a fraction of a single CPU unit, so let's say $100-ish/unit dollars, to be really generous. A rented non-fractional GPU should be easily minimally 40x the cost of a website. Don't expect these ratios to be authoritative; there are obviously different economies of scale and many many details I papered over, but this should just give you an idea of the back of the envelope "relative costs"; just that 5x is way out of the range you should have guessed.

ttymck 5y ago

I appreciate your point about "they can't hire the expertise they need" but I feel it's missing the "because": because those hires are so expensive? Because they only exist as consultants? Why can't they hire the expertise?

baq 5y ago
because nobody wants to be hired for the position. it isn't even about the money, there's simply no candidates - and any candidates competent enough are already consultants and/or freelancers.
- lotsofpulp 5y ago
  I.e. those hires are expensive. I am sure there would be candidates with an offer of $1M/year.
  - baq 5y ago
    of course, everyone has a number - but also, for that money you'll just hire a consultant and will probably come out ahead.
Suchos 5y ago
For company I am working at, it is acceptable to pay more for even long term consultanst, than to pay market price for experienced developers. The reason explained was, that it is different budget line - HR does not want to "overpay" for emplyees, so consultancy is the only option.
wodenokoto 5y ago
I would so that about 3/4 places I've consulted, I wouldn't want to work full time. Mostly because they simply didn't know how to run an IT office, and so they couldn't build a culture where good IT people want to work. When they finally get a strong candidate, they end up overloading the poor hire, because finally, it looks like they can get their infrastructure fixed or dream project off the ground.