3 ms·

I used to be an AWS true believer. Now I find it increasingly difficult to be enthused about anything AWS. It’s all so expensive, complex and locked in. I recen

I used to be an AWS true believer. Now I find it increasingly difficult to be enthused about anything AWS. It’s all so expensive, complex and locked in. I recently shut down everything I had on AWS except Route 53 and workmail, both of which I really like. The craziest AWS thing is that they moved all GPU instances to a quota system where you have to request access, specifying the number and type and location of instances that you want to run. It’s like Azure which has an equally terrible quota system. Anyhow I needed to run some performance tests in GPU instances. In the past I would have spun up a bunch of different instance types, run my tests and moved on. In this case I applied for a quota of one machine type and 24 hours later AWS got back to me to approve my request to run one instance. At that point I gave up on the idea of AWS being the heart of any GPU based infrastructure.

qbasic_forever 4y ago
It's not like "Open" AI is any less locked in or opaque though...
- verdverm 4y ago
  OpenAI is a very limited API surface and there are already projects out there creating interfaces on top of N GPT implementations spanning private & public, giving you choice. The comparison here is apples & oranges
dubcanada 4y ago
They all are, what did you switch too? Suppose it's kind of the point really, it's a cloud with a bunch of "tools". You are welcome to install MySQL or elasticsearch on another VPS outside of AWS and use it. But that's not what customers want.
crazygringo 4y ago
Everything you say seems equally applicable to any cloud provider. They're all expensive and they all want to lock you in, because they're businesses. GPU's are in extremely high demand now so that's just how it is. And they're complex because different customers have different needs, so that's inherent. I don't see how any of your complaints are anything a for-profit enterprise cloud provider could change and still survive as a business.
mochomocha 4y ago
They behave this way because they are so tight on GPU capacity given incredible demand. Getting a GPU instance on GCP is as miserable of an experience. Try Lambda Labs, there's usually more availability, and more importantly you get some visibility into currently available capacity.
- ceejayoz 4y ago
  GPU instances are also a favorite option for crypto miners using compromised AWS accounts. They run up $20k in bills in a short period, which AWS typically nulls out if you beg.
metadat 4y ago
I know Character.ai prefers Oracle Cloud for GPU compute workloads because OCI can actually deliver significant capacity in a reasonable amount of time (often < 24 hours, almost always within a few days for hundreds [or more] of GPU instances, especially if you aren't picky about the region). I'm all for bagging in Oracle in general, yet it's undeniable this is one area where they're leagues ahead and winning.
- asah 4y ago
  is that because they are 10x the scale... or because they have 1/10th the customers demand...
- skybrian 4y ago
  It sounds like they temporarily have excess capacity? I don't think you can count on that to last.
SanderNL 4y ago
We told you this would happen, but no, we were luddites.
ayhoung 4y ago
besides GPU instances, have you tried out lightsail? i find its quite competitive with DO, Linode, etc. with AWS infra backing
cavisne 4y ago
GPU's in cloud is just a bad business unfortunately. There is only one vendor (for anything scientific/ML anyway) and you have to buy specific (overpriced) models from them to be compliant with the license. So zero negotiating power, and everything is proprietary so you cant build any software/support edge. You cant oversell a GPU like you can CPU/memory, and most of the computation happens on the chip so no one cares how good the rest of your stack is. And finally you are in a neverending battle with crypto miners, which would never be profitable on cloud so they have no intention of paying you.
- lovesdogsnsnow 4y ago
  Aren't interconnect bandwidth and latency important for training workloads?
  - cavisne 4y ago
    Yes, but to move the needle you need to use another single vendor Mellanox (which nvidia owns). So the (slight) differentiations are. AWS - gpudirect and high bandwidth (but also high latency) GCP - 16 gpus on one machine Azure/OCI - mellanox

qbasic_forever 4y ago

It's not like "Open" AI is any less locked in or opaque though...

verdverm 4y ago
OpenAI is a very limited API surface and there are already projects out there creating interfaces on top of N GPT implementations spanning private & public, giving you choice. The comparison here is apples & oranges

dubcanada 4y ago

They all are, what did you switch too? Suppose it's kind of the point really, it's a cloud with a bunch of "tools". You are welcome to install MySQL or elasticsearch on another VPS outside of AWS and use it. But that's not what customers want.

crazygringo 4y ago

Everything you say seems equally applicable to any cloud provider. They're all expensive and they all want to lock you in, because they're businesses. GPU's are in extremely high demand now so that's just how it is. And they're complex because different customers have different needs, so that's inherent. I don't see how any of your complaints are anything a for-profit enterprise cloud provider could change and still survive as a business.

mochomocha 4y ago

They behave this way because they are so tight on GPU capacity given incredible demand. Getting a GPU instance on GCP is as miserable of an experience. Try Lambda Labs, there's usually more availability, and more importantly you get some visibility into currently available capacity.

ceejayoz 4y ago
GPU instances are also a favorite option for crypto miners using compromised AWS accounts. They run up $20k in bills in a short period, which AWS typically nulls out if you beg.

metadat 4y ago

I know Character.ai prefers Oracle Cloud for GPU compute workloads because OCI can actually deliver significant capacity in a reasonable amount of time (often < 24 hours, almost always within a few days for hundreds [or more] of GPU instances, especially if you aren't picky about the region). I'm all for bagging in Oracle in general, yet it's undeniable this is one area where they're leagues ahead and winning.

asah 4y ago
is that because they are 10x the scale... or because they have 1/10th the customers demand...
skybrian 4y ago
It sounds like they temporarily have excess capacity? I don't think you can count on that to last.

SanderNL 4y ago

We told you this would happen, but no, we were luddites.

ayhoung 4y ago

besides GPU instances, have you tried out lightsail? i find its quite competitive with DO, Linode, etc. with AWS infra backing

cavisne 4y ago

GPU's in cloud is just a bad business unfortunately. There is only one vendor (for anything scientific/ML anyway) and you have to buy specific (overpriced) models from them to be compliant with the license. So zero negotiating power, and everything is proprietary so you cant build any software/support edge. You cant oversell a GPU like you can CPU/memory, and most of the computation happens on the chip so no one cares how good the rest of your stack is. And finally you are in a neverending battle with crypto miners, which would never be profitable on cloud so they have no intention of paying you.

lovesdogsnsnow 4y ago
Aren't interconnect bandwidth and latency important for training workloads?
- cavisne 4y ago
  Yes, but to move the needle you need to use another single vendor Mellanox (which nvidia owns). So the (slight) differentiations are. AWS - gpudirect and high bandwidth (but also high latency) GCP - 16 gpus on one machine Azure/OCI - mellanox