4 ms·

We were trying to move to their platform because we needed high-scale TPU or GPU compute. We were not existing customers. We currently spend all those millions

We were trying to move to their platform because we needed high-scale TPU or GPU compute. We were not existing customers. We currently spend all those millions elsewhere (AWS, LambdaLabs, Coreweave, and many more) and will continue to do so. At my previous company, we even had an announced partnership with Google and did lots of co-marketing. That doesn't mean much when a GCP engineer changes something in GCP and breaks your production. I wish it worked out. I liked working in GCP outside of these problems. I really like GKE.

WendyTheWillow 3y ago
FWIW, my experience in AWS, as far as "engineer changes something and breaks your production," is not without its own set of issues. Perhaps the AWS version of this story hasn't impacted you, but AWS also constantly tweaks its services, sometimes to the detriment of its users.
- belter 3y ago
  References please.
  - matthewcford 3y ago
    I've seen issues with RDS after an update. We were seeing CPU max out; adding more readers didn't help, and in the end, we had to rebuild the entire cluster. This somehow fixed the issue, and CPU use returned to normal levels; we didn't make any code changes to the app.
  - WendyTheWillow 3y ago
    You want me to provide references for my personal experience?
    - belter 3y ago
      Not necessarily. I would like examples of where "...AWS also constantly tweaks its services, sometimes to the detriment of its users..." so I can be better informed, and make better decisions within my projects.
      - WendyTheWillow 3y ago
        And you thought posting, “references please” would be the best way to get that info? Also those are examples, or details, even. Not references.
        belter 3y ago
        Are you going to mention, describe or post some examples....Or...keep focusing on the way I asked for it?
        WendyTheWillow 3y ago
        No, because you've been a jerk during this conversation.
- leetharris 3y ago
  I totally believe you. It could absolutely be bad luck / good luck. I think if you search across the internet you find a lot more GCP horror stories than AWS ones. Considering the market ownership of the two platforms, I'd expect the opposite to be true.
edgoode 3y ago
What company is this? Curious if you'd be open to talking about how you manage compute across providers.
- freedomben 3y ago
  I'm not GP but have been in this boat. We tried a number of different approaches, and kubernetes was by far the most successful. Terraform to provision the k8s clusters, workloads deployed to k8s. With OpenShift it gets even better, though I left the project before we finished implementation so I can't say how it went in prod. Early tests were very good though. If you're feeling bold you can do a "stretch" cluster which has nodes in different data centers (some on-prem for example, some in the cloud, or all in the cloud but different zones). The latency between the masters and nodes can cause problems though so I wouldn't separate them geographically very far.

WendyTheWillow 3y ago

FWIW, my experience in AWS, as far as "engineer changes something and breaks your production," is not without its own set of issues. Perhaps the AWS version of this story hasn't impacted you, but AWS also constantly tweaks its services, sometimes to the detriment of its users.

belter 3y ago
References please.
- matthewcford 3y ago
  I've seen issues with RDS after an update. We were seeing CPU max out; adding more readers didn't help, and in the end, we had to rebuild the entire cluster. This somehow fixed the issue, and CPU use returned to normal levels; we didn't make any code changes to the app.
- WendyTheWillow 3y ago
  You want me to provide references for my personal experience?
  - belter 3y ago
    Not necessarily. I would like examples of where "...AWS also constantly tweaks its services, sometimes to the detriment of its users..." so I can be better informed, and make better decisions within my projects.
    - WendyTheWillow 3y ago
      And you thought posting, “references please” would be the best way to get that info? Also those are examples, or details, even. Not references.
      - belter 3y ago
        Are you going to mention, describe or post some examples....Or...keep focusing on the way I asked for it?
        WendyTheWillow 3y ago
        No, because you've been a jerk during this conversation.
leetharris 3y ago
I totally believe you. It could absolutely be bad luck / good luck. I think if you search across the internet you find a lot more GCP horror stories than AWS ones. Considering the market ownership of the two platforms, I'd expect the opposite to be true.

edgoode 3y ago

What company is this? Curious if you'd be open to talking about how you manage compute across providers.

freedomben 3y ago
I'm not GP but have been in this boat. We tried a number of different approaches, and kubernetes was by far the most successful. Terraform to provision the k8s clusters, workloads deployed to k8s. With OpenShift it gets even better, though I left the project before we finished implementation so I can't say how it went in prod. Early tests were very good though. If you're feeling bold you can do a "stretch" cluster which has nodes in different data centers (some on-prem for example, some in the cloud, or all in the cloud but different zones). The latency between the masters and nodes can cause problems though so I wouldn't separate them geographically very far.