4 ms·
I think this is another instance of GCP just being really, really terrible at interacting with customers. I'm biased a bit (in favor of GCP), I suppose in that
by halbritt 8y ago
I think this is another instance of GCP just being really, really terrible at interacting with customers. I'm biased a bit (in favor of GCP), I suppose in that I have a fair bit of infra in GCP and I really like it. I'll share a couple anecdotes.
Last year my team migrated most of our infrastructure out of AWS into GCP. We'd been running k8s with kops in AWS and really liked using GKE. We also developed a bizdev arrangement.
As I was scaling up my spend in GCP, I began purchasing committed use discounts. Roughly similar to reserved instances in AWS. I'd already made one purchase for around $5k a month and these are tied to a specific region and can't be moved. I went to purchase a second $5k block, and typo'd the request ending up with $5k worth of infra in us-central rather than us-west. The purchase doesn't go into effect until the following day and showed as "pending" in the console. No big deal, I thought, I'll just contact support and they'll fix it right away, I'm sure. I had this preconceived notion based on my experiences with AWS.
I open a support request and about an hour later I get a response that basically tells me that once I've clicked the thing, there's no undoing it and have a nice day.
I've literally just erroneously spent $5k for infra in us-central that I can't use and their response was basically, "tough". $5k is a sufficiently large loss that I'd be inclined to burn a few hours of my legal team's time dealing with this issue, something I shared with the support person. After much hemming and hawing over the course of a few days, they eventually fixed the issue.
More recently, I've been dealing with an issue that is apparently called a "stockout".
Unlike AWS, GCP does not randomize their AZ names for each customer. This means that for any given region, the "A" availability zone is where most of the new infrastructure is going to land by default. Some time in May, we started seeing resource requests for persistent SSD failing in us-west1-a. The assumption is that it would clear up pretty quickly after an hour or two, but persisted. After about a day of this, we opened up a support case asking what was going on and explaining the need for some kind of visibility or metrics for this kind of issue. The response we received was that this issue was "exceedingly rare" which was why there was no visibility and that it would be rectified shortly, but couldn't be given any specific timeline.
I followed up with my account rep, he escalated to a "customer engineer" who read the support engineers notes and elaborated how "rare" this event was and how unlikely it was to recur. Again, I contacted my account rep and explained my unhappiness with the qualitative responses from "engineers" and that I needed quite a bit more information on which to act. He was sympathetic throughout this whole process escalating inside the organization as needed and shared with me some fairly candid information that I couldn't get from anyone else.
Apparently, the issue is called "stockout" and us-west1-a had a long history of them. The issues had been getting progressively worse from the beginning of the year and at this time, this AZ was the most notorious of any in any region. Basically, the support engineer either patently lied or was just making stuff up. Also, I shared with my GCP rep how AWS load-balances across AZs. He promised to pass that along.
The moral of the story is that if you want to be in us-west, then maybe try us-west1-c. Also, GCP is a relatively young arm of a well-established company that has a terrible reputation of being able to communicate with consumers. They'll eventually figure it out, it will just take some time.