9 ms·

Once upon a time I was one of 3 people globally that was the entirety of the GCP support team :) You should have seen the extent to which the rest of the team

by verst 6y ago

Once upon a time I was one of 3 people globally that was the entirety of the GCP support team :) You should have seen the extent to which the rest of the team and I tried to resolve customer issues. Random App Engine app getting 500s calling an Open ID endpoint (the service wasn't strictly speaking managed by GCP). I absolutely could not reproduce this and had no other reports, but I knew it was happening for the customer. I examined everything that could be different... After 2 weeks I finally got it: The customer app was running in a different data center compared to where I was testing. Calls to the API endpoint were timing out internally (pretty aggressive timeout interval). It turns out the Open ID service wasn't running in the same data center region. That should not have been the case but occurred because the internal BigTable service had to be upgraded in a particular region and hence all App Engine apps in that region were moved to another one - a fairly seamless process. The Open ID service unfortunately wasn't provisioned in that region (it's not a dependency of App Engine). Once I identified the issue it was a matter of telling the on call SRE about it who then spun up Open ID in that region and thereby resolved the issues in a matter of 5 minutes. Often a new support case meant discovering platform limitations - for example running into BigTable anti patterns because App Engine Datastore allocated IDs sequentially by default. That could cause poor read / write performance, in part due to automatic sharding behavior of BigTable. We later switched to snowflake IDs. And then there were lots of Java folks used to Classpath scanning (default behavior of Spring Framework?) which for larger projects caused App Engine container startup to exceed the allowed time limit. That became a user education issue for which we then had to write an in depth article. Good times :) I absolutely hated dealing with billing inquiries though... so difficult to map someone's code to billable operations. Some SDK methods were of course utilizing multiple billable operations. Determining SLA violation refunds was much easier.

nostrebored 6y ago
One thing that AWS Premium Support does very well is make situations like this seem unremarkable. This is the default way you do work. Going above and beyond is solving the issue, the next issue the customer hasn't asked you about, and giving practical guidance on things like cost optimization or availability concerns after weeks of deep diving. I'm not saying you didn't do great work, but I think the secret to success at AWS Support is normalizing this level of customer obsession.
- webo 6y ago
  Is paying extra 3-10% of your already expensive invoice worth the premium support that you _might_ need? Not sure I would consider that customer obsession... more like insurance.
- verst 6y ago
  What I am saying is that this obsession was normal to us also. I didn't know any Googlers (whether in support or engineering) who didn't care about customers. We did whatever it took to resolve a support case in the quickest yet most satisfactory manner, even if it meant doing some crazy research, meanwhile we got of course overloaded with more support cases coming in. I had 100% customer satisfaction from all surveys. That was definitely hard to achieve because happy people usually don't complete surveys. :) These were the early days of many GCP services. I worked there when Compute Engine and Big Query GA'd. This was just before and possibly during the time the paid support tiers were launched (I don't recall). I left mid 2013. No idea what their support is like now. And those technical account managers just bounced customer questions to us 3-5 support engineers (technical solutions engineer) Anyways, your AWS comparison may apply to the GCP support culture now (I have no idea) but back then we were all extremely customer obsessed - by choice.
  - numbsafari 6y ago
    I really appreciate that. Both as a customer, but also as a fellow practitioner. At the end of the day, I don't think the complaints you hear about GCP Support / Service has anything to do with the people working on the line. There's always better or not better examples of folks you encounter as a customer. As a support consumer, I've learned that my feelings about the person on the other side are very much colored by the fact that I almost never call support when things are going well or I'm in a good place. I typically call support when all else fails and I'm probably ready to throw a product out the window. That's not their fault, and it shouldn't be their problem, either, to a certain extent. They're humans, doin' a job, just like me. As a support provider, I've learned that the same is true for all of my customers. I need to know that this isn't who they are (usually) when they go home to their family, or out with their friends. They're under a lot of stress and, one way or another, we've let them down and have to deal with that. I think the challenge with GCP support is all about the overall strategy as a business. My experiences with GCP support is that I have to work way too hard to get past the robots and the scripted responses in order to find someone who is willing to listen and engage with my question or problem. And then, to make matters worse, about 80% of my interactions with Google support involves them trying to introduce me to some consulting company or third party provider. That is pretty much never what I want. I don't want some third party billing integrator that is going to mess up my reporting with their "value added" service. I already pay for support, I don't want to hear a pitch about paying for more support when I feel like I'm not getting what I've already paid for. I don't hold this against the support folks at Google. I hold this against the leadership at Google. This is their strategy to contain costs or to juice sales. If it were working, I think their numbers would be very different in the market. So the proof is in the pudding. Something isn't working at GCP. I don't think it's the engineers or the teams. It's the leadership and the strategy.

by verst 6y ago

nostrebored 6y ago

One thing that AWS Premium Support does very well is make situations like this seem unremarkable. This is the default way you do work. Going above and beyond is solving the issue, the next issue the customer hasn't asked you about, and giving practical guidance on things like cost optimization or availability concerns after weeks of deep diving. I'm not saying you didn't do great work, but I think the secret to success at AWS Support is normalizing this level of customer obsession.

webo 6y ago
Is paying extra 3-10% of your already expensive invoice worth the premium support that you _might_ need? Not sure I would consider that customer obsession... more like insurance.
verst 6y ago
What I am saying is that this obsession was normal to us also. I didn't know any Googlers (whether in support or engineering) who didn't care about customers. We did whatever it took to resolve a support case in the quickest yet most satisfactory manner, even if it meant doing some crazy research, meanwhile we got of course overloaded with more support cases coming in. I had 100% customer satisfaction from all surveys. That was definitely hard to achieve because happy people usually don't complete surveys. :) These were the early days of many GCP services. I worked there when Compute Engine and Big Query GA'd. This was just before and possibly during the time the paid support tiers were launched (I don't recall). I left mid 2013. No idea what their support is like now. And those technical account managers just bounced customer questions to us 3-5 support engineers (technical solutions engineer) Anyways, your AWS comparison may apply to the GCP support culture now (I have no idea) but back then we were all extremely customer obsessed - by choice.
- numbsafari 6y ago
  I really appreciate that. Both as a customer, but also as a fellow practitioner. At the end of the day, I don't think the complaints you hear about GCP Support / Service has anything to do with the people working on the line. There's always better or not better examples of folks you encounter as a customer. As a support consumer, I've learned that my feelings about the person on the other side are very much colored by the fact that I almost never call support when things are going well or I'm in a good place. I typically call support when all else fails and I'm probably ready to throw a product out the window. That's not their fault, and it shouldn't be their problem, either, to a certain extent. They're humans, doin' a job, just like me. As a support provider, I've learned that the same is true for all of my customers. I need to know that this isn't who they are (usually) when they go home to their family, or out with their friends. They're under a lot of stress and, one way or another, we've let them down and have to deal with that. I think the challenge with GCP support is all about the overall strategy as a business. My experiences with GCP support is that I have to work way too hard to get past the robots and the scripted responses in order to find someone who is willing to listen and engage with my question or problem. And then, to make matters worse, about 80% of my interactions with Google support involves them trying to introduce me to some consulting company or third party provider. That is pretty much never what I want. I don't want some third party billing integrator that is going to mess up my reporting with their "value added" service. I already pay for support, I don't want to hear a pitch about paying for more support when I feel like I'm not getting what I've already paid for. I don't hold this against the support folks at Google. I hold this against the leadership at Google. This is their strategy to contain costs or to juice sales. If it were working, I think their numbers would be very different in the market. So the proof is in the pudding. Something isn't working at GCP. I don't think it's the engineers or the teams. It's the leadership and the strategy.