7 ms·

We had an issue a few weeks ago where the google front-end servers were mangling responses from Pub/Sub and returning 502 responses, making the service complete

We had an issue a few weeks ago where the google front-end servers were mangling responses from Pub/Sub and returning 502 responses, making the service completely unusable and knocking over a number of things we have running in production. Despite paying for enterprise support and having in a P1 ticket, we had to spend Friday to Sunday gathering evidence to prove to the support staff that there was indeed a problem, because their monitoring wasn't detecting it. Right now I'm doing something similar (and since Friday!) but for TLS issues they're having. Again, because their support reps don't believe there's a problem. There are so many more problems than they ever show on their status page...

Jedi72 8y ago
They work for Google so obviously they are much smarter than you. If theres a problem its probably the customers fault. /sarcasm
- deleted 8y ago
  [deleted]
- mbrumlow 8y ago
  I was so mad to read that until you said /sarcasm :p That being said I really do think there is a difference between who is working at google today and the google we all fell in love with pre-2008. I am sure there are a amazing people still working at google, but nowhere near like it was. The way I like to think about google is that some amazing people mad ea awesome train that builds tracks in front of it -- you can call them gods maybe -- but those people are gone -- or a least the critical mass required to build such a train has dwindled to just dust. What we have left is a awesome train full of people pulling the many levers left behind. To make things even worse my last interview as a SRE left me wondering if even the people who are there know this as well, and they are actually working hard to keep out those who might expose light on to this. I don't say that because I did not get the job -- I am actually happy I did not get extended a offer. I say this with one exception, the old-timer who was my last interview. I could tell he was dripping in knowledge and eager to share it with any that would listen. I came out of his 45 min session learning many things -- I wold actually pay to work with a guy like that. I would also like to point out that the work ethic was not what I expected. I was told that when on call, my duty was to figure out the root cause was in the segment I was responsible for. I don't know about you, but if my phone rings at night I am going to see through to a resolution and understand the problem in full -- even if it is not on the segment that I was assigned. /end rant
  - mav3rick 8y ago
    The work ethic is in tact. It is not fair to load people with stress and ask them to drop everything. You're conflating poor resource allocation with "work ethic". Burning the midnight oil when it can be avoided is not work ethic. The correct way is to load balance outage resolution.
    - mbrumlow 8y ago
      I really don't know how to reply to you as you have set up a bunch of windmills you assumed from my previous post. Who said anything about poor resource allocation? Who said we need to load people with stress? That being said -- when you are on call -- dropping everything is exactly what is expected.
      - gmueckl 8y ago
        But limiting responsibilities for on call employees is a way of limiting the workload they have during that time - ultimately benefiting the employees. They are on call, not working. I don't see the windmills here.
        mav3rick 8y ago
        Yes this was my point.
      - mav3rick 8y ago
        Many many AWS people have left citing on call as the worst part of their job. Also, it is really a far fetched allegation that interviewers try to fail interviewees to hide their own incompetence. I get that you may not think much of Google SREs but to allege that is just in bad spirit. I hope you do get to see that the people inside Google are one of the best perks of working here. They are smart and motivated and willing to help each other.
        antt 8y ago
        Were they that motivated to help their customers.
- lstamour 8y ago
  I totally heard that when trying to get engineer attention to YouTube Premium’s frequent “download errors” due to their transcoding-on-the-fly (or something). I was telling a support rep I had evidence that if I switched the setting from standard to high def (or vice versa) that the error would go away, but I could reproduce it with the same video, and thought it was a CDN/transcode issue. They kept marking the ticket as “unable to reproduce” and I had to wonder — as a paying customer, don’t they have analytics on my phone that would show exactly the request I was making which was failing in their logs? And if they saw it succeed, why not tell me the problem was my ISP? I’d have been happy to follow up... but nothing’s ever wrong in Google-land. :/
- jkaplowitz 8y ago
  Many of the tier 1 GCP support reps work for external vendors nowadays, which is probably part of the problem. During my time on the GCE team (note I don't work at Google now) I knew multiple full-time Google employee support reps, including some still at the company. They have the good attitude and deep knowledge you'd hope for. The problem is simply about how Google scales their GCP support org. To be completely clear, AWS support is by and large not great either. If you're a big or strategically important customer, of course, you can get a good response from either company.
  - deleted 8y ago
    [deleted]
  - halbritt 8y ago
    80% of my support experiences are laughably bad. 20% of my support experiences are amazing. Fortunately, I don't require decent support to keep my service running. My sales rep tells me that he's aware of the problem. I speculate it's simply the result of GCP trying to grow the org very quickly.
navinsylvester 8y ago
In general GCP has quota limits so its expected of customers to catch 5xx and do exponential back off. But this info is not explicitly stated. From my personal experience - i think all big cloud providers first two level support staff is no good if it isn't an obvious dumb one on your part. I always prefer to forgo support and try to go through every bit of their documentation to figure it out on our own. This helps to save huge amount of time. But if you have developer support - it can help to expedite things little faster though.
halbritt 8y ago
Did they ask you for a screenshot? That's my favorite.
reaperducer 8y ago
Again, because their support reps don't believe there's a problem. Perhaps if you explained it on a whiteboard...
StreamBright 8y ago
I just took a screenshot of this and any time somebody asks my why AWS i will forward them. Thanks!

Jedi72 8y ago

They work for Google so obviously they are much smarter than you. If theres a problem its probably the customers fault. /sarcasm

deleted 8y ago
[deleted]
mbrumlow 8y ago
I was so mad to read that until you said /sarcasm :p That being said I really do think there is a difference between who is working at google today and the google we all fell in love with pre-2008. I am sure there are a amazing people still working at google, but nowhere near like it was. The way I like to think about google is that some amazing people mad ea awesome train that builds tracks in front of it -- you can call them gods maybe -- but those people are gone -- or a least the critical mass required to build such a train has dwindled to just dust. What we have left is a awesome train full of people pulling the many levers left behind. To make things even worse my last interview as a SRE left me wondering if even the people who are there know this as well, and they are actually working hard to keep out those who might expose light on to this. I don't say that because I did not get the job -- I am actually happy I did not get extended a offer. I say this with one exception, the old-timer who was my last interview. I could tell he was dripping in knowledge and eager to share it with any that would listen. I came out of his 45 min session learning many things -- I wold actually pay to work with a guy like that. I would also like to point out that the work ethic was not what I expected. I was told that when on call, my duty was to figure out the root cause was in the segment I was responsible for. I don't know about you, but if my phone rings at night I am going to see through to a resolution and understand the problem in full -- even if it is not on the segment that I was assigned. /end rant
- mav3rick 8y ago
  The work ethic is in tact. It is not fair to load people with stress and ask them to drop everything. You're conflating poor resource allocation with "work ethic". Burning the midnight oil when it can be avoided is not work ethic. The correct way is to load balance outage resolution.
  - mbrumlow 8y ago
    I really don't know how to reply to you as you have set up a bunch of windmills you assumed from my previous post. Who said anything about poor resource allocation? Who said we need to load people with stress? That being said -- when you are on call -- dropping everything is exactly what is expected.
    - gmueckl 8y ago
      But limiting responsibilities for on call employees is a way of limiting the workload they have during that time - ultimately benefiting the employees. They are on call, not working. I don't see the windmills here.
      - mav3rick 8y ago
        Yes this was my point.
    - mav3rick 8y ago
      Many many AWS people have left citing on call as the worst part of their job. Also, it is really a far fetched allegation that interviewers try to fail interviewees to hide their own incompetence. I get that you may not think much of Google SREs but to allege that is just in bad spirit. I hope you do get to see that the people inside Google are one of the best perks of working here. They are smart and motivated and willing to help each other.
      - antt 8y ago
        Were they that motivated to help their customers.
lstamour 8y ago
I totally heard that when trying to get engineer attention to YouTube Premium’s frequent “download errors” due to their transcoding-on-the-fly (or something). I was telling a support rep I had evidence that if I switched the setting from standard to high def (or vice versa) that the error would go away, but I could reproduce it with the same video, and thought it was a CDN/transcode issue. They kept marking the ticket as “unable to reproduce” and I had to wonder — as a paying customer, don’t they have analytics on my phone that would show exactly the request I was making which was failing in their logs? And if they saw it succeed, why not tell me the problem was my ISP? I’d have been happy to follow up... but nothing’s ever wrong in Google-land. :/
jkaplowitz 8y ago
Many of the tier 1 GCP support reps work for external vendors nowadays, which is probably part of the problem. During my time on the GCE team (note I don't work at Google now) I knew multiple full-time Google employee support reps, including some still at the company. They have the good attitude and deep knowledge you'd hope for. The problem is simply about how Google scales their GCP support org. To be completely clear, AWS support is by and large not great either. If you're a big or strategically important customer, of course, you can get a good response from either company.
- deleted 8y ago
  [deleted]
- halbritt 8y ago
  80% of my support experiences are laughably bad. 20% of my support experiences are amazing. Fortunately, I don't require decent support to keep my service running. My sales rep tells me that he's aware of the problem. I speculate it's simply the result of GCP trying to grow the org very quickly.

navinsylvester 8y ago

In general GCP has quota limits so its expected of customers to catch 5xx and do exponential back off. But this info is not explicitly stated. From my personal experience - i think all big cloud providers first two level support staff is no good if it isn't an obvious dumb one on your part. I always prefer to forgo support and try to go through every bit of their documentation to figure it out on our own. This helps to save huge amount of time. But if you have developer support - it can help to expedite things little faster though.

halbritt 8y ago

Did they ask you for a screenshot? That's my favorite.

reaperducer 8y ago

Again, because their support reps don't believe there's a problem. Perhaps if you explained it on a whiteboard...

StreamBright 8y ago

I just took a screenshot of this and any time somebody asks my why AWS i will forward them. Thanks!