7 ms·
Google OAuth Is Failing with 500 Error Code
- t0astbread 7y agoIs this just my perception or is GCP really down a lot this year?
- DevKoala 7y agoI like the majority of GCP products I work with, but judging by the amount of issues in the past year, GCP feels amateurish compared to AWS, which we continue to user in order to host most mission critical operations. I cant access any of my Airflow clusters atm. :/
- DevKoala 7y agoTo whoever downvoted me. Censoring the fact customers have issues, doesn't make the product better.
- mrobins 7y agoLink to incident on Google Cloud status: https://status.cloud.google.com/incident/developers-console/19008 https://status.cloud.google.com/incident/developers-console/...
- bmurphy1976 7y agoIt sure feels like there have been quite a few big outages this summer (Google in particular). I wonder if they are getting sloppy or this is just bad luck?
- farisjarrah 7y agoMy gut reaction is that many companies make the majority of their income/deals during the fall/winter/spring (think back to school shopping, Christmas, re-signing contracts for the next year, tax season, etc.) Thus, many companies try to make major infrastructure changes during the summer, when there will be a slightly smaller negative impact on the business's bottom line in case something does go wrong.
- derefr 7y agoAnother way to think about that is that PMs spend three quarters of the year distracted by meetings with external clients, and only have summer to actually ride their teams on implementing the things they've negotiated.
- FrobozzElectric 7y agoMy take is that many people are out on vacation during the summer months and sometimes things break (or break harder than usual) when certain knowledgeable people aren't available.
- jdm2212 7y agoAlso, summer interns. A friend who worked at FB said outages there go up markedly when interns start pushing to prod.
- remyp 7y ago> interns start pushing to prod hey everybody, I spotted the problem! I'll keep an eye on my mailbox for the giant consulting fee that I assume is enroute.
- deleted 7y ago[deleted]
- IAmGraydon 7y agoYou really think interns are allowed to push code to production at FB?
- umanwizard 7y agoI worked at FB for four years, and they are in fact allowed to do so.
- swedish_mafia 7y agoMaybe memorising algorithms from a textbook does not translate to great system engineering and SRE skills? Who would have thought.
- DannyBee 7y agoI'm glad you have a good handle on the data about the predictive power Google interviews have about the individuals. Can you share the data?
- swedish_mafia 7y agoI have a good handle on how the customer base for cloud computing services perceives cloud vendors and aws is way ahead of everyone else by a mile. This is because they have a far better and complete platform, and it appears to be better engineered too with fewer of the downtime incidents we’ve been seeing at the other vendors. Why do you suspect this is? Let me tell you. Amazon had better people. Full stop.
- DannyBee 7y agoThat's a whole lot of words to say "no"
- randomgoog 7y agoIt’s random. You might as well read chicken entrails; at least then there’d be less racial and gender bias in the process.
- DannyBee 7y agoThis would also require killing a lot of chickens, which feels less pleasant.
- rhizome 7y agoGoogle only hires the best of the best.
- herpderperator 7y agoJust as I was implementing and testing high-priority Google SSO changes at work... it goes down :-)
- ilikehurdles 7y agoYou broke Google. I hope you feel good about yourself.
- stereobit 7y agoSame :-/
- rinchik 7y agosame. and also thought that i broke it. btw never saw 500 from G before, usually just timeouts during outages
- frostyj 7y ago*Again
- frostyj 7y agoEh, I was stating a fact, don't understand why it got down voted.
- gundmc 7y agoLogin from incognito works for me so I assumed one of my extensions was causing an issue somehow and started disabling them. I'm sorry I doubted you, extensions.
- henpa 7y agoI can't login to my business gmail account using Google Chrome, but I can login successfully using Internet Explorer and Firefox. Duh. :-)
- psetq 7y agoChrome in incognito mode seems to work as well.
- samcheng 7y agoMy users are reporting that it's back up, or at least intermittently working now.
- jordanthoms 7y agoGoogle often has a outage or two around this time of the year when all the US schools come back and millions of students log in at the same time.
- stronglikedan 7y agoSays something that they know the what, where, when, and why, but the still don't (or won't) prevent it.
- bena 7y agoIt's a matter of cost. It's a temporary spike of traffic far outside of average use. In order to handle it, they'd need a lot of resources that just aren't justifiable for the rest of the year. You seem similar things when places sell highly desired items in online stores.
- vorpalhex 7y agoThey're a bleeding cloud provider. I think they can temporarily scale up a bit.
- Animats 7y agoWhen you use Google OAuth, you give Google the power to turn you off. Think about it.
- IAmGraydon 7y agoUnless you own the complete path from server to user (and no one does) there is always someone who can “turn you off”.
- Animats 7y agoRemember when you could route around failures? Have connections to two different backbone providers? Does anyone still do that?
- TheIronYuppie 7y agoDisclosure: Former Googler I didn't think I would laugh that hard coming in here and reading these comments, but here we are. I can virtually guarantee that the following things are not the root of the problem: * Lack of compute/networking/storage * Incompetence of employees * Back to school traffic spikes * Just about anything else here My $0.02. these are almost always due to bad roll outs, usually configuration changes. But I've been wrong countless times before!
- codingslave 7y agoThanks for the reassuring note that google is still a hegemonic super power. Almost spilled my kool aid
- VectorLock 7y ago"these are almost always due to bad roll outs, usually configuration changes." Some would consider that falling under #2. (Not me though)
- heleninboodler 7y agoGiven that "87.623% of all outages are caused by changes" is generally accepted wisdom of running large scale services, I would tend to disagree with those people. Incompetence: no. Opportunity for improvement: yes. Edit: made-up number has large margin of error
- eecsninja 7y agoGoogler not on Cloud team, but using Google Cloud Platform for an internal project. I've encountered my share of bugs and other flaws while using this platform. I think all these platforms are just too damn complex and brittle. It's easy for even a smart SWE or SRE to overlook one little thing that will bring down a bigger part of system.