7 ms·

Google OAuth Login Issues

118 points by derwiki 8y ago

shereadsthenews 8y ago
This kind of thing has the tendency to paralyze my organization. And these outages happen with surprising frequency. Isn't this terribly disruptive to lots of people?
- toomuchtodo 8y ago
  Kinda dispels the myth that Google is better at uptime than your own org. Sure they have SREs, sure they have global infrastructure, but they still have outages like everyone else just the same.
  - shereadsthenews 8y ago
    I'm not sure it really dispels it. Isn't it still possible that an in-house IT org would have daily outages of their auth/authz systems?
  - deleted 8y ago
    [deleted]
  - dekhn 8y ago
    ex-SRE here. Of course Google has outages, but it would be crazy to think that nearly any smaller outfit could manage the level of service $GOOG provides.
    - toomuchtodo 8y ago
      The gap between 3 9s and 5 9s isn't that wide. Gmail itself has already had ~4-5 hours of downtime for the year [1] [2]. A couple more hours and they'll hit 3 9s (99.9%) for 2019. A small org can do about the same, and they're not paying what Google pays for infra and staff. Does Google have it harder at their scale? Of course, but they also argue they're better than most at it, at scale. [1] https://downdetector.com/status/gmail/archive https://downdetector.com/status/gmail/archive [2] https://www.theguardian.com/technology/2019/mar/13/googles-gmail-and-drive-suffer-global-outages https://www.theguardian.com/technology/2019/mar/13/googles-g... (Google's Gmail and Drive suffer global outages; Users in Australia, the US, Europe and Asia report problems with various applications for several hours)
      - dekhn 8y ago
        How could a smaller org run a million+QPS service with petabytes of storage and have the same uptime as google. that doesn't even make sense.
        toomuchtodo 8y ago
        A smaller org doesn't have to. Google has made the complexity very high ("a million+QPS service with petabytes of storage") without a corresponding increase in availability is my point. EDIT: Disclaimer: FastMail customer. Whatever downtime they've had (and I'm sure they've had some), I don't notice.
        kerng 8y ago
        Commenter is highlighting that a small org doesnt need that scale, so it's easier, maybe also cheaper, to run yourself. Also worth highlighting that for the "email user experience" pretty much any system is better then Gmail these days.
        dekhn 8y ago
        Personally (and I think many others do) consider "Availability" as being uptime * bytes, not uptime. Any true distributed system is going to have partial availability all the time (small ranges of data which cannot be fetched within a reasonable latency limit, or somebody's ISP being down). Google accepts tiny ranges of data unavailibility for limited times to not be a real service outage, while the service being hard down for 25+% of users to be a service outage. It doesn't even really make sense to compare. And since we're dealing with cloud services where the large players have billions of users increasingly providing these services to everybody from small shops (I work for a startup and we use gSuite) to enterprises (many large companies host all their email on one of 2-3 major email providers). Nothing I said above has anything to do with user experience as you describe it- that's a non sequitur.
        jjeaff 8y ago
        It absolutely makes sense to compare. Because the comparison is generally made in the context of using Google services or hosting your own or going with a much smaller provider. One could easily host their own email or other similar service for a very small org and likely have better uptime.
    - tlarkworthy 8y ago
      Ex-SWE here. A metric fuckton of engineering in cloud goes into protecting systems against douchebags (e.g. rate limits, isolation). Those systems are not required for single tenancy systems in small orgs. Furthermore, those protection systems are often a cause of downtime. I think there are grains of truth in OPs suggestion you can roll your own on at small scale cheaper and with similar reliability. The key saving is it would never survive a multi tenancy situation.
      - raxxorrax 8y ago
        On the other hand I would always recommend using auth providers. Maybe use something external like auth0 for your google hosted services. But it just seriously reduces development time.
        tlarkworthy 8y ago
        On balance I would go hosted too, for everything, but I am sympathetic to the argument that they can roll their own cheaper, especially at non-SV rates. I think it's possible.
  - NikolaeVarius 8y ago
    Google having outages does not dispel the myth Google is better at uptime than any generic org
    - pm90 8y ago
      For their own services, yes. For cloud services, my experience has been mixed. I’ve been alarmed at the frequency of issues, but they do fix them quickly and the same issues don’t happen again. As an engineer, I can empathize with this and understand. But explaining it to my PM/execs is definitely a challenge.
  - manigandham 8y ago
    The only myth is thinking that outages don't happen. If you look at actual reliability normalized over time, usage and service scale, Google is far better than what any small organization can do. It only seems like your organization can do better because you're nowhere near the same requirements. Whether this is a valid trade-off by taking on more risk and operational overhead is a judgement call but it rarely works out unless it's a business-critical service that you must control and can afford the resources to do so.
- frenchman99 8y ago
  I don't know about Google, but we've had some issues with our Git hosting service last year. At times, it paralyzed us so much that we had to move to another one, which was not quite as good but more stable at the time.
ereyes01 8y ago
For me the problem is Google-wide, got kicked off my gmail accounts, and I can't sign in again- get an error telling me to clear cookies and cache (which doesn't help). Lots of people having the same issue: https://support.google.com/chrome/thread/4399961?hl=en https://support.google.com/chrome/thread/4399961?hl=en
- jwilliams 8y ago
  In fact it makes it just slightly worse, as deleting cookies potentially logs the user out of other services (some of which require Google).
  - raxxorrax 8y ago
    Google should host backup services in AWS. I like cognito very much, even if it still lacks some features.
- c0llision 8y ago
  I had this problem earlier today too. I was able to sign-in using an incognito window and after a few hours all was back to normal.
- ereyes01 8y ago
  This appears to be working again for me as of a few minutes ago. Thanks to the folks involved in fixing. EDIT: minutes instead of hours
zelon88 8y ago
I couldn't even get to google[.]com for about 5 minutes yesterday.
sethvargo 8y ago
Hey everyone - Seth from Google here. We are aware of this issue and our team is working on it.
- chris_wot 8y ago
  Your dashboard that shows outages doesn't seem to show OAuth. Has caused us some issues working out what's going on in our organization!
  - aaronharnly 8y ago
    Yeah, TIL that https://status.cloud.google.com https://status.cloud.google.com is the relevant place for login.
  - sethvargo 8y ago
    You can see our outages here: https://status.cloud.google.com/ https://status.cloud.google.com/. OAuth is included under "Identity & Security": https://status.cloud.google.com/incident/cloud-iam/19001 https://status.cloud.google.com/incident/cloud-iam/19001
aaronharnly 8y ago
Our incident management includes chatops automation that, among other things - generates a per-incident Google Doc - opens a per-incident Google Hangout - pages appropriate people via PagerDuty During this incident (which affected our customers who use Google to log in), people had difficulty accessing all three (our PagerDuty uses Google Oauth). This was a good reminder to have alternatives for all crucial incident management services. (We have backup plans in case Slack or our automation scripts are down, but didn’t have backups for Google...)
- shhehebehdh 8y ago
  Good advice. Within Google, they have backups that depend on no production infrastructure. Something as simple as an IRC server that is always running on prem can be invaluable.
- ViViDboarder 8y ago
  Can’t emphasize this enough! Outages will eventually happen, you need to plan for them.
- lathiat 8y ago
  One of canonical’s employees was about to do a fairly important presentation (a few years ago) when our google oath went down. Could no longer load the slides from google docs and the had to make up new ones on the spot.

118 points by derwiki 8y ago

This kind of thing has the tendency to paralyze my organization. And these outages happen with surprising frequency. Isn't this terribly disruptive to lots of people?

toomuchtodo 8y ago
Kinda dispels the myth that Google is better at uptime than your own org. Sure they have SREs, sure they have global infrastructure, but they still have outages like everyone else just the same.
- shereadsthenews 8y ago
  I'm not sure it really dispels it. Isn't it still possible that an in-house IT org would have daily outages of their auth/authz systems?
- deleted 8y ago
  [deleted]
- dekhn 8y ago
  ex-SRE here. Of course Google has outages, but it would be crazy to think that nearly any smaller outfit could manage the level of service $GOOG provides.
  - toomuchtodo 8y ago
    The gap between 3 9s and 5 9s isn't that wide. Gmail itself has already had ~4-5 hours of downtime for the year [1] [2]. A couple more hours and they'll hit 3 9s (99.9%) for 2019. A small org can do about the same, and they're not paying what Google pays for infra and staff. Does Google have it harder at their scale? Of course, but they also argue they're better than most at it, at scale. [1] https://downdetector.com/status/gmail/archive https://downdetector.com/status/gmail/archive [2] https://www.theguardian.com/technology/2019/mar/13/googles-gmail-and-drive-suffer-global-outages https://www.theguardian.com/technology/2019/mar/13/googles-g... (Google's Gmail and Drive suffer global outages; Users in Australia, the US, Europe and Asia report problems with various applications for several hours)
    - dekhn 8y ago
      How could a smaller org run a million+QPS service with petabytes of storage and have the same uptime as google. that doesn't even make sense.
      - toomuchtodo 8y ago
        A smaller org doesn't have to. Google has made the complexity very high ("a million+QPS service with petabytes of storage") without a corresponding increase in availability is my point. EDIT: Disclaimer: FastMail customer. Whatever downtime they've had (and I'm sure they've had some), I don't notice.
      - kerng 8y ago
        Commenter is highlighting that a small org doesnt need that scale, so it's easier, maybe also cheaper, to run yourself. Also worth highlighting that for the "email user experience" pretty much any system is better then Gmail these days.
        dekhn 8y ago
        Personally (and I think many others do) consider "Availability" as being uptime * bytes, not uptime. Any true distributed system is going to have partial availability all the time (small ranges of data which cannot be fetched within a reasonable latency limit, or somebody's ISP being down). Google accepts tiny ranges of data unavailibility for limited times to not be a real service outage, while the service being hard down for 25+% of users to be a service outage. It doesn't even really make sense to compare. And since we're dealing with cloud services where the large players have billions of users increasingly providing these services to everybody from small shops (I work for a startup and we use gSuite) to enterprises (many large companies host all their email on one of 2-3 major email providers). Nothing I said above has anything to do with user experience as you describe it- that's a non sequitur.
        jjeaff 8y ago
        It absolutely makes sense to compare. Because the comparison is generally made in the context of using Google services or hosting your own or going with a much smaller provider. One could easily host their own email or other similar service for a very small org and likely have better uptime.
  - tlarkworthy 8y ago
    Ex-SWE here. A metric fuckton of engineering in cloud goes into protecting systems against douchebags (e.g. rate limits, isolation). Those systems are not required for single tenancy systems in small orgs. Furthermore, those protection systems are often a cause of downtime. I think there are grains of truth in OPs suggestion you can roll your own on at small scale cheaper and with similar reliability. The key saving is it would never survive a multi tenancy situation.
    - raxxorrax 8y ago
      On the other hand I would always recommend using auth providers. Maybe use something external like auth0 for your google hosted services. But it just seriously reduces development time.
      - tlarkworthy 8y ago
        On balance I would go hosted too, for everything, but I am sympathetic to the argument that they can roll their own cheaper, especially at non-SV rates. I think it's possible.
- NikolaeVarius 8y ago
  Google having outages does not dispel the myth Google is better at uptime than any generic org
  - pm90 8y ago
    For their own services, yes. For cloud services, my experience has been mixed. I’ve been alarmed at the frequency of issues, but they do fix them quickly and the same issues don’t happen again. As an engineer, I can empathize with this and understand. But explaining it to my PM/execs is definitely a challenge.
- manigandham 8y ago
  The only myth is thinking that outages don't happen. If you look at actual reliability normalized over time, usage and service scale, Google is far better than what any small organization can do. It only seems like your organization can do better because you're nowhere near the same requirements. Whether this is a valid trade-off by taking on more risk and operational overhead is a judgement call but it rarely works out unless it's a business-critical service that you must control and can afford the resources to do so.
frenchman99 8y ago
I don't know about Google, but we've had some issues with our Git hosting service last year. At times, it paralyzed us so much that we had to move to another one, which was not quite as good but more stable at the time.

ereyes01 8y ago

For me the problem is Google-wide, got kicked off my gmail accounts, and I can't sign in again- get an error telling me to clear cookies and cache (which doesn't help). Lots of people having the same issue: https://support.google.com/chrome/thread/4399961?hl=en https://support.google.com/chrome/thread/4399961?hl=en

jwilliams 8y ago
In fact it makes it just slightly worse, as deleting cookies potentially logs the user out of other services (some of which require Google).
- raxxorrax 8y ago
  Google should host backup services in AWS. I like cognito very much, even if it still lacks some features.
c0llision 8y ago
I had this problem earlier today too. I was able to sign-in using an incognito window and after a few hours all was back to normal.
ereyes01 8y ago
This appears to be working again for me as of a few minutes ago. Thanks to the folks involved in fixing. EDIT: minutes instead of hours

zelon88 8y ago

I couldn't even get to google[.]com for about 5 minutes yesterday.

sethvargo 8y ago

Hey everyone - Seth from Google here. We are aware of this issue and our team is working on it.

chris_wot 8y ago
Your dashboard that shows outages doesn't seem to show OAuth. Has caused us some issues working out what's going on in our organization!
- aaronharnly 8y ago
  Yeah, TIL that https://status.cloud.google.com https://status.cloud.google.com is the relevant place for login.
- sethvargo 8y ago
  You can see our outages here: https://status.cloud.google.com/ https://status.cloud.google.com/. OAuth is included under "Identity & Security": https://status.cloud.google.com/incident/cloud-iam/19001 https://status.cloud.google.com/incident/cloud-iam/19001

aaronharnly 8y ago

Our incident management includes chatops automation that, among other things - generates a per-incident Google Doc - opens a per-incident Google Hangout - pages appropriate people via PagerDuty During this incident (which affected our customers who use Google to log in), people had difficulty accessing all three (our PagerDuty uses Google Oauth). This was a good reminder to have alternatives for all crucial incident management services. (We have backup plans in case Slack or our automation scripts are down, but didn’t have backups for Google...)

shhehebehdh 8y ago
Good advice. Within Google, they have backups that depend on no production infrastructure. Something as simple as an IRC server that is always running on prem can be invaluable.
ViViDboarder 8y ago
Can’t emphasize this enough! Outages will eventually happen, you need to plan for them.
lathiat 8y ago
One of canonical’s employees was about to do a fairly important presentation (a few years ago) when our google oath went down. Could no longer load the slides from google docs and the had to make up new ones on the spot.