6 ms·

This kind of thing has the tendency to paralyze my organization. And these outages happen with surprising frequency. Isn't this terribly disruptive to lots of

This kind of thing has the tendency to paralyze my organization. And these outages happen with surprising frequency. Isn't this terribly disruptive to lots of people?

toomuchtodo 8y ago
Kinda dispels the myth that Google is better at uptime than your own org. Sure they have SREs, sure they have global infrastructure, but they still have outages like everyone else just the same.
- shereadsthenews 8y ago
  I'm not sure it really dispels it. Isn't it still possible that an in-house IT org would have daily outages of their auth/authz systems?
- deleted 8y ago
  [deleted]
- dekhn 8y ago
  ex-SRE here. Of course Google has outages, but it would be crazy to think that nearly any smaller outfit could manage the level of service $GOOG provides.
  - toomuchtodo 8y ago
    The gap between 3 9s and 5 9s isn't that wide. Gmail itself has already had ~4-5 hours of downtime for the year [1] [2]. A couple more hours and they'll hit 3 9s (99.9%) for 2019. A small org can do about the same, and they're not paying what Google pays for infra and staff. Does Google have it harder at their scale? Of course, but they also argue they're better than most at it, at scale. [1] https://downdetector.com/status/gmail/archive https://downdetector.com/status/gmail/archive [2] https://www.theguardian.com/technology/2019/mar/13/googles-gmail-and-drive-suffer-global-outages https://www.theguardian.com/technology/2019/mar/13/googles-g... (Google's Gmail and Drive suffer global outages; Users in Australia, the US, Europe and Asia report problems with various applications for several hours)
    - dekhn 8y ago
      How could a smaller org run a million+QPS service with petabytes of storage and have the same uptime as google. that doesn't even make sense.
      - toomuchtodo 8y ago
        A smaller org doesn't have to. Google has made the complexity very high ("a million+QPS service with petabytes of storage") without a corresponding increase in availability is my point. EDIT: Disclaimer: FastMail customer. Whatever downtime they've had (and I'm sure they've had some), I don't notice.
      - kerng 8y ago
        Commenter is highlighting that a small org doesnt need that scale, so it's easier, maybe also cheaper, to run yourself. Also worth highlighting that for the "email user experience" pretty much any system is better then Gmail these days.
        dekhn 8y ago
        Personally (and I think many others do) consider "Availability" as being uptime * bytes, not uptime. Any true distributed system is going to have partial availability all the time (small ranges of data which cannot be fetched within a reasonable latency limit, or somebody's ISP being down). Google accepts tiny ranges of data unavailibility for limited times to not be a real service outage, while the service being hard down for 25+% of users to be a service outage. It doesn't even really make sense to compare. And since we're dealing with cloud services where the large players have billions of users increasingly providing these services to everybody from small shops (I work for a startup and we use gSuite) to enterprises (many large companies host all their email on one of 2-3 major email providers). Nothing I said above has anything to do with user experience as you describe it- that's a non sequitur.
        jjeaff 8y ago
        It absolutely makes sense to compare. Because the comparison is generally made in the context of using Google services or hosting your own or going with a much smaller provider. One could easily host their own email or other similar service for a very small org and likely have better uptime.
  - tlarkworthy 8y ago
    Ex-SWE here. A metric fuckton of engineering in cloud goes into protecting systems against douchebags (e.g. rate limits, isolation). Those systems are not required for single tenancy systems in small orgs. Furthermore, those protection systems are often a cause of downtime. I think there are grains of truth in OPs suggestion you can roll your own on at small scale cheaper and with similar reliability. The key saving is it would never survive a multi tenancy situation.
    - raxxorrax 8y ago
      On the other hand I would always recommend using auth providers. Maybe use something external like auth0 for your google hosted services. But it just seriously reduces development time.
      - tlarkworthy 8y ago
        On balance I would go hosted too, for everything, but I am sympathetic to the argument that they can roll their own cheaper, especially at non-SV rates. I think it's possible.
- NikolaeVarius 8y ago
  Google having outages does not dispel the myth Google is better at uptime than any generic org
  - pm90 8y ago
    For their own services, yes. For cloud services, my experience has been mixed. I’ve been alarmed at the frequency of issues, but they do fix them quickly and the same issues don’t happen again. As an engineer, I can empathize with this and understand. But explaining it to my PM/execs is definitely a challenge.
- manigandham 8y ago
  The only myth is thinking that outages don't happen. If you look at actual reliability normalized over time, usage and service scale, Google is far better than what any small organization can do. It only seems like your organization can do better because you're nowhere near the same requirements. Whether this is a valid trade-off by taking on more risk and operational overhead is a judgement call but it rarely works out unless it's a business-critical service that you must control and can afford the resources to do so.
frenchman99 8y ago
I don't know about Google, but we've had some issues with our Git hosting service last year. At times, it paralyzed us so much that we had to move to another one, which was not quite as good but more stable at the time.

This kind of thing has the tendency to paralyze my organization. And these outages happen with surprising frequency. Isn't this terribly disruptive to lots of people?

toomuchtodo 8y ago

Kinda dispels the myth that Google is better at uptime than your own org. Sure they have SREs, sure they have global infrastructure, but they still have outages like everyone else just the same.

shereadsthenews 8y ago
I'm not sure it really dispels it. Isn't it still possible that an in-house IT org would have daily outages of their auth/authz systems?
deleted 8y ago
[deleted]
dekhn 8y ago
ex-SRE here. Of course Google has outages, but it would be crazy to think that nearly any smaller outfit could manage the level of service $GOOG provides.
- toomuchtodo 8y ago
  The gap between 3 9s and 5 9s isn't that wide. Gmail itself has already had ~4-5 hours of downtime for the year [1] [2]. A couple more hours and they'll hit 3 9s (99.9%) for 2019. A small org can do about the same, and they're not paying what Google pays for infra and staff. Does Google have it harder at their scale? Of course, but they also argue they're better than most at it, at scale. [1] https://downdetector.com/status/gmail/archive https://downdetector.com/status/gmail/archive [2] https://www.theguardian.com/technology/2019/mar/13/googles-gmail-and-drive-suffer-global-outages https://www.theguardian.com/technology/2019/mar/13/googles-g... (Google's Gmail and Drive suffer global outages; Users in Australia, the US, Europe and Asia report problems with various applications for several hours)
  - dekhn 8y ago
    How could a smaller org run a million+QPS service with petabytes of storage and have the same uptime as google. that doesn't even make sense.
    - toomuchtodo 8y ago
      A smaller org doesn't have to. Google has made the complexity very high ("a million+QPS service with petabytes of storage") without a corresponding increase in availability is my point. EDIT: Disclaimer: FastMail customer. Whatever downtime they've had (and I'm sure they've had some), I don't notice.
    - kerng 8y ago
      Commenter is highlighting that a small org doesnt need that scale, so it's easier, maybe also cheaper, to run yourself. Also worth highlighting that for the "email user experience" pretty much any system is better then Gmail these days.
      - dekhn 8y ago
        Personally (and I think many others do) consider "Availability" as being uptime * bytes, not uptime. Any true distributed system is going to have partial availability all the time (small ranges of data which cannot be fetched within a reasonable latency limit, or somebody's ISP being down). Google accepts tiny ranges of data unavailibility for limited times to not be a real service outage, while the service being hard down for 25+% of users to be a service outage. It doesn't even really make sense to compare. And since we're dealing with cloud services where the large players have billions of users increasingly providing these services to everybody from small shops (I work for a startup and we use gSuite) to enterprises (many large companies host all their email on one of 2-3 major email providers). Nothing I said above has anything to do with user experience as you describe it- that's a non sequitur.
        jjeaff 8y ago
        It absolutely makes sense to compare. Because the comparison is generally made in the context of using Google services or hosting your own or going with a much smaller provider. One could easily host their own email or other similar service for a very small org and likely have better uptime.
- tlarkworthy 8y ago
  Ex-SWE here. A metric fuckton of engineering in cloud goes into protecting systems against douchebags (e.g. rate limits, isolation). Those systems are not required for single tenancy systems in small orgs. Furthermore, those protection systems are often a cause of downtime. I think there are grains of truth in OPs suggestion you can roll your own on at small scale cheaper and with similar reliability. The key saving is it would never survive a multi tenancy situation.
  - raxxorrax 8y ago
    On the other hand I would always recommend using auth providers. Maybe use something external like auth0 for your google hosted services. But it just seriously reduces development time.
    - tlarkworthy 8y ago
      On balance I would go hosted too, for everything, but I am sympathetic to the argument that they can roll their own cheaper, especially at non-SV rates. I think it's possible.
NikolaeVarius 8y ago
Google having outages does not dispel the myth Google is better at uptime than any generic org
- pm90 8y ago
  For their own services, yes. For cloud services, my experience has been mixed. I’ve been alarmed at the frequency of issues, but they do fix them quickly and the same issues don’t happen again. As an engineer, I can empathize with this and understand. But explaining it to my PM/execs is definitely a challenge.
manigandham 8y ago
The only myth is thinking that outages don't happen. If you look at actual reliability normalized over time, usage and service scale, Google is far better than what any small organization can do. It only seems like your organization can do better because you're nowhere near the same requirements. Whether this is a valid trade-off by taking on more risk and operational overhead is a judgement call but it rarely works out unless it's a business-critical service that you must control and can afford the resources to do so.

frenchman99 8y ago

I don't know about Google, but we've had some issues with our Git hosting service last year. At times, it paralyzed us so much that we had to move to another one, which was not quite as good but more stable at the time.