17 ms·

Gmail Services Global Outage

272 points by nedsma 8y ago

nkozyra 8y ago
Strange to get a 404 error there. This happens once or twice a year with Gmail, does it not?
zerop 8y ago
Have been consistently getting 404 on Gmail post login. App works fine.
theduality 8y ago
Issue affects my GMail account, but thankfully not any of my GSuite accounts.
nedsma 8y ago
Signed in accounts are not affected. However, if you sign out and attempt to sign in again, you'll get 404.
- cristianbica 8y ago
  Woraround is going to https://mail.google.com/mail?labs=0 https://mail.google.com/mail?labs=0
Felz 8y ago
As someone working on a mail service of my own, this is heartening to see. Even Google messes up sometimes.
- Waterluvian 8y ago
  Yup. And that's by design. Stability is asymptotic and increasingly expensive to reach toward 100%. There's a very intentional "this is good enough" point.
  - x3tm 8y ago
    What are the main factors that could lead to such a situation?
    - taneq 8y ago
      Well, the "good enough" point is reached when the cost of improving the reliability of the service is more than people would pay for that improved reliability.
    - gizzlon 8y ago
      From the first SRE book [1]: "The error budget stems from the observation that 100% is the wrong reliability target for basically everything (pacemakers and anti-lock brakes being notable exceptions). In general, for any software service or system, 100% is not the right reliability target because no user can tell the difference between a system being 100% available and 99.999% available. There are many other systems in the path between user and service (their laptop, their home WiFi, their ISP, the power grid…) and those systems collectively are far less than 99.999% available. Thus, the marginal difference between 99.999% and 100% gets lost in the noise of other unavailability, and the user receives no benefit from the enormous effort required to add that last 0.001% of availability. If 100% is the wrong reliability target for a system, what, then, is the right reliability target for the system? This actually isn’t a technical question at all—it’s a product question.." [1] https://landing.google.com/sre/sre-book/chapters/introduction/ https://landing.google.com/sre/sre-book/chapters/introductio... https://landing.google.com/sre/books/ https://landing.google.com/sre/books/
      - SilasX 8y ago
        Please, please stop using mono space for quotes. It’s hard to read on small screens. Just > is better.
        codetrotter 8y ago
        It’s not hard. It’s actually impossible. I agree, just use >. If I worked for YC as a HN mod I would literally spend a bit of time every day to review as many mono space-using comments I could every day and edit them to use > instead.
- nabla9 8y ago
  Gmail has had almost four nines availability at least in some years and no scheduled downtime. That's still a hour of downtime per year. Generally the availability of web services for individual users seems to be between 99.5 - 99.9 depending on the service. Too high availability target wastes resources and network failures and third parties limit what actual effective availability can be.
- maxtheman 8y ago
  What service are you working on? I'd be interested in learning about it.
  - Felz 8y ago
    It's not ready for release yet (in a week or two maybe), but I'm working off the idea of a pay-per-use email service. Instead of a flat per month fee, you're charged for resources used. It looks like it'll end up being about an order of magnitude cheaper than other paid email services so far. Hopefully from there it'll be a good base to try out other improvements to email too.
mattlondon 8y ago
Does not appear to be any problems today in Switzerland. "Global" might be overstating it? Edit: actual status page says "significant subset of users": https://www.google.com/appsstatus#hl=en-GB&v=status https://www.google.com/appsstatus#hl=en-GB&v=status
- puzzle 8y ago
  Given how many services it affected and the mention of 404 errors, I would suspect a GFE bug or bad configuration that started rolling out worldwide (hence the geographically diverse, but not 100% spread of the issue). A decade ago, Tuesdays were a special day for GFEs, but that hasn't been the case for years now. Perhaps it's just the Tuesday curse that persists. :-) My money is either on that or the static content service.
  - bamboozled 8y ago
    GFE?
    - chx 8y ago
      https://medium.com/@jerub/the-production-environment-at-google-part-2-610884268aaa https://medium.com/@jerub/the-production-environment-at-goog... > The browser connects to the HTTP server on this IP. This server (named the Google Frontend, or GFE) is a reverse proxy that terminates the TCP connection (2). The GFE looks up which service is required (web search, maps
    - skj 8y ago
      "Google Front End", the process directly between the open internet and internal Google services.
    - kerng 8y ago
      Historically Google terminated all security at that front door layer- hope that changed by now. But interestingly a cert issue down the pipe could easily be the root cause.
jatsign 8y ago
Got into gmail, but youtube login gives me a 404.
rusk 8y ago
No issues here ...
sidcool 8y ago
No problems for GSuite users in India.
gtt 8y ago
Semi-related, but can anyone suggest good open source email client? I'm working on several computers with macos/linux and Gmail isn't slow only on my deep learning rig, so I'm looking for something to replace it.
- wlesieutre 8y ago
  Basic HTML Gmail isn’t bad! https://support.google.com/mail/answer/15049?hl=en https://support.google.com/mail/answer/15049?hl=en
- phireal 8y ago
  Thunderbird works well for me. Interfaces with GMail IMAP. Has a nice quick filter and all the usuals (filtering, folders, labels). The calendar interfaces with your Google calendar too (with the Lightning and Provider for Google Calendar add-ons). Available for Mac, Windows and Linux.
- logicallee 8y ago
  Could you list the specifications (roughly) such as amount of RAM, processor (cores/speed/architecture or era) on your "deep learning" rig and on your "several computers"? I'm curious what it takes for the new gmail web interface to run smoothly without being frustrating.
- andyjohnson0 8y ago
  I've been happily using Thunderbird for 10+ years. Supports Windows, Mac, and Linux natively. https://www.thunderbird.net https://www.thunderbird.net
  - gtt 8y ago
    In my experience it crashes a few times a week and have abandoned feel. Is there any modern fork?
    - tethys 8y ago
      Mozilla is starting to throw resources at Thunderbird again, so it will hopefully get better this year. https://blog.mozilla.org/thunderbird/ https://blog.mozilla.org/thunderbird/
deleted 8y ago
[deleted]
irascible 8y ago
I'd like to take a moment to recognize how stable an experience gmail has been for me and how transformative a tool it has been for my life. Rock On you crazy Gmail elves, you!
ts330 8y ago
Well Microsoft had their annual mail outage the other day... I guess they could have coordinated a bit better to get them done and out of the way at the same time.
- jeromebaek 8y ago
  There was an outage on Outlook?
  - bovermyer 8y ago
    Office 365.
- wil421 8y ago
  Microsoft has had at least 2 major Azure outages that affected their SSO product. My system was unreachable by anyone but admins. Our systems engineer couldn’t do anything but wait on Microsoft to fix the issue on their end.
  - bharrison 8y ago
    We spent ~4 hours troubleshooting an Office365 deployment before realizing the SSO outage was the cause. The simultaneous fury and relief when we successfully logged users on the next day having made no functional changes was a watershed moment in our self-hosted services commitment.
- deleted 8y ago
  [deleted]
sneilan 8y ago
They just announced a price hike for personal accounts from 5 to 6 dollars per user too.
close04 8y ago
Global does not mean "affecting absolutely everyone around the globe (or disc, for the flatearthers out there)". Affecting a "significant subset of users" that are spread around every continent is global and it shows it's not one specific DC, link, or ISP that was affected. The fact that you didn't experience the disruption means absolutely nothing and it's ridiculous to even suggest it does. Especially since there are other reports from Switzerland flagging downtime. See the map in the link.
- mapcars 8y ago
  >or disc, for the flatearthers out there I mean, seriously? :)
  - yuy910616 8y ago
    i thought it was on an elephant's back on turtle's back?
    - basicplus2 8y ago
      Its turtles all the way down..
    - mont 8y ago
      4 elephants if we're being pedantic
      - egwynn 8y ago
        I personally think my own position would technically be “nitpicky” rather than “pedantic”.
      - wombatpm 8y ago
        Although at one time there was a fifth elephant
    - rbobby 8y ago
      Four: https://goo.gl/images/BfXfC7 https://goo.gl/images/BfXfC7 Cite: https://wiki.lspace.org/mediawiki/World_Elephant https://wiki.lspace.org/mediawiki/World_Elephant :)
steve1977 8y ago
I guess that's why they've just announced a price increase ;)
moviuro 8y ago
https://inbox.google.com https://inbox.google.com though programmed for termination looks unaffected here. Maybe worth a shot?
- TurningCanadian 8y ago
  I'm really going to miss Inbox. Its interface is so much cleaner than gmail's.
  - Drew_ 8y ago
    It's also been untouched by ads for its whole lifetime.
- vitaminCPP 8y ago
  Any inbox alternative ? (I can't believe I'm writing this... RIP Inbox, you will be missed.)
- tomerbd 8y ago
  i am going to continue navigating to inbox.google.com also after its termination no way i'm giving up.
zulgan 8y ago
We managed to centralize everything, email, git, even the web. I understand 99.99 looks fine, but is somewhat sad to see half the world without email.
- malchow 8y ago
  It would be nice to have an internet again.
  - jasonvorhe 8y ago
    They posted on the internet, using the internet.
- deleted 8y ago
  [deleted]
- airstrike 8y ago
  I'd be happy to move away from Gmail. Unfortunatelty, I happen to hate spam so that's not really an option.
  - ocdtrekkie 8y ago
    My Gmail account gets more spam in a week than I've gotten in the two years I've had FastMail. And Gmail's spam filter is surprisingly easy to defeat, considering. For months, putting "- -" at the start of the subject line would trick Gmail and put it in your normal inbox. (I just looked, and it looks like they've fixed it, but back in December it was bad.) This myth that nobody but Gmail can handle spam needs to die. It was true in 2006, but it is not true in 2019.
  - leokennis 8y ago
    I’m on Fastmail. Spam is filtered out perfectly and no false positives in 2 years of service. Make the move away from Google, don’t be scared and don’t be guided by false excuses.
    - 8y ago
franky_g 8y ago
No problems in Leeds UK
franky_g 8y ago
No problems here in Leeds, UK
bovermyer 8y ago
I am not having any problems accessing Gmail. However, I'm now worried that I might not be receiving email that I otherwise should be...
kyrra 8y ago
It's fixed now. Not sure the cause, but it seems to have only impacted GSutie related services? (not global to Google) https://www.google.com/appsstatus#hl=en&v=status https://www.google.com/appsstatus#hl=en&v=status
echofavor 8y ago
Thanks for sharing. You can also share your ideas for free on www.echofavor.com
steventhedev 8y ago
What are the chances this is related to the G+ phase out? The timing is a little too convenient.
postit 8y ago
I'm sure that wasn't correlated with my today's select 100k messages, mark then as read and archive all - but the timing was perfect
- jlawson 8y ago
  It was correlated - it wasn't caused. Sorry - grammar eye twitch!
  - deleted 8y ago
    [deleted]
- TrainedMonkey 8y ago
  Based on my extensive arguing on the internet experience, I can 100% say that you caused this outage. That caused mayhem and made a lot of people less happy, well done.
- bobbyi_settv 8y ago
  Just in case, next time you should manually read each of the 100k messages.
air7 8y ago
Ha. Apparently this site (outage.report) is used by scammers to lure victims. >Discussion| Please don't call "support numbers" posted below — most probably it's a scam. Make sure to report and "downvote" such posts.

272 points by nedsma 8y ago

nkozyra 8y ago

Strange to get a 404 error there. This happens once or twice a year with Gmail, does it not?

zerop 8y ago

Have been consistently getting 404 on Gmail post login. App works fine.

theduality 8y ago

Issue affects my GMail account, but thankfully not any of my GSuite accounts.

nedsma 8y ago

Signed in accounts are not affected. However, if you sign out and attempt to sign in again, you'll get 404.

cristianbica 8y ago
Woraround is going to https://mail.google.com/mail?labs=0 https://mail.google.com/mail?labs=0

Felz 8y ago

As someone working on a mail service of my own, this is heartening to see. Even Google messes up sometimes.

Waterluvian 8y ago
Yup. And that's by design. Stability is asymptotic and increasingly expensive to reach toward 100%. There's a very intentional "this is good enough" point.
- x3tm 8y ago
  What are the main factors that could lead to such a situation?
  - taneq 8y ago
    Well, the "good enough" point is reached when the cost of improving the reliability of the service is more than people would pay for that improved reliability.
  - gizzlon 8y ago
    From the first SRE book [1]: "The error budget stems from the observation that 100% is the wrong reliability target for basically everything (pacemakers and anti-lock brakes being notable exceptions). In general, for any software service or system, 100% is not the right reliability target because no user can tell the difference between a system being 100% available and 99.999% available. There are many other systems in the path between user and service (their laptop, their home WiFi, their ISP, the power grid…) and those systems collectively are far less than 99.999% available. Thus, the marginal difference between 99.999% and 100% gets lost in the noise of other unavailability, and the user receives no benefit from the enormous effort required to add that last 0.001% of availability. If 100% is the wrong reliability target for a system, what, then, is the right reliability target for the system? This actually isn’t a technical question at all—it’s a product question.." [1] https://landing.google.com/sre/sre-book/chapters/introduction/ https://landing.google.com/sre/sre-book/chapters/introductio... https://landing.google.com/sre/books/ https://landing.google.com/sre/books/
    - SilasX 8y ago
      Please, please stop using mono space for quotes. It’s hard to read on small screens. Just > is better.
      - codetrotter 8y ago
        It’s not hard. It’s actually impossible. I agree, just use >. If I worked for YC as a HN mod I would literally spend a bit of time every day to review as many mono space-using comments I could every day and edit them to use > instead.
nabla9 8y ago
Gmail has had almost four nines availability at least in some years and no scheduled downtime. That's still a hour of downtime per year. Generally the availability of web services for individual users seems to be between 99.5 - 99.9 depending on the service. Too high availability target wastes resources and network failures and third parties limit what actual effective availability can be.
maxtheman 8y ago
What service are you working on? I'd be interested in learning about it.
- Felz 8y ago
  It's not ready for release yet (in a week or two maybe), but I'm working off the idea of a pay-per-use email service. Instead of a flat per month fee, you're charged for resources used. It looks like it'll end up being about an order of magnitude cheaper than other paid email services so far. Hopefully from there it'll be a good base to try out other improvements to email too.

mattlondon 8y ago

Does not appear to be any problems today in Switzerland. "Global" might be overstating it? Edit: actual status page says "significant subset of users": https://www.google.com/appsstatus#hl=en-GB&v=status https://www.google.com/appsstatus#hl=en-GB&v=status

puzzle 8y ago
Given how many services it affected and the mention of 404 errors, I would suspect a GFE bug or bad configuration that started rolling out worldwide (hence the geographically diverse, but not 100% spread of the issue). A decade ago, Tuesdays were a special day for GFEs, but that hasn't been the case for years now. Perhaps it's just the Tuesday curse that persists. :-) My money is either on that or the static content service.
- bamboozled 8y ago
  GFE?
  - chx 8y ago
    https://medium.com/@jerub/the-production-environment-at-google-part-2-610884268aaa https://medium.com/@jerub/the-production-environment-at-goog... > The browser connects to the HTTP server on this IP. This server (named the Google Frontend, or GFE) is a reverse proxy that terminates the TCP connection (2). The GFE looks up which service is required (web search, maps
  - skj 8y ago
    "Google Front End", the process directly between the open internet and internal Google services.
  - kerng 8y ago
    Historically Google terminated all security at that front door layer- hope that changed by now. But interestingly a cert issue down the pipe could easily be the root cause.

jatsign 8y ago

Got into gmail, but youtube login gives me a 404.

rusk 8y ago

No issues here ...

sidcool 8y ago

No problems for GSuite users in India.

gtt 8y ago

Semi-related, but can anyone suggest good open source email client? I'm working on several computers with macos/linux and Gmail isn't slow only on my deep learning rig, so I'm looking for something to replace it.

wlesieutre 8y ago
Basic HTML Gmail isn’t bad! https://support.google.com/mail/answer/15049?hl=en https://support.google.com/mail/answer/15049?hl=en
phireal 8y ago
Thunderbird works well for me. Interfaces with GMail IMAP. Has a nice quick filter and all the usuals (filtering, folders, labels). The calendar interfaces with your Google calendar too (with the Lightning and Provider for Google Calendar add-ons). Available for Mac, Windows and Linux.
logicallee 8y ago
Could you list the specifications (roughly) such as amount of RAM, processor (cores/speed/architecture or era) on your "deep learning" rig and on your "several computers"? I'm curious what it takes for the new gmail web interface to run smoothly without being frustrating.
andyjohnson0 8y ago
I've been happily using Thunderbird for 10+ years. Supports Windows, Mac, and Linux natively. https://www.thunderbird.net https://www.thunderbird.net
- gtt 8y ago
  In my experience it crashes a few times a week and have abandoned feel. Is there any modern fork?
  - tethys 8y ago
    Mozilla is starting to throw resources at Thunderbird again, so it will hopefully get better this year. https://blog.mozilla.org/thunderbird/ https://blog.mozilla.org/thunderbird/

deleted 8y ago

[deleted]

irascible 8y ago

I'd like to take a moment to recognize how stable an experience gmail has been for me and how transformative a tool it has been for my life. Rock On you crazy Gmail elves, you!

ts330 8y ago

Well Microsoft had their annual mail outage the other day... I guess they could have coordinated a bit better to get them done and out of the way at the same time.

jeromebaek 8y ago
There was an outage on Outlook?
- bovermyer 8y ago
  Office 365.
wil421 8y ago
Microsoft has had at least 2 major Azure outages that affected their SSO product. My system was unreachable by anyone but admins. Our systems engineer couldn’t do anything but wait on Microsoft to fix the issue on their end.
- bharrison 8y ago
  We spent ~4 hours troubleshooting an Office365 deployment before realizing the SSO outage was the cause. The simultaneous fury and relief when we successfully logged users on the next day having made no functional changes was a watershed moment in our self-hosted services commitment.
deleted 8y ago
[deleted]

sneilan 8y ago

They just announced a price hike for personal accounts from 5 to 6 dollars per user too.

close04 8y ago

Global does not mean "affecting absolutely everyone around the globe (or disc, for the flatearthers out there)". Affecting a "significant subset of users" that are spread around every continent is global and it shows it's not one specific DC, link, or ISP that was affected. The fact that you didn't experience the disruption means absolutely nothing and it's ridiculous to even suggest it does. Especially since there are other reports from Switzerland flagging downtime. See the map in the link.

mapcars 8y ago
>or disc, for the flatearthers out there I mean, seriously? :)
- yuy910616 8y ago
  i thought it was on an elephant's back on turtle's back?
  - basicplus2 8y ago
    Its turtles all the way down..
  - mont 8y ago
    4 elephants if we're being pedantic
    - egwynn 8y ago
      I personally think my own position would technically be “nitpicky” rather than “pedantic”.
    - wombatpm 8y ago
      Although at one time there was a fifth elephant
  - rbobby 8y ago
    Four: https://goo.gl/images/BfXfC7 https://goo.gl/images/BfXfC7 Cite: https://wiki.lspace.org/mediawiki/World_Elephant https://wiki.lspace.org/mediawiki/World_Elephant :)

steve1977 8y ago

I guess that's why they've just announced a price increase ;)

moviuro 8y ago

https://inbox.google.com https://inbox.google.com though programmed for termination looks unaffected here. Maybe worth a shot?

TurningCanadian 8y ago
I'm really going to miss Inbox. Its interface is so much cleaner than gmail's.
- Drew_ 8y ago
  It's also been untouched by ads for its whole lifetime.
vitaminCPP 8y ago
Any inbox alternative ? (I can't believe I'm writing this... RIP Inbox, you will be missed.)
tomerbd 8y ago
i am going to continue navigating to inbox.google.com also after its termination no way i'm giving up.

zulgan 8y ago

We managed to centralize everything, email, git, even the web. I understand 99.99 looks fine, but is somewhat sad to see half the world without email.

malchow 8y ago
It would be nice to have an internet again.
- jasonvorhe 8y ago
  They posted on the internet, using the internet.
deleted 8y ago
[deleted]
airstrike 8y ago
I'd be happy to move away from Gmail. Unfortunatelty, I happen to hate spam so that's not really an option.
- ocdtrekkie 8y ago
  My Gmail account gets more spam in a week than I've gotten in the two years I've had FastMail. And Gmail's spam filter is surprisingly easy to defeat, considering. For months, putting "- -" at the start of the subject line would trick Gmail and put it in your normal inbox. (I just looked, and it looks like they've fixed it, but back in December it was bad.) This myth that nobody but Gmail can handle spam needs to die. It was true in 2006, but it is not true in 2019.
- leokennis 8y ago
  I’m on Fastmail. Spam is filtered out perfectly and no false positives in 2 years of service. Make the move away from Google, don’t be scared and don’t be guided by false excuses.
  - 8y ago

franky_g 8y ago

No problems in Leeds UK

franky_g 8y ago

No problems here in Leeds, UK

bovermyer 8y ago

I am not having any problems accessing Gmail. However, I'm now worried that I might not be receiving email that I otherwise should be...

kyrra 8y ago

It's fixed now. Not sure the cause, but it seems to have only impacted GSutie related services? (not global to Google) https://www.google.com/appsstatus#hl=en&v=status https://www.google.com/appsstatus#hl=en&v=status

echofavor 8y ago

Thanks for sharing. You can also share your ideas for free on www.echofavor.com

steventhedev 8y ago

What are the chances this is related to the G+ phase out? The timing is a little too convenient.

postit 8y ago

I'm sure that wasn't correlated with my today's select 100k messages, mark then as read and archive all - but the timing was perfect

jlawson 8y ago
It was correlated - it wasn't caused. Sorry - grammar eye twitch!
- deleted 8y ago
  [deleted]
TrainedMonkey 8y ago
Based on my extensive arguing on the internet experience, I can 100% say that you caused this outage. That caused mayhem and made a lot of people less happy, well done.
bobbyi_settv 8y ago
Just in case, next time you should manually read each of the 100k messages.

air7 8y ago

Ha. Apparently this site (outage.report) is used by scammers to lure victims. >Discussion| Please don't call "support numbers" posted below — most probably it's a scam. Make sure to report and "downvote" such posts.