11 ms·

Just got this from the ProtonMail team: > Dear ProtonMail user, Starting at around 4:30PM New York (10:30PM Zurich), Gmail suffered a global outage. A catast

Just got this from the ProtonMail team: > Dear ProtonMail user, Starting at around 4:30PM New York (10:30PM Zurich), Gmail suffered a global outage. A catastrophic failure at Gmail is causing emails sent to Gmail to permanently fail and bounce back. The error message from Gmail is the following: 550-5.1.1 The email account that you tried to reach does not exist. This is a global issue, and it impacts all email providers trying to send email to Gmail, not just ProtonMail. Because Gmail is sending a permanent failure, our mail servers will not automatically retry sending these messages (this is standard practice at all email services for handling permanent failures). We are closely monitoring the situation. At this time, little can be done until Google fixes the problem. We recommend attempting to resend the messages to Gmail users when Google has fixed the problem. You can find the latest status from Google's status page: https://www.google.com/appsstatus#hl=en&v=issue&sid=1&iid=a8b67908fadee664c68c240ff9f529ab https://www.google.com/appsstatus#hl=en&v=issue&sid=1&iid=a8... Best Regards, The ProtonMail Team

systemvoltage 6y ago
This feels like a cheap shot at Google. Shit happens, and they're not immune to it even if the servers are located in Zurich. Running a datacenter is no easy task.
- toomuchtodo 6y ago
  Being down is okay. Returning an error message that results in the data being thrown away instead of being requeued is not. Block incoming smtp connections until your app layer is fixed.
  - Sebguer 6y ago
    Yeah, this is a particularly pernicious failure given how email works. Many mailing providers will just mark these as blacklisted, now, and lots of unsophisticated users likely won't notice.
    - croon 6y ago
      I consider myself sophisticated enough, but my Bitwarden has 700 accounts, of which ~30% old ones are registered with a gmail address, and the rest are handled behind g suite. Granted that last bit might be partly my fault, even though I paid for it. But even for a "sophisticated" user, I have no easy way of knowing if any of these accounts have silently failed to function now, other than by the passage of time and eventually finding out.
      - Sebguer 6y ago
        Oh, absolutely, even for sophisticated users mitigating may be difficult or impossible depending on exactly what bounced and how. But you at least are aware that this happened, and that you have a problem. Think how many people are out there with no clue what this error meant, or that it signaled an ecosystem problem, or that just had hundreds or thousands of emails silently bounce and unsubscribe.
  - dathinab 6y ago
    > Block incoming smtp connections until you app layer is fixed. Or returning one of the 4xx status codes which indicate less-permanent failure state like: - 451 Requested action aborted: local error in processing Which is kinda like a HTTP internal server error as it can mean anything.
    - toomuchtodo 6y ago
      For my comment’s purposes, I assume if this was possible with a flag or config setting (and the code path existed), it would’ve already been done. Doesn’t seem like they can, so they should’ve pulled the handbrake and gone “full stop” without throwing everyone’s mail away (hence blocking incoming connections and let the mail sit in all of the external MTA queues). Another option would’ve been to accept everything with a very lightweight smtp ingest service, journal it all, and play it back to the full frontend after their code fix was pushed out. Not an SRE so ¯\_(ツ)_/¯ just some thoughts from my time in a similar role and similar pain points (but thankfully not at this scale)
- jolmg 6y ago
  I see it as Protonmail explaining to their users that the failure is not on their end and why they can't do much about a remedy. Seems purely factual. A cheap shot would be generalizing from the event, but I don't see them doing that.
  - systemvoltage 6y ago
    I think I got this completely wrong. What you and other responses are saying makes sense.
- the_duke 6y ago
  A lot of people and companies use Gmail. Email providers are definitely getting support requests from users that don't know what's going on. This is not a cheap shot, but a message to inform users that it's an issue with Google that Protonmail can do nothing about.
- dathinab 6y ago
  More like "if mail to gmail fails it's not us so pleas don't flood the support with comaplains". > Running a datacenter is no easy task. Sure, but then there are very view companies which have more experience with running data-centers and (normally) providing reliable email service. So any outage for more then just a short time is very unusual. I'm really interested what went wrong.
dredmorbius 6y ago
Confirmed likewise.
octoberfranklin 6y ago
This is the Nightmare Scenario for mailing lists. Many of them auto-unsubscribe after a bounce.
- rob-olmos 6y ago
  Great point. And potentially email delivery services that have auto-suppression lists to protect reputation, at least they might be able to remove entries on behalf of their customers.
- dataflow 6y ago
  I said this in another comment but this seems like a naive way to react to an "address does not exist error" that they've already delivered to before. The only legit scenario in which that happens is when the user deletes the address, which is a rare event (pretty much always <= 1 time in the lifetime of any address), and there shouldn't be anything wrong with treating that kind of situation the same as any soft error. If you're wrong, your mail will just get rejected a few more times anyway, and you'll know it's genuinely a dead end. The underlying issue (wherever this occurs) seems to be lack of nuance regarding error codes when people try to implement robust systems. Different codes imply different things and shouldn't all just fall back into generic buckets.
  - mcqueenjordan 6y ago
    Good callout. The underlying issue of the lack of nuance is probably /state/. Being more nuanced about these errors probably requires managing state, which tends to increase the complexity and scaling challenges.
    - mehrdadn 6y ago
      Right. In this case it's already pretty typical for mailing lists to track bounces and retry under some errors, so I imagined that part is mostly done, and the missing piece would be taking more care in checking the error conditions.
    - throwaway201103 6y ago
CapriciousCptl 6y ago
Interesting response. And spot on from the technical integrity side. It’s also more fair to email providers as a whole to treat them all the same and respect their error messages. I mean, maybe there’s even requirements in some jurisdictions to deal with the address not found error in a specific way. As an email sender I think I’d prefer the message get auto re-sent after Gmail comes back online though.
karlzt 6y ago
>P.S. You might also consider asking your contacts who are still using Gmail to switch to ProtonMail for more private communications
lqet 6y ago
> Because Gmail is sending a permanent failure, our mail servers will not automatically retry sending these messages (this is standard practice at all email services for handling permanent failures). I fear that this will lead to many lost mails. In my experience, users are often confused by the technical "Mail delivery failed" mails and tend to ignore them or write them off as spam.

systemvoltage 6y ago

This feels like a cheap shot at Google. Shit happens, and they're not immune to it even if the servers are located in Zurich. Running a datacenter is no easy task.

toomuchtodo 6y ago
Being down is okay. Returning an error message that results in the data being thrown away instead of being requeued is not. Block incoming smtp connections until your app layer is fixed.
- Sebguer 6y ago
  Yeah, this is a particularly pernicious failure given how email works. Many mailing providers will just mark these as blacklisted, now, and lots of unsophisticated users likely won't notice.
  - croon 6y ago
    I consider myself sophisticated enough, but my Bitwarden has 700 accounts, of which ~30% old ones are registered with a gmail address, and the rest are handled behind g suite. Granted that last bit might be partly my fault, even though I paid for it. But even for a "sophisticated" user, I have no easy way of knowing if any of these accounts have silently failed to function now, other than by the passage of time and eventually finding out.
    - Sebguer 6y ago
      Oh, absolutely, even for sophisticated users mitigating may be difficult or impossible depending on exactly what bounced and how. But you at least are aware that this happened, and that you have a problem. Think how many people are out there with no clue what this error meant, or that it signaled an ecosystem problem, or that just had hundreds or thousands of emails silently bounce and unsubscribe.
- dathinab 6y ago
  > Block incoming smtp connections until you app layer is fixed. Or returning one of the 4xx status codes which indicate less-permanent failure state like: - 451 Requested action aborted: local error in processing Which is kinda like a HTTP internal server error as it can mean anything.
  - toomuchtodo 6y ago
    For my comment’s purposes, I assume if this was possible with a flag or config setting (and the code path existed), it would’ve already been done. Doesn’t seem like they can, so they should’ve pulled the handbrake and gone “full stop” without throwing everyone’s mail away (hence blocking incoming connections and let the mail sit in all of the external MTA queues). Another option would’ve been to accept everything with a very lightweight smtp ingest service, journal it all, and play it back to the full frontend after their code fix was pushed out. Not an SRE so ¯\_(ツ)_/¯ just some thoughts from my time in a similar role and similar pain points (but thankfully not at this scale)
jolmg 6y ago
I see it as Protonmail explaining to their users that the failure is not on their end and why they can't do much about a remedy. Seems purely factual. A cheap shot would be generalizing from the event, but I don't see them doing that.
- systemvoltage 6y ago
  I think I got this completely wrong. What you and other responses are saying makes sense.
the_duke 6y ago
A lot of people and companies use Gmail. Email providers are definitely getting support requests from users that don't know what's going on. This is not a cheap shot, but a message to inform users that it's an issue with Google that Protonmail can do nothing about.
dathinab 6y ago
More like "if mail to gmail fails it's not us so pleas don't flood the support with comaplains". > Running a datacenter is no easy task. Sure, but then there are very view companies which have more experience with running data-centers and (normally) providing reliable email service. So any outage for more then just a short time is very unusual. I'm really interested what went wrong.

dredmorbius 6y ago

Confirmed likewise.

octoberfranklin 6y ago

This is the Nightmare Scenario for mailing lists. Many of them auto-unsubscribe after a bounce.

rob-olmos 6y ago
Great point. And potentially email delivery services that have auto-suppression lists to protect reputation, at least they might be able to remove entries on behalf of their customers.
dataflow 6y ago
I said this in another comment but this seems like a naive way to react to an "address does not exist error" that they've already delivered to before. The only legit scenario in which that happens is when the user deletes the address, which is a rare event (pretty much always <= 1 time in the lifetime of any address), and there shouldn't be anything wrong with treating that kind of situation the same as any soft error. If you're wrong, your mail will just get rejected a few more times anyway, and you'll know it's genuinely a dead end. The underlying issue (wherever this occurs) seems to be lack of nuance regarding error codes when people try to implement robust systems. Different codes imply different things and shouldn't all just fall back into generic buckets.
- mcqueenjordan 6y ago
  Good callout. The underlying issue of the lack of nuance is probably /state/. Being more nuanced about these errors probably requires managing state, which tends to increase the complexity and scaling challenges.
  - mehrdadn 6y ago
    Right. In this case it's already pretty typical for mailing lists to track bounces and retry under some errors, so I imagined that part is mostly done, and the missing piece would be taking more care in checking the error conditions.
  - throwaway201103 6y ago

CapriciousCptl 6y ago

Interesting response. And spot on from the technical integrity side. It’s also more fair to email providers as a whole to treat them all the same and respect their error messages. I mean, maybe there’s even requirements in some jurisdictions to deal with the address not found error in a specific way. As an email sender I think I’d prefer the message get auto re-sent after Gmail comes back online though.

karlzt 6y ago

>P.S. You might also consider asking your contacts who are still using Gmail to switch to ProtonMail for more private communications

lqet 6y ago

> Because Gmail is sending a permanent failure, our mail servers will not automatically retry sending these messages (this is standard practice at all email services for handling permanent failures). I fear that this will lead to many lost mails. In my experience, users are often confused by the technical "Mail delivery failed" mails and tend to ignore them or write them off as spam.