13 ms·

"Type: Permanent; SubType: General; Code: smtp; 550-5.1.1 The email account that you tried to reach does not exist. Please try 550-5.1.1 double-checking the rec

by wenbin 6y ago

"Type: Permanent; SubType: General; Code: smtp; 550-5.1.1 The email account that you tried to reach does not exist. Please try 550-5.1.1 double-checking the recipient's email address for typos or 550-5.1.1 unnecessary spaces. Learn more at 550 5.1.1 https://support.google.com/mail/?p=NoSuchUser https://support.google.com/mail/?p=NoSuchUser y128si147264pfg.177 - gsmtp"

com2kid 6y ago
This is pretty much the worst response possible. Hard bounces mean that email delivery services are going to start automatically removing, or at least stopping delivery to, entire slews of email addresses. A lot of clean up is going to be needed as a result of this. To add some more details, when using a 3rd party email delivery service, those services will either black-list or just outright remove email addresses when they get a hard bounce "email address no longer exists" message back. Some providers make re-adding an address after a hard bounce a non-trivial task, since after all, the authority on that email address just said it doesn't exist. This is going to be really ugly.
- petercooper 6y ago
  Yes. I email hundreds of thousands of Gmail users each week (yes, double opt in, they all want the mails!) and we immediately delete any user for whom any Gmail error comes up at all in order to keep a solid delivery record with them. Sounds like we might have deleted 80% of our list if we'd sent today..!
  - com2kid 6y ago
    My sanity tests started acting flaky ~3 hours ago, I never thought it was Gmail... Kind of happy I had to do something else and I didn't burn hours investigating.
  - ta988 6y ago
    Hope you have a backup just in case.
    - petercooper 6y ago
      Yes, we're unusual in not relying on third parties for list management. We can rollback. Or I might just comment out the 'unsub on hard bounce' code for the rest of the week..! :)
      - Jap2-0 6y ago
        Unsub on two consecutive bounces seems more reasonable to catch flukes (or Gmail going down)?
        mehrdadn 6y ago
        Logically you'd expect unsubscribe to only act after lots of bounces of this format when the address has been receiving mail fine before. It also seems reasonable not to trust such bounces for the entire domain for a while when this happens to lots of other addresses that have worked fine before. Not that I expect software currently works this way, but it does seem like a common sense thing to code in.
        petercooper 6y ago
        I mean, it's possible, but you'd need to queue up a day's worth of bounces, do the analysis, and then handle the bounces asynchronously later on to do that. Most systems operate more immediately in isolation on individual addresses than that right now, because such analysis is generally not needed (until today, of course ;-)).
        mehrdadn 6y ago
        Mail agents already queue emails that bounce though; it's a matter of changing the conditions for when you retry and/or unsubscribe. I imagine you can do the analysis in real time too... just look at the bounce and see if it pertains to an email you sent to in the past, and if so, increment some rolling counter for that domain.
        petercooper 6y ago
  - dathinab 6y ago
    So new think to do: Quarantine addresses instead of deleting them and if for one provider most addresses fail don't give them another (maybe manually triggered) try later one. (And if no such thing is detected deleted quarantined mail addresses.)
    - webbie917 6y ago
      My guess is that how most email service providers handle this - they don't actually delete the email and just have a flag on it - bounced, complain, unsub. This way the list owner can run an export and see all the status code.
- octoberfranklin 6y ago
  I really cannot believe they did not immediately hack in a new rule to their SMTP server: never return a 5xx (permanent failure), instead return a 421 (temporary failure try again later). That simple fix buys them 24-72 hours to solve this properly. Yeah, it burdens servers sending mail to them because now they have to hold on to all mail (including mail that really is permanently undeliverable) for another day or so, but that's still better than what's happening right now.
  - jeffbee 6y ago
    Why would that be better than just shutting off the delivery stack altogether?
    - shmoogy 6y ago
      5xx error results in suppression list addition of an email, so future emails won't be delivered (by most ESPs), and not returning MX response would probably be just as bad, or worse (or result in millions/billions of emails being re-queued due to timeouts?) His solution would result in exponential retry failures baked into most services, which would buy them a few hours, and result in no lost emails, and no suppression list additions.
      - jeffbee 6y ago
        Inability to contact the destination would be treated as a temp-failure by the origin, and taking the service off the air could be effected instantly.
      - pmlnr 6y ago
        Failure of response from the server is nearly always treated as temp failure, because it could be down to network connectivity, name resolution, etc. That is a better scenario, than 5xx.
    - octoberfranklin 6y ago
      In case less than 100% of gmail is experiencing this bug.
- marvion 6y ago
  Mailgun send a warning mail about increased bounces from our account. Sure, they know what's going on... but we send 4-5 digit mails per hour - it's a lot of bounces That means I can't just resend the the emails blindly, because I'm too scared to trigger some sort of automatic suspension... (I don't do this regularly, so I'm not familiar with all features... additional mail verification could help probably ....)
robarr 6y ago
Yes, hard bounces even between Gmail addresses.
- caogecym 6y ago
  just curious, how did you check bounces stats for Gmail?
LinuxBender 6y ago
They should be returning 421 for backend outages so that sending servers queue and retry the emails. 550 can be interpreted by some as deleted [1] or even banned accounts in some cases. Maybe someone here could convince them to change the logic that occurs during an outage. [1] - https://en.wikipedia.org/wiki/List_of_SMTP_server_return_codes#%E2%80%94_5yz_Permanent_negative_completion https://en.wikipedia.org/wiki/List_of_SMTP_server_return_cod...
- mlyle 6y ago
  Yah. Maybe there's an unexpected way that things can fail resulting in 550's. But maybe at Google's scale you should have some kind of kill switch to stop answering SMTP or to not send permanent errors at all, so that you could flip a switch and prevent the worst consequences of this rather than let it go on for a couple of hours.
  - deleted 6y ago
    [deleted]
  - octoberfranklin 6y ago
    Absolutely this. I am astonished that either (a) this switch has not been flipped yet or (b) this switch does not exist. Somebody is incompetent here.
    - mrich 6y ago
      Perhaps Gmail is just being discontinued ;)
      - octoberfranklin 6y ago
        don't get my hopes up!
awb 6y ago
Incoming Gmail is bouncing, but I'm still able to access all prior received messages.
da_allgeier 6y ago
TL;DR; Don't sent your newsletters today if you can avoid it.
jhawkinson 6y ago
It may be helpful to note that Google has acknowledged they are working on similar issues (the description is vague!) with an ETTR of 1900 EST: https://www.google.com/appsstatus#hl=en&v=issue&sid=1&iid=a8b67908fadee664c68c240ff9f529ab https://www.google.com/appsstatus#hl=en&v=issue&sid=1&iid=a8... On the other hand, their status dashboard reported similar issues yesterday and here we are again: https://www.google.com/appsstatus#hl=en&v=status https://www.google.com/appsstatus#hl=en&v=status
megous 6y ago
A lot of people will lose transactional email messages, because of this. I'd absolutely hate to be hit by this at this time. Thankfully I've made an time investment to run my own mail server years ago. A handful of times it broke down, it either went offline or started returning 4xx codes due to misconfigured or broken milter after an update. Neither meant lost messages from normal senders that use queuing MTAs.
- alphadevx 6y ago
  Same for me, mainly for privacy concerns. And I back it up daily to my local NAS. It's so easy to configure and run your own mail server, that I'm surprised we are the minority in the tech community.
  - snazz 6y ago
    > It's so easy to configure and run your own mail server Is it? Is dealing with IP reputation, getting your emails accepted by major providers, and being on the hook for fixing everything yourself very easy? I haven't tried, so I don't have personal experience, but I've heard enough horror stories to think that it's not a good use of my time.
    - EarthIsHome 6y ago
      I had an IP reputation issue and managed to resolve it after some time. TLDR: Before you spin up a mail server, check if your IP address is on any of the blacklists [0]-[1] as well as Proof Point's list [2]. If it is, then try and get a different IP address. I spun up a hosted server on Digital Ocean and received an IP address. I checked several black lists from a few email testing/troubleshooting sites [0] and [1] and all was groovy; my IP address wasn't on any list. I got a bunch of 521 bounces when I tried emailing a neighbor who had an att.net address. So, I checked the troubleshooting websites, and my IP address was listed as clean. My logs said I should forward the error to abuse_rbl@abuse-att.net, so I did. Those emails were never delivered, because abuse-att.net had its own blacklist. I was getting 553 errors. In the logs, the message from their server told me to check https://ipcheck.proofpoint.com https://ipcheck.proofpoint.com. Proof point runs their own blacklist that some enterprises use (e.g. att and apple [3]). I checked their list, and lo and behold, my IP address from Digital Ocean was blocked [2]. Digital Ocean wasn't able to remove the IP address from their blocklist and suggested I spin up a new droplet with a different IP address. I didn't want to do that, so I sent Proof Point an email that went unanswered; the email asked them to remove my IP address. I forgot about the issue for five or six months (this is a personal server), and ran into the issue again a few months ago. So I sent Proof Point an email again, this time with different wording emphasizing that "my clients" were having delivery issues. Within a day, they removed my IP address from their block list. So, my main suggestion is to check if your IP address is on any of the blacklists as well as Proof Point's list before you start on your server. If it is, then try and get a different IP address. Does anyone have more "enterprise" lists, like Proof Point, to check? [0]: https://www.mail-tester.com/ https://www.mail-tester.com/ [1]: https://mxtoolbox.com/blacklists.aspx https://mxtoolbox.com/blacklists.aspx [2]: https://ipcheck.proofpoint.com https://ipcheck.proofpoint.com [3]: https://www.reddit.com/r/email/comments/6toxzr/ip_blocked_by_proofpoint/ https://www.reddit.com/r/email/comments/6toxzr/ip_blocked_by...
clairegraham 6y ago
I also had the same hard bounce (when emailing from a non-gmail address -- fastmail -- to a gmail address). Sent it again minutes later and then it worked.

by wenbin 6y ago

com2kid 6y ago

This is pretty much the worst response possible. Hard bounces mean that email delivery services are going to start automatically removing, or at least stopping delivery to, entire slews of email addresses. A lot of clean up is going to be needed as a result of this. To add some more details, when using a 3rd party email delivery service, those services will either black-list or just outright remove email addresses when they get a hard bounce "email address no longer exists" message back. Some providers make re-adding an address after a hard bounce a non-trivial task, since after all, the authority on that email address just said it doesn't exist. This is going to be really ugly.

petercooper 6y ago
Yes. I email hundreds of thousands of Gmail users each week (yes, double opt in, they all want the mails!) and we immediately delete any user for whom any Gmail error comes up at all in order to keep a solid delivery record with them. Sounds like we might have deleted 80% of our list if we'd sent today..!
- com2kid 6y ago
  My sanity tests started acting flaky ~3 hours ago, I never thought it was Gmail... Kind of happy I had to do something else and I didn't burn hours investigating.
- ta988 6y ago
  Hope you have a backup just in case.
  - petercooper 6y ago
    Yes, we're unusual in not relying on third parties for list management. We can rollback. Or I might just comment out the 'unsub on hard bounce' code for the rest of the week..! :)
    - Jap2-0 6y ago
      Unsub on two consecutive bounces seems more reasonable to catch flukes (or Gmail going down)?
      - mehrdadn 6y ago
        Logically you'd expect unsubscribe to only act after lots of bounces of this format when the address has been receiving mail fine before. It also seems reasonable not to trust such bounces for the entire domain for a while when this happens to lots of other addresses that have worked fine before. Not that I expect software currently works this way, but it does seem like a common sense thing to code in.
        petercooper 6y ago
        I mean, it's possible, but you'd need to queue up a day's worth of bounces, do the analysis, and then handle the bounces asynchronously later on to do that. Most systems operate more immediately in isolation on individual addresses than that right now, because such analysis is generally not needed (until today, of course ;-)).
        mehrdadn 6y ago
        Mail agents already queue emails that bounce though; it's a matter of changing the conditions for when you retry and/or unsubscribe. I imagine you can do the analysis in real time too... just look at the bounce and see if it pertains to an email you sent to in the past, and if so, increment some rolling counter for that domain.
      - petercooper 6y ago
- dathinab 6y ago
  So new think to do: Quarantine addresses instead of deleting them and if for one provider most addresses fail don't give them another (maybe manually triggered) try later one. (And if no such thing is detected deleted quarantined mail addresses.)
  - webbie917 6y ago
    My guess is that how most email service providers handle this - they don't actually delete the email and just have a flag on it - bounced, complain, unsub. This way the list owner can run an export and see all the status code.
octoberfranklin 6y ago
I really cannot believe they did not immediately hack in a new rule to their SMTP server: never return a 5xx (permanent failure), instead return a 421 (temporary failure try again later). That simple fix buys them 24-72 hours to solve this properly. Yeah, it burdens servers sending mail to them because now they have to hold on to all mail (including mail that really is permanently undeliverable) for another day or so, but that's still better than what's happening right now.
- jeffbee 6y ago
  Why would that be better than just shutting off the delivery stack altogether?
  - shmoogy 6y ago
    5xx error results in suppression list addition of an email, so future emails won't be delivered (by most ESPs), and not returning MX response would probably be just as bad, or worse (or result in millions/billions of emails being re-queued due to timeouts?) His solution would result in exponential retry failures baked into most services, which would buy them a few hours, and result in no lost emails, and no suppression list additions.
    - jeffbee 6y ago
      Inability to contact the destination would be treated as a temp-failure by the origin, and taking the service off the air could be effected instantly.
    - pmlnr 6y ago
      Failure of response from the server is nearly always treated as temp failure, because it could be down to network connectivity, name resolution, etc. That is a better scenario, than 5xx.
  - octoberfranklin 6y ago
    In case less than 100% of gmail is experiencing this bug.
marvion 6y ago
Mailgun send a warning mail about increased bounces from our account. Sure, they know what's going on... but we send 4-5 digit mails per hour - it's a lot of bounces That means I can't just resend the the emails blindly, because I'm too scared to trigger some sort of automatic suspension... (I don't do this regularly, so I'm not familiar with all features... additional mail verification could help probably ....)

robarr 6y ago

Yes, hard bounces even between Gmail addresses.

caogecym 6y ago
just curious, how did you check bounces stats for Gmail?

LinuxBender 6y ago

They should be returning 421 for backend outages so that sending servers queue and retry the emails. 550 can be interpreted by some as deleted [1] or even banned accounts in some cases. Maybe someone here could convince them to change the logic that occurs during an outage. [1] - https://en.wikipedia.org/wiki/List_of_SMTP_server_return_codes#%E2%80%94_5yz_Permanent_negative_completion https://en.wikipedia.org/wiki/List_of_SMTP_server_return_cod...

mlyle 6y ago
Yah. Maybe there's an unexpected way that things can fail resulting in 550's. But maybe at Google's scale you should have some kind of kill switch to stop answering SMTP or to not send permanent errors at all, so that you could flip a switch and prevent the worst consequences of this rather than let it go on for a couple of hours.
- deleted 6y ago
  [deleted]
- octoberfranklin 6y ago
  Absolutely this. I am astonished that either (a) this switch has not been flipped yet or (b) this switch does not exist. Somebody is incompetent here.
  - mrich 6y ago
    Perhaps Gmail is just being discontinued ;)
    - octoberfranklin 6y ago
      don't get my hopes up!

awb 6y ago

Incoming Gmail is bouncing, but I'm still able to access all prior received messages.

da_allgeier 6y ago

TL;DR; Don't sent your newsletters today if you can avoid it.

jhawkinson 6y ago

It may be helpful to note that Google has acknowledged they are working on similar issues (the description is vague!) with an ETTR of 1900 EST: https://www.google.com/appsstatus#hl=en&v=issue&sid=1&iid=a8b67908fadee664c68c240ff9f529ab https://www.google.com/appsstatus#hl=en&v=issue&sid=1&iid=a8... On the other hand, their status dashboard reported similar issues yesterday and here we are again: https://www.google.com/appsstatus#hl=en&v=status https://www.google.com/appsstatus#hl=en&v=status

megous 6y ago

A lot of people will lose transactional email messages, because of this. I'd absolutely hate to be hit by this at this time. Thankfully I've made an time investment to run my own mail server years ago. A handful of times it broke down, it either went offline or started returning 4xx codes due to misconfigured or broken milter after an update. Neither meant lost messages from normal senders that use queuing MTAs.

alphadevx 6y ago
Same for me, mainly for privacy concerns. And I back it up daily to my local NAS. It's so easy to configure and run your own mail server, that I'm surprised we are the minority in the tech community.
- snazz 6y ago
  > It's so easy to configure and run your own mail server Is it? Is dealing with IP reputation, getting your emails accepted by major providers, and being on the hook for fixing everything yourself very easy? I haven't tried, so I don't have personal experience, but I've heard enough horror stories to think that it's not a good use of my time.
  - EarthIsHome 6y ago
    I had an IP reputation issue and managed to resolve it after some time. TLDR: Before you spin up a mail server, check if your IP address is on any of the blacklists [0]-[1] as well as Proof Point's list [2]. If it is, then try and get a different IP address. I spun up a hosted server on Digital Ocean and received an IP address. I checked several black lists from a few email testing/troubleshooting sites [0] and [1] and all was groovy; my IP address wasn't on any list. I got a bunch of 521 bounces when I tried emailing a neighbor who had an att.net address. So, I checked the troubleshooting websites, and my IP address was listed as clean. My logs said I should forward the error to abuse_rbl@abuse-att.net, so I did. Those emails were never delivered, because abuse-att.net had its own blacklist. I was getting 553 errors. In the logs, the message from their server told me to check https://ipcheck.proofpoint.com https://ipcheck.proofpoint.com. Proof point runs their own blacklist that some enterprises use (e.g. att and apple [3]). I checked their list, and lo and behold, my IP address from Digital Ocean was blocked [2]. Digital Ocean wasn't able to remove the IP address from their blocklist and suggested I spin up a new droplet with a different IP address. I didn't want to do that, so I sent Proof Point an email that went unanswered; the email asked them to remove my IP address. I forgot about the issue for five or six months (this is a personal server), and ran into the issue again a few months ago. So I sent Proof Point an email again, this time with different wording emphasizing that "my clients" were having delivery issues. Within a day, they removed my IP address from their block list. So, my main suggestion is to check if your IP address is on any of the blacklists as well as Proof Point's list before you start on your server. If it is, then try and get a different IP address. Does anyone have more "enterprise" lists, like Proof Point, to check? [0]: https://www.mail-tester.com/ https://www.mail-tester.com/ [1]: https://mxtoolbox.com/blacklists.aspx https://mxtoolbox.com/blacklists.aspx [2]: https://ipcheck.proofpoint.com https://ipcheck.proofpoint.com [3]: https://www.reddit.com/r/email/comments/6toxzr/ip_blocked_by_proofpoint/ https://www.reddit.com/r/email/comments/6toxzr/ip_blocked_by...

clairegraham 6y ago

I also had the same hard bounce (when emailing from a non-gmail address -- fastmail -- to a gmail address). Sent it again minutes later and then it worked.