6 ms·

I'm skimming this but so far I still don't understand how any kind of authentication failure can or should lead to an SMTP server returning "this address doesn'

I'm skimming this but so far I still don't understand how any kind of authentication failure can or should lead to an SMTP server returning "this address doesn't exist". Does anyone see an explanation? Edit 1: Oh wow I didn't even realize there were multiple incidents. Thanks! Edit 2 (after reading the Gmail post-mortem): AH! It was a messed up domain name! When this event happened, I said senders need to avoid taking "invalid address" at face value when they've recently succeeded delivering to the same addresses. But despite the RFC saying senders "should not" repeat requests (rather than "must not"), many people had a lot of resistance to this idea, and instead just blamed Google for messing up implementing the RFC. People seemed to think every other server was right to treat this as a permanent error. But this post-mortem makes it crystal clear how that completely missed the point, and in fact wasn't the case at all. The part of the service concluding "address not found" was working correctly! They didn't mess up implementing the spec—they messed up their domain name in the configuration. Which is an operational error you just can't assume will never happen—it's exactly like the earlier analogy to someone opening the door to the mailman and not recognizing the name of a resident. A robust system (in this case the sender) needs to be able to recognize sudden anomalies—in this case, failure to deliver to an address that was accepting mail very recently—and retry things a bit later. You just can't assume nobody will make mistake operational mistakes, even if you somehow assume the software is bug-free.

klysm 6y ago
If I had to guess I would say sometimes when you request something you aren't authorized to see you get a 404 because they don't want you to be able to tell what exists or not without any creds.
- mehrdadn 6y ago
  Sending an email doesn't require "seeing" anything other than whether the server is willing to receive email for that address though, right? Also, that problem occurs due to lack of authorization, not due to authentication failure, right? Authentication failure is a different kind of error to the client—if the server fails to authenticate you, clearly you already know that it's not going to show you anything?
jeffbee 6y ago
That's a different outage. SMTP 550 was the next day.
- adrianmonk 6y ago
  Some outage information showing the SMTP 550 incident was at a different time: "Gmail - Service Details" ( https://www.google.com/appsstatus#hl=en&v=issue&sid=1&iid=a8b67908fadee664c68c240ff9f529ab https://www.google.com/appsstatus#hl=en&v=issue&sid=1&iid=a8...) shows 12/15/20 as the date. That links to "Google Cloud Issue Summary" "Gmail - 2020-12-14 and 2020-12-15" (https://static.googleusercontent.com/media/www.google.com/en//appsstatus/ir/4et50yp2ckm8otv.pdf https://static.googleusercontent.com/media/www.google.com/en...) which mentions the 550 error code.
ryanobjc 6y ago
You ask the central database of accounts “does this exist?” And if it doesn’t say yes you bounce the email. Obviously an error condition should not result in this. But complex systems. It happens.
antoinealb 6y ago
It is not the same incident. This was for the global outage of google, not for the Gmail incident.
echelon 6y ago
This is a massive distributed system. Can't get quota? Certain pieces accidentally turn "TooManyRequests / 429s" or related semantics into 500s, 400s, 401s, etc. as you percolate upstream. Auth is one of the most central components of any system, so there would be cascading failures everywhere.
jrockway 6y ago
Different outage. The Gmail postmortem is linked in another thread, but the gist was that "gmail.com" is a configuration value that can be changed at runtime, and someone changed the configuration. Thus, *@gmail.com stopped being a valid address, and they returned "that mailbox is unavailable". If you don't want to scroll to the other thread, here's the postmortem: https://static.googleusercontent.com/media/www.google.com/en//appsstatus/ir/4et50yp2ckm8otv.pdf https://static.googleusercontent.com/media/www.google.com/en...
- deleted 6y ago
  [deleted]
- deleted 6y ago
  [deleted]
- cpncrunch 6y ago
  Indeed, it seems to be a completely unrelated issue. Two major Google outages on the same day for different reasons.
  - maxander 6y ago
    The odds that they would happen simultaneously if they’re completely unrelated seem astronomically small, certainly? Both are noted as being related to “ongoing migrations,” though AFAICT not related ones. I would bet there’s a human factor connection- e.g., the day before there was a big meeting where a higher-level management gave multiple ops teams go-ahead on their respective plans, resulting in a multiple potentially breaking changes occurring at the same time.
    - NoodleIncident 6y ago
      I think it's as simple as a case of the Mondays; you wouldn't roll out a migration like that on a Friday or the weekend, and rolling it out on a Monday gives you the least chances of problem occurring on those dates.
      - 6y ago
bbarnett 6y ago
This is how mail works. Follow the RFC or you're wrong. https://tools.ietf.org/html/rfc5321 https://tools.ietf.org/html/rfc5321 I haven't read the article, but SMTP response codes are very specific. If there were SMTP response codes in the 5xx range, that's a perm failure, end of story. Temp fail messages, 4xx, can be used for 'try later'. That's how SMTP works. What you are suggesting would break email.
- mehrdadn 6y ago
  > What you are suggesting would break email. No it would not. Firstly, because you would just rejected twice; the sky wouldn't fall down. Secondly, because the RFC in fact permits this too; it's within spec even if you want to follow it blindly to the letter with zero consideration for the context. People already litigated this in the earlier discussion and there's no point rehashing it so I'll just leave it at this.
  - bbarnett 6y ago
    Rejected twice? When do you retry? 5 seconds later? 5 minutes? An hour? When is the message requeued to send? When do you stop trying, when the remote side has already said "Hey, this user has no account here!". Meanwhile, the end user sent a message to bod@ by accident, instead of bob@, and your mail server keeps retrying to send mail, even when the remote mail server said "Hey! That account doesn't exist!". It's a typo, but you've decided to 'make things better', so now, until that bounce happens, the end user won't find out they made that mistake. Things work as they do for a reason, and the people saying 5xx means bounce! are right.
    - growse 6y ago
      Generally speaking, if you're trying to write a reliable distributed system (and email is a massively distributed system), a good principle to follow is to retry on failure, no matter the failure. Obviously there are edge cases, and obviously you don't just retry every 1ms forever, but to assume that an error that comes back from a system you've called once is both (a) the product of a 100% working non-faulty system and/or (b) not in any way a transitory issue or failure is naive. This type of thinking is how we end up with brittle systems. A cautious implementation might see an SMTP 500 and then may just choose to try it again once or twice a little later, to see exactly how 'permanent' that failure is. What the "but the RFC says!" people seem to forget is that the spec is only meaningful when the system is working correctly. Good engineering never assumes that.
mook 6y ago
My understanding from the other threads about the Gmail outage is that Google will assume you're a spammer if you trigger no such mailbox too often (because they think you're generating random addresses and trying them), so they have conditioned other operators to permanently drop such addresses. So they kind of created the whole situation with the opaque spam filter system…
- mehrdadn 6y ago
  That would make more sense (for others) if that's the case, but I'm kind of skeptical because it would mean they've massively screwed up on that front against their own interests. I would think they would have the common sense to understand that a couple re-delivery attempts to an account that was recently closed is clearly not someone trying to spam.

klysm 6y ago

If I had to guess I would say sometimes when you request something you aren't authorized to see you get a 404 because they don't want you to be able to tell what exists or not without any creds.

mehrdadn 6y ago
Sending an email doesn't require "seeing" anything other than whether the server is willing to receive email for that address though, right? Also, that problem occurs due to lack of authorization, not due to authentication failure, right? Authentication failure is a different kind of error to the client—if the server fails to authenticate you, clearly you already know that it's not going to show you anything?

jeffbee 6y ago

That's a different outage. SMTP 550 was the next day.

adrianmonk 6y ago
Some outage information showing the SMTP 550 incident was at a different time: "Gmail - Service Details" ( https://www.google.com/appsstatus#hl=en&v=issue&sid=1&iid=a8b67908fadee664c68c240ff9f529ab https://www.google.com/appsstatus#hl=en&v=issue&sid=1&iid=a8...) shows 12/15/20 as the date. That links to "Google Cloud Issue Summary" "Gmail - 2020-12-14 and 2020-12-15" (https://static.googleusercontent.com/media/www.google.com/en//appsstatus/ir/4et50yp2ckm8otv.pdf https://static.googleusercontent.com/media/www.google.com/en...) which mentions the 550 error code.

ryanobjc 6y ago

You ask the central database of accounts “does this exist?” And if it doesn’t say yes you bounce the email. Obviously an error condition should not result in this. But complex systems. It happens.

antoinealb 6y ago

It is not the same incident. This was for the global outage of google, not for the Gmail incident.

echelon 6y ago

This is a massive distributed system. Can't get quota? Certain pieces accidentally turn "TooManyRequests / 429s" or related semantics into 500s, 400s, 401s, etc. as you percolate upstream. Auth is one of the most central components of any system, so there would be cascading failures everywhere.

jrockway 6y ago

Different outage. The Gmail postmortem is linked in another thread, but the gist was that "gmail.com" is a configuration value that can be changed at runtime, and someone changed the configuration. Thus, *@gmail.com stopped being a valid address, and they returned "that mailbox is unavailable". If you don't want to scroll to the other thread, here's the postmortem: https://static.googleusercontent.com/media/www.google.com/en//appsstatus/ir/4et50yp2ckm8otv.pdf https://static.googleusercontent.com/media/www.google.com/en...

deleted 6y ago
[deleted]
deleted 6y ago
[deleted]
cpncrunch 6y ago
Indeed, it seems to be a completely unrelated issue. Two major Google outages on the same day for different reasons.
- maxander 6y ago
  The odds that they would happen simultaneously if they’re completely unrelated seem astronomically small, certainly? Both are noted as being related to “ongoing migrations,” though AFAICT not related ones. I would bet there’s a human factor connection- e.g., the day before there was a big meeting where a higher-level management gave multiple ops teams go-ahead on their respective plans, resulting in a multiple potentially breaking changes occurring at the same time.
  - NoodleIncident 6y ago
    I think it's as simple as a case of the Mondays; you wouldn't roll out a migration like that on a Friday or the weekend, and rolling it out on a Monday gives you the least chances of problem occurring on those dates.
    - 6y ago

bbarnett 6y ago

This is how mail works. Follow the RFC or you're wrong. https://tools.ietf.org/html/rfc5321 https://tools.ietf.org/html/rfc5321 I haven't read the article, but SMTP response codes are very specific. If there were SMTP response codes in the 5xx range, that's a perm failure, end of story. Temp fail messages, 4xx, can be used for 'try later'. That's how SMTP works. What you are suggesting would break email.

mehrdadn 6y ago
> What you are suggesting would break email. No it would not. Firstly, because you would just rejected twice; the sky wouldn't fall down. Secondly, because the RFC in fact permits this too; it's within spec even if you want to follow it blindly to the letter with zero consideration for the context. People already litigated this in the earlier discussion and there's no point rehashing it so I'll just leave it at this.
- bbarnett 6y ago
  Rejected twice? When do you retry? 5 seconds later? 5 minutes? An hour? When is the message requeued to send? When do you stop trying, when the remote side has already said "Hey, this user has no account here!". Meanwhile, the end user sent a message to bod@ by accident, instead of bob@, and your mail server keeps retrying to send mail, even when the remote mail server said "Hey! That account doesn't exist!". It's a typo, but you've decided to 'make things better', so now, until that bounce happens, the end user won't find out they made that mistake. Things work as they do for a reason, and the people saying 5xx means bounce! are right.
  - growse 6y ago
    Generally speaking, if you're trying to write a reliable distributed system (and email is a massively distributed system), a good principle to follow is to retry on failure, no matter the failure. Obviously there are edge cases, and obviously you don't just retry every 1ms forever, but to assume that an error that comes back from a system you've called once is both (a) the product of a 100% working non-faulty system and/or (b) not in any way a transitory issue or failure is naive. This type of thinking is how we end up with brittle systems. A cautious implementation might see an SMTP 500 and then may just choose to try it again once or twice a little later, to see exactly how 'permanent' that failure is. What the "but the RFC says!" people seem to forget is that the spec is only meaningful when the system is working correctly. Good engineering never assumes that.

mook 6y ago

My understanding from the other threads about the Gmail outage is that Google will assume you're a spammer if you trigger no such mailbox too often (because they think you're generating random addresses and trying them), so they have conditioned other operators to permanently drop such addresses. So they kind of created the whole situation with the opaque spam filter system…

mehrdadn 6y ago
That would make more sense (for others) if that's the case, but I'm kind of skeptical because it would mean they've massively screwed up on that front against their own interests. I would think they would have the common sense to understand that a couple re-delivery attempts to an account that was recently closed is clearly not someone trying to spam.