7 ms·

The real problem is that SMTP is a "plain-text" protocol that includes in-band signaling. It literally happened because SMTP defined "a line that only contains

The real problem is that SMTP is a "plain-text" protocol that includes in-band signaling. It literally happened because SMTP defined "a line that only contains a single period in it" as a control sequence and not a literal line that only contains a single period in it. SMTP is an example of an unnecessarily complex design, and the implementation bugs reflect it. SMTP shouldn't be hard for someone to correctly implement by themselves (even though I agree that people shouldn't be re-inventing the wheel).

davedx 2y ago
Honestly this part with the periods, while unusual, isn’t really complex. The two rules regarding the periods were like a small paragraph of text. I agree with sibling comment from Temporal, this is purely a “skill issue”, not a protocol issue
- meisenhus 2y ago
  [dead]
pja 2y ago
At some point, all protocols include in-band signalling somewhere: You have to put packets on the line, and those packets are ultimately just a stream of anonymous bytes. If it wasn’t a period, it would be something else & you’d have to handle that instead.
- throw10920 2y ago
  > all protocols include in-band signalling somewhere That's an incredibly reductionistic view of the world that's utterly useless for anything (including actually engineering systems) except pedantry. It's obvious that the level at which you include control information is meaningful and significantly affects the design of the protocol, as we see in the submission. Directly embedding the control information into the message body does not lead to a design that is easy to implement. > If it wasn’t a period, it would be something else & you’d have to handle that instead. Yes, and there are many other design choices that'd be significantly easier to handle.
  - TeMPOraL 2y ago
    No, it's not reductionist and pedantic. It's a reminder that there is no magic. Building an abstraction layer that separates control and data doesn't win you anything if, like the people in the article, you then forget it's a thing and write directly to the level below it.
    - throw10920 2y ago
      > No, it's not reductionist and pedantic. It's very reductionistic, because it intentionally ignores meaningful detail, and it's pedantic because it's making a meaningless distinction. > It's a reminder that there is no magic. This is irrelevant. Nobody is claiming that there's any magic. I'm pointing out the true fact that details about the abstraction layers matter. In this case, the abstraction layer was poorly-designed. Good abstraction layer: length prefix, or JSON encoding. Bad abstraction layer: "the body of the email is mostly plain text, except when there's a line that only contains a single period". There are very, very few problems to which the latter is a good solution. It is a bad engineering decision, and it also obfuscates the fact that there even is an abstraction layer unless you carefully read the spec. ------------- In fact, the underlying problem goes deeper than that - the design of SMTP is intrinsically flawed because it's a text-based ad-hoc protocol that has in-band signaling. There are very few good reasons to use a text-based data interchange format. One of them is to make the format self-documenting, such that people can easily read and write it without consulting the spec. If the spec is complex enough that you get these ridiculous footguns, then it shouldn't be text-based in the first place. Instead, it should be binary - then you have to either read the spec or use someone else's implementation. Failing that, use a standardized structured format like XML or JSON. But there's no excuse for the brain-dead approach that SMTP took. They didn't even use length prefixing,
      - nottorp 2y ago
        Why not protobufs inside protobufs then?
      - kjellsbells 2y ago
        I dont disagree with your criticisms of SMTP, but reading those early RFCs (eg 772) is a reminder of what a wildly different place the Internet was back then, and in that light, I feel it only fair to grant some grace. MTP had one concern which was to get mail over to a host that stood a better chance of delivering it, where the total host pool was maybe a hundred nodes? I speculate that Postel and Sluizer were aware of alternatives and rejected them in favor of things that were easily implemented on highly diverse, low powered hardware. Not everyone had IBM-grade budgets after all. Alternative implementations of mail that did follow the kinds of precepts that you suggest existed at one time. X.400 is the obvious example. If I recall correctly, it did have rigorous protocol spec definitions, message length tags for every entity sent on the wire, bounds and limits on each PDU, the whole hog. It was also crushed by SMTP, and this was in the era when you needed to understand sendmail and its notoriously arcane config to do anything. So sometimes the technically worse solution just wins, and we are stuck with it.
- BoppreH 2y ago
  That's not a very useful definition of "in-band signaling". For me, the main difference is an out-of-band protocol that says: "The first two bytes represent the string length, in big-endian, followed by that many bytes presenting the string text." and an in-band signalling protocol: "The string is ended by a period and a newline." In the second one, you're indicating the end of the string from within the string. It looks simpler, but that's where accidents happen. Now you have to guarantee that the text never contains that control sequence, and you need an escaping method to represent the control sequence as part of the text.
- akvadrako 2y ago
  That isn't true at all. In most binary protocols, you put the size of the message in the header. Then any sequence of bytes is allowed to follow. There are no escape sequences.
  - pja 2y ago
    That’s still in-band signalling: The metadata is in the same channel as the data.
- treflop 2y ago
  From my experience, almost no protocols have in-band signaling. No protocol I’ve ever built has in-band signaling because it’s nuts. You always know what the next byte means because either you did a prefixed length, your protocol has stringent escaping rules, or you chose an obvious and consistent terminator like null.
  - yencabulator 2y ago
    SMTP has stringent escaping rules. The authors of the code in the article were incompetent.
  - pja 2y ago
    If your metadata about the data is in the same channel as the data then you’re doing in-band signalling.
    - treflop 2y ago
      Not in modern times. The terms harken back from the day of circuit switched networks but now that we have heavily transitioned to packets, bands are an artificial construct on top of packets and applying the term isn’t very clear cut. The main property of in-band data in the circuit-switched network days is that you could inject commands into your data stream. If we apply that criteria that to a modern protocol, even if you mix metadata and data in the same “band,” if your data can never be interpreted as commands then “out of band” makes an apt description. See https://en.m.wikipedia.org/wiki/Out-of-band_data https://en.m.wikipedia.org/wiki/Out-of-band_data
      - TeMPOraL 2y ago
        That's only true if you're not breaking the protocol abstraction layer. There is no "out-of-band" once you serialize your messages. If you start injecting random bytes into the data stream on the wire, you can absolutely start introducing commands, or confuse the receiver where the next piece of metadata/control is. In this case, somewhere the protocol abstraction layer got broken, and the message text ended up being treated as already serialized. It's not a problem with the protocol per se, but with bad implementation of its API (or no implementation at all, just printf-ing into the wire format).
Animats 2y ago
SMTP is that way because running the entire contents of a message through some escape character processing was expensive back then, in the era of 0.25 MIPS machines.
- throw10920 2y ago
  Thank you for weighing in, John Nagle! I understand that constraint, and it seems reasonable - but in that case, why not use a length prefix? That should be even more efficient than having to scan for a line containing a single period and nothing else.
  - Animats 2y ago
    Because many paths of that era were not binary-transparent. There was CR-LF to LF conversion, the possibility of ASCII/EBCDIC translation, and other transformations. Hence such horrors as MIME delimiters: -----=_Part_2827761_1947716067.1716352583971 Content-Type: multipart/alternative; boundary="----=_Part_2827760_1372041171.1716352583736" We still have the mess that is the required and standardized behavior of HTML5 parsers faced with bad data.
    - throw10920 2y ago
      That's genuinely horrifying, but it explains why SMTP made a decision that I would otherwise categorize as "insane" - it was forced to by constraints of the time. Thank you for explaining!
eviks 2y ago
Why didn't they use actual control sequences instead of text chars, ASCII had plenty of those?
- inejge 2y ago
  When your debugger is telnet circa 1980, non-printable characters are a liability. (You pay for it later, when people who've never seen "...end with <CRLF>.<CRLF>" botch the implementation.)

davedx 2y ago

Honestly this part with the periods, while unusual, isn’t really complex. The two rules regarding the periods were like a small paragraph of text. I agree with sibling comment from Temporal, this is purely a “skill issue”, not a protocol issue

meisenhus 2y ago
[dead]

pja 2y ago

At some point, all protocols include in-band signalling somewhere: You have to put packets on the line, and those packets are ultimately just a stream of anonymous bytes. If it wasn’t a period, it would be something else & you’d have to handle that instead.

throw10920 2y ago
> all protocols include in-band signalling somewhere That's an incredibly reductionistic view of the world that's utterly useless for anything (including actually engineering systems) except pedantry. It's obvious that the level at which you include control information is meaningful and significantly affects the design of the protocol, as we see in the submission. Directly embedding the control information into the message body does not lead to a design that is easy to implement. > If it wasn’t a period, it would be something else & you’d have to handle that instead. Yes, and there are many other design choices that'd be significantly easier to handle.
- TeMPOraL 2y ago
  No, it's not reductionist and pedantic. It's a reminder that there is no magic. Building an abstraction layer that separates control and data doesn't win you anything if, like the people in the article, you then forget it's a thing and write directly to the level below it.
  - throw10920 2y ago
    > No, it's not reductionist and pedantic. It's very reductionistic, because it intentionally ignores meaningful detail, and it's pedantic because it's making a meaningless distinction. > It's a reminder that there is no magic. This is irrelevant. Nobody is claiming that there's any magic. I'm pointing out the true fact that details about the abstraction layers matter. In this case, the abstraction layer was poorly-designed. Good abstraction layer: length prefix, or JSON encoding. Bad abstraction layer: "the body of the email is mostly plain text, except when there's a line that only contains a single period". There are very, very few problems to which the latter is a good solution. It is a bad engineering decision, and it also obfuscates the fact that there even is an abstraction layer unless you carefully read the spec. ------------- In fact, the underlying problem goes deeper than that - the design of SMTP is intrinsically flawed because it's a text-based ad-hoc protocol that has in-band signaling. There are very few good reasons to use a text-based data interchange format. One of them is to make the format self-documenting, such that people can easily read and write it without consulting the spec. If the spec is complex enough that you get these ridiculous footguns, then it shouldn't be text-based in the first place. Instead, it should be binary - then you have to either read the spec or use someone else's implementation. Failing that, use a standardized structured format like XML or JSON. But there's no excuse for the brain-dead approach that SMTP took. They didn't even use length prefixing,
    - nottorp 2y ago
      Why not protobufs inside protobufs then?
    - kjellsbells 2y ago
      I dont disagree with your criticisms of SMTP, but reading those early RFCs (eg 772) is a reminder of what a wildly different place the Internet was back then, and in that light, I feel it only fair to grant some grace. MTP had one concern which was to get mail over to a host that stood a better chance of delivering it, where the total host pool was maybe a hundred nodes? I speculate that Postel and Sluizer were aware of alternatives and rejected them in favor of things that were easily implemented on highly diverse, low powered hardware. Not everyone had IBM-grade budgets after all. Alternative implementations of mail that did follow the kinds of precepts that you suggest existed at one time. X.400 is the obvious example. If I recall correctly, it did have rigorous protocol spec definitions, message length tags for every entity sent on the wire, bounds and limits on each PDU, the whole hog. It was also crushed by SMTP, and this was in the era when you needed to understand sendmail and its notoriously arcane config to do anything. So sometimes the technically worse solution just wins, and we are stuck with it.
BoppreH 2y ago
That's not a very useful definition of "in-band signaling". For me, the main difference is an out-of-band protocol that says: "The first two bytes represent the string length, in big-endian, followed by that many bytes presenting the string text." and an in-band signalling protocol: "The string is ended by a period and a newline." In the second one, you're indicating the end of the string from within the string. It looks simpler, but that's where accidents happen. Now you have to guarantee that the text never contains that control sequence, and you need an escaping method to represent the control sequence as part of the text.
akvadrako 2y ago
That isn't true at all. In most binary protocols, you put the size of the message in the header. Then any sequence of bytes is allowed to follow. There are no escape sequences.
- pja 2y ago
  That’s still in-band signalling: The metadata is in the same channel as the data.
treflop 2y ago
From my experience, almost no protocols have in-band signaling. No protocol I’ve ever built has in-band signaling because it’s nuts. You always know what the next byte means because either you did a prefixed length, your protocol has stringent escaping rules, or you chose an obvious and consistent terminator like null.
- yencabulator 2y ago
  SMTP has stringent escaping rules. The authors of the code in the article were incompetent.
- pja 2y ago
  If your metadata about the data is in the same channel as the data then you’re doing in-band signalling.
  - treflop 2y ago
    Not in modern times. The terms harken back from the day of circuit switched networks but now that we have heavily transitioned to packets, bands are an artificial construct on top of packets and applying the term isn’t very clear cut. The main property of in-band data in the circuit-switched network days is that you could inject commands into your data stream. If we apply that criteria that to a modern protocol, even if you mix metadata and data in the same “band,” if your data can never be interpreted as commands then “out of band” makes an apt description. See https://en.m.wikipedia.org/wiki/Out-of-band_data https://en.m.wikipedia.org/wiki/Out-of-band_data
    - TeMPOraL 2y ago
      That's only true if you're not breaking the protocol abstraction layer. There is no "out-of-band" once you serialize your messages. If you start injecting random bytes into the data stream on the wire, you can absolutely start introducing commands, or confuse the receiver where the next piece of metadata/control is. In this case, somewhere the protocol abstraction layer got broken, and the message text ended up being treated as already serialized. It's not a problem with the protocol per se, but with bad implementation of its API (or no implementation at all, just printf-ing into the wire format).

Animats 2y ago

SMTP is that way because running the entire contents of a message through some escape character processing was expensive back then, in the era of 0.25 MIPS machines.

throw10920 2y ago
Thank you for weighing in, John Nagle! I understand that constraint, and it seems reasonable - but in that case, why not use a length prefix? That should be even more efficient than having to scan for a line containing a single period and nothing else.
- Animats 2y ago
  Because many paths of that era were not binary-transparent. There was CR-LF to LF conversion, the possibility of ASCII/EBCDIC translation, and other transformations. Hence such horrors as MIME delimiters: -----=_Part_2827761_1947716067.1716352583971 Content-Type: multipart/alternative; boundary="----=_Part_2827760_1372041171.1716352583736" We still have the mess that is the required and standardized behavior of HTML5 parsers faced with bad data.
  - throw10920 2y ago
    That's genuinely horrifying, but it explains why SMTP made a decision that I would otherwise categorize as "insane" - it was forced to by constraints of the time. Thank you for explaining!

eviks 2y ago

Why didn't they use actual control sequences instead of text chars, ASCII had plenty of those?

inejge 2y ago
When your debugger is telnet circa 1980, non-printable characters are a liability. (You pay for it later, when people who've never seen "...end with <CRLF>.<CRLF>" botch the implementation.)