11 ms·

> Viva.com's outgoing verification emails lack a Message-ID header, a requirement that has been part of the Internet Message Format specification (RFC 5322) sin

• zokier 8mo ago
  Also email as a protocol (SMTP) predates RFC5322 by 25 years or so.
• ale42 8mo ago
  The official definition of SHOULD per RFC2119: 3. SHOULD This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course. Not sure how the people at Google interpreted this about the message-id
  • Juliate 8mo ago
    For producers, ignoring a SHOULD is riskier because it shifts the burden to every consumer. For consumers, ignoring a SHOULD mostly affects their own robustness. But here Google seems to understand it as a MUST... maybe the scale of spam is enough to justify it. Users are stuck between two parties that expect the other to behave.
    • zer00eyz 8mo ago
      > maybe the scale of spam is enough to justify it. This is 100 percent the case, and why these things are this way. If you wanted to make email two point oh, I dont think it would look a lot like what we have today.
      • DANmode 8mo ago
        https://jmap.io https://jmap.io
        • tracker1 8mo ago
          jmap is the communication between a mail client and shared directory/mail services on a server. It does not include server to server communications (that I am aware of) for sending mail to other users/servers.
          • DANmode 8mo ago
            Couldn’t resist replying to: > If you wanted to make email two point oh, I dont think it would look a lot like what we have today.
      • tracker1 8mo ago
        I think a mail 2.0 would be notify and pull based.... you notify a recipient's mail server that there's a message from <address> for them, then that server connects to the MX of record for the domain of <address> and retrieves <message-id> message. Would this make mass emails and spam harder, absolutely. Would it be a huge burden for actual communications with people, not so much. From there actual white/black listing processes would work all that much better.
• the_mitsuhiko 8mo ago
  Exactly. Message-ID is not required. An unrelated frustration of mine is that Message-ID really should not be overridden but SES for instance throws away your Message-ID and replaces it with another one :(
  • dathinab 8mo ago
    It is de-facto required and has been for many years. Should in most RFCs also mean "do it as long as you don't have a very good technical reason not to do it". Like it's most times a "weak must". And in that case the only reason it isn't must is for backward compatibility with older mail system not used for sending automated mails. And it is documented if you read any larger mail providers docs about "what to do that my automated mails don't get misclassified as spam". And spam rejection is a whole additional non-standardized layer on top of RFCs anyone working with mail should be aware of. In any decades old non centralized communication system without ever green standards having other "industry standard/de-factor" but not "standardized" requirements is pretty normal btw.
    • mort96 8mo ago
      What's de-facto required and not is completely irrelevant here. The claim was that "Message-ID header [is a] a requirement that has been part of the Internet Message Format specification ... since 2008" (emphasis mine), which is false; it's a recommendation that has been part of the Internet Message Format specification since 2008
• deleted 8mo ago
  [deleted]
• OJFord 8mo ago
  The only messages I receive without one are spam/phishing. I check because they're not recognised by notmuch, so I don't see them otherwise.
• thatha7777 8mo ago
  You're totally right. I've updated the blog to reflect this. Thank you!
• elAhmo 8mo ago
  I would read this as a requirement for email to be 'legit' and not classified as spam. Sure, you can send email with whatever headers you want, use weird combos, IP addresses, reply-to, and it might be still a technically valid email, but not something that should land in people's inboxes. Also, a payment processor not testing their email on the most popular email provider in the world is quite ridiculous.
• zoobab 8mo ago
  Avoid SHALL, SHOULD and all other crap, use Elon MUST.
  • roysting 8mo ago
    SHALL has been interpreted/clarified by US courts as not being a fancy MUST or REQUIRED that many people were taught it to mean, but SHOULD still has it's purposes, e.g., to provide contractual flexibility in development, i.e., a MUST/REQUIRED requirement was more challenging or complicated and took up more time/resources than anticipated, so SHOULDs can be trimmed due to contingencies. Another example may be a lightweight implementation of a spec in a limited and/or narrow environment, which remains technically compliant with full implementations of a spec but interaction with such a limited/narrow environment comes with awareness about such limitations.
• s17n 8mo ago
  The reason that European tech sucks is that people in Europe are open to such arguments. If an engineer in the US started talking about SHOULD vs MUST, some PM would just give them that "what the fuck did I just listen to" face, spend the next few minutes gently trying to convince them that the customer experience matters more than the spec, and if they fail, escalate and get the decision they want. For example, why does Google handle this differently for consumer and enterprise accounts? Well it's Google so the answer could always just be "they are disorganized" but there's a good chance that in both cases, it was the pragmatic choice given the slightly different priorities of these types of customers.
  • youknownothing 8mo ago
    Not my PM (in the US). My PM would try to avoid anything that is not absolutely necessary and therefore ask developers not to develop anything that isn't a MUST. I know that we like making fun of Europe for their alleged lack of innovation but this isn't a Europe thing.
    • lingrush4 8mo ago
      Your PM most definitely would not tell you to skip a feature that is needed for your emails to be delivered to Gmail accounts. What a preposterous thing to lie about.
      • Dylan16807 8mo ago
        The PM wouldn't know about that gmail behavior at that point in the development cycle.
    • deleted 8mo ago
      [deleted]
    • dlopes7 8mo ago
      Implementing a 20 years old needed RFC header is the cutting edge of innovation
  • shaan7 8mo ago
• deepsun 8mo ago
  GMail SHOULD handle your messages, not MUST.
• hermannj314 8mo ago
  You SHOULD follow the wording of the RFC, you MUST follow Google's interpretation of the RFC. That is the difference.
  • redeeman 8mo ago
    evidently they must not
• layer8 8mo ago
  The reason it’s recommended is that it’s useful for detecting when an email you receive is already in your mailbox, so that you don’t accumulate duplicates. Otherwise one would have to compare the complete email, which probably no MUA does. Another reason is that replies can include a reference to the original message, so that it can be properly threaded by MUAs. So these are mostly quality-of-life reasons, it’s not a reason to reject an email.
• Arnt 8mo ago
  SHOULD is a requirement. It means that you have to do it unless you know some specific reason that the requirement doesn't apply in your case. "I don't want to" is not a valid excuse, "I don't see a reason to" isn't either. IIRC this particular rule is a SHOULD because MUAs often send messages without a Message-ID to their submission server, and the submission server adds one if necessary. https://www.rfc-editor.org/rfc/rfc6409.html#section-8.3 https://www.rfc-editor.org/rfc/rfc6409.html#section-8.3 The SHOULD lets those messages be valid. Low-entropy devices that can't generate a good random ID are rare these days, but old devices remain in service, so the workaround is IMO justified.
  • L_226 8mo ago
    As someone who does systems engineering, the only valid requirements include the word "shall".
    • stackskipton 8mo ago
      As someone else who does System Engineering, when dealing with ancient protocols, "shall" is extremely difficult barrier to get over since there is always ancient stuff out there and there might be cases not to do it, esp if it's internal communication. "SHOULD" is basically, if you control both sides of conversation, you can decide if it's required looking at your requirements. If you are talking between systems where you don't control both sides of conversation, you should do all "SHOULD" requirements with fail back in cases where other side won't understand you. If for reason you don't do "SHOULD" requirement, reason should be a blog article that people understand. For example, "SHOULD" requirement would be "all deployable artifacts SHOULD be packaged in OCI container". There are cases where "SHOULD" doesn't work but those are well documented.
      • josephg 8mo ago
        > … when dealing with ancient protocols I’m doing some work with an email company at the moment. The company has been in the email space for decades. Holy moly email is just full of stuff like this. There is an insane amount of institutional knowledge about how email actually works - not what the specs say but what email servers need to actually do to process real emails and deal with real email clients and servers. The rfcs try to keep up, but they’re missing a lot of details and all important context for why recommendations are as they are, and what you actually need to do to actually process real email you see in the wild. (And talk to the long tail of email software). This conversation makes me think about cornering some of the engineers with a microphone. It’d be great to talk through the specs with them, to capture some of that missing commentary.
• ZWoz 8mo ago
  My take, as a postmaster for hosting company, who don't have any sympathy to gmail (that should be visible from my comments history): Message-ID is absolutely MUST in production e-mails. You can send your test stuff without it, but real messages always have it. Not having Message-ID's causes lot of fun things. All somewhat competent software is capable to add Message-ID's, so lack of it is good indication of poorly made custom (usually spamming) solution. Rspamd and spamassassin have missing MID check in their default rules, I am sure that most antispam software is same.
  • stefan_ 8mo ago
    Why? If I'm writing a mail receiver, and I'm told there is some unique ID generated by the sender in a loosely specified way, the first thing I'm doing is ignoring that value forever. One lesson surely most everyone learns in CS is that unique identifiers are maybe unique to the system generating them, but to rely on foreign generated IDs being unique globally is a terrible idea that will break within the minute. So at that point the ID has no value to me except being obliged to carry it around with the message, so maybe the originating system can at some point make sense of it. But then there is obviously no reason to ever reject mail without it, it's an ID valid for the sender and the sender didn't care to include one, great, we save on storage.
    • ZWoz 8mo ago
      MID-s are used by MUA-s for referring earlier messages, tracking answers and so on. So any software expecting dialog (messages coming back) needs to deal with MID-s correctly. Missing MID-s show that said communication is one direction, because broken dialog has not been problem.
    • jasode 8mo ago
      >Why? If I'm writing a mail receiver, and I'm told there is some unique ID generated by the sender in a loosely specified way, the first thing I'm doing is ignoring that value forever. [...] So at that point the ID has no value to me Your framework of analysis is based on someone else's database key ids being irrelevant to you. That's true. But another framework of analysis is tracking statistical correlations of what spam looks like. Lots of spam often don't have message ids. Therefore it's used as a heuristic in scoring it as potential spam. That's why other postmasters even without SpamAssassin independently arrive at the same answer of trying to block messages without a message id. Example: https://serverfault.com/questions/629923/blocking-messages-with-an-empty-message-id-in-postfix https://serverfault.com/questions/629923/blocking-messages-w...
• b00ty4breakfast 8mo ago
  I know you're looking for "pedant points" but the specification generally take a backseat to implementation. If Message-ID is expected out here where the rubber meets the road, then you are the squeaky wheel in this scenario for not including it.
  • bossyTeacher 8mo ago
    > the specification generally take a backseat to implementation. And we should be raising hell for it. Should never happen. Using your popularity to violate protocol should be not be tolerated
    • b00ty4breakfast 8mo ago
      Per the RFC. "should" means "you better have a good reason for not implementing {thing}".
• thatha7777 8mo ago
  And the definition of "SHOULD" (from RFC 2119) is "This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course." Having said that, I regret my original characterization of the Message-ID header as a "requirement" and have updated the blogpost to be fair to all sides. Thank you for bringing this up.
• philipallstar 8mo ago
  As indicated in the RFC, it uses another RFC[0] to define those words. Here's the relevant excerpt from that one: 3. SHOULD This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course. [0] https://www.rfc-editor.org/rfc/rfc2119 https://www.rfc-editor.org/rfc/rfc2119
  • dathinab 8mo ago
    yeah, but that RFC isn't the only relevant document Mail RFCs do not cover at all spam detection and malicious mail rejection, but it's a thing every large mail provider has and you really have to care about when producing automated mails all looking similar. And large mail providers like google tend to document what "base line" of additional requirements they have for accepting (automated mail). Having a Message-Id is in there, and in pretty much any larger mail providers documentation about that topic. Tbh. I have worked with mail before a bunch of years ago and the need for Message-Id was back then really no hidden gotcha but pretty well known. and the design space mail provides is larger then any client could reasonable support (like it's really a huge mess covering docent of standards which allow all kind of nonsense and hypothetical use cases practical unsupported), so you anyway have to look at "what everyone does" and only then make sure it's also RFC compatible, instead of starting with the RFC. That was a painful lessen to learn. In addition there are some de-facto standards not pinned down in any RFC, like e.g.: - Message-Id being required for any automated mails by many mail providers (through how bad the consequences are if you don't have it diverges largely). - You can't punycode encode the local part of an email address (it would be a different email), and there is no standard way (as far as I remember) to convert non us-ascii local parts to us-ascii. This is based on the fact that iff your mail server allows you to have non us-ascii local parts it should also support "internationalized mail" (SMTPUTF8 and co.). But it's a semi industry standard to give the user with an unicode local part also the mail with the punycode encoding of the local part so it often "just works" and dev are frequently surprised when it fails to work... - You can have quoted text in local part, like whitespaces. But most of the industry decided to not give users such mail addresses so you can see it as soft deprecated and using it is asking for trouble. - Attachements. The MIME encoding allows a lot of different ways to put mails together and doesn't force a specific semantic interpretation by mail clients. As such you if you naively use it you might run into surprises how/if your attachments or embedding(s) are displayed. Through today embedded resources often are either not done or uses data URLs. Again which ways work well and which don't is somewhat an industry standard and not in any RFC. - A lot of different ways to encode Unicode to us-ascii. If you produce any mails you probably should by default use the latest revision (where encoding is often just not needed as things are utf8), but might need a fallback with it often being fully unclear if . But if you are a client you probably have to support older versions. And in some parts of the world/business segments usage of very very old mail servers is a thing which is a major pain if you run into it. so quoting that something isn't strictly required by mail RFCs is kinda pointless as even many things explicitly allowed won't work well in practice As a rule of thump: If you can afford it test you system will all widely used mail providers as if you where an external customers. And redo the tests yearly. oh and as a bonus, if you mails looks too similar to known phishing mails it will also just disappear. That seems irrelevant, but e.g. if you use Keycloak with the default mail templates for password reset and co. there is a high chance of your mails ending up in spam or not even being delivered as scammers have used Keycloak for their means, too. And that isn't just a case for Keycloak but any "decently widely used open source software producing mails and having default templates". So you pretty much always need to change the default templates (you will do so anyway for branding, but skipping it in the earliest stages of a startup where branding might still be in flux isn't that uncommon either).
• deleted 8mo ago
  [deleted]
• PunchyHamster 8mo ago
  > That says SHOULD, not MUST, so how is it a requirement? Battle with spam has been for long part just trying to algorithmically fingerprint the scam bots and reject the message if it looks like it wasn't sent by "real" mail server/client. So a lot of things that are optional like SPF/DKIM are basically "implement this else your mail have good chance of being put into spam automatically".
• torlavd 8mo ago
  Standard RFC naming, optional field.
• jiggawatts 8mo ago
  The HTTP User-Agent header is also optional, but if you omit it, something like half of all endpoints will respond with a 500 error code.
• tlogan 8mo ago
  SHOULD = You are strongly recommended to do this, but it’s not absolutely required. - In most cases, you are expected to follow it. - You can choose not to follow it, but you must have a very good reason. For example, RFC 7231 say that there should be DATE header but some embedded devices have no real-time clock so it ok not to implement.
• dathinab 8mo ago
  1. SHOULD means, do it if you can/you have to have a really good reason if you don't do it. The only reason it's SHOULD and not MUST is backward compatibility. Mostly in context of "personal send mails", i.e. not automated mails. (For automated mail sending the expectations of you running somewhat up to date software is much higher). 2. You can't really implement mail stuff just based on RFCs: - There docent overlapping RFCs which can sometimes influence each other and many of them obsolete older versions why others still relevant RFCs reference this older versions. This makes it hard to even know what actually is required/recommendation. - Then you have a lot of "irrelevant" parts, which where standardized but are hardly supported/if at all. You probably should somewhat support them as recipient but should never produce them as sender today (mostly stuff related to pro-"everything is utf8" days). Like in general the ideas of "how mail should probably work" in old RFCs and "how it is done IRL today" are in some aspects _very_ far away. - Lastly RFCs are not sufficient by themself. They don't cover large parts of the system for "spam detection/suspicious mail rejection". So it's a must have to go to the support pages of all large mail providers and read through what they expect of mails. And "automated mails need a message id" is a pretty common requirement. In addition you have to e.g. make sure the domain you use isn't black listed (e.g. due to behavior of a previous user), and that your servers IP addresses aren't black listed (they never should be black listed long term, but happens anyway, and e.g. MS has based on very questionable excuses "conveniently" black listed smaller local data center competition while also being one of the most widely used providers for commercial mail in that area).