10 ms·
Comcast Shot Themselves in the Foot with MTA-STS
- usbqk 5y agoOh boy. I sure hope you guys aren’t using your ISP’s email address.
- syntheticnature 5y agoI'm not, but far too many folks do as a matter of course.
- hffftz 5y ago
- tyingq 5y agoInteresting. I'm curious if popular MTA software supports some kind of configuration to reload caches after some amount of MTA-STS failures.
- pas 5y agoIsn't by design it must cache for the amount of time specified in the TXT file? So it has to be durable. Of course the MTA probably should provide ways for admins to override this. Maybe as simple as deleting a file, modifying a sql(ite) DB.
- agwa 5y ago> Isn't by design it must cache for the amount of time specified in the TXT file? So it has to be durable. Kind of. The max_age in the text file is the max time a cached policy should be used if a new policy can't be retrieved (e.g. it's being blocked by an attacker). If a new policy can be retrieved over HTTPS, it's totally OK to use it even if the max_age of the cached policy isn't up yet. Thus, tyingq's suggestion is reasonable. In fact, the RFC already tells mail servers to look for a new policy if just one MTA-STS failure occurs (https://datatracker.ietf.org/doc/html/rfc8461#section-5.1 https://datatracker.ietf.org/doc/html/rfc8461#section-5.1 step 3). The problem is that it checks for a new policy using the ID in the TXT record, which as explained in the blog post is unreliable.
- pas 5y agoThanks for the detailed explanation. It seems silly to introduce that extra level of indirection when there's already a well-known URL to try to fetch a new version. Is there any rationale for this? Also if DNS is explicitly a not trusted component this seems to strictly lead to a less secure system :o (But I guess the designers did not want to "completely" override MX records just with yet another out of band magic.)
- agwa 5y ago> Is there any rationale for this? Do you mean for the TXT record? The rationale for that was performance. I don't know if this rationale was backed up by hard data or just a theoretical concern that HTTPS requests are more expensive than DNS requests. > Also if DNS is explicitly a not trusted component this seems to strictly lead to a less secure system The TXT record doesn't contain any security-sensitive information so it doesn't actually reduce security. The worst an attacker can do is block the retrieval of a new policy, but they could also accomplish this by removing the mta-sts A/AAAA record. Either way, the cached policy will be used until its max_age expires.
- pas 5y ago> Do you mean for the TXT record? I mean that during the max_age time the MTA knows that the target MTA has/had a policy, so it could directly query that again instead of doing the DNS lookup. But probably the spec writers did not want to risk MTAs getting hammered with HTTPS requests on every failure. > Either way, the cached policy will be used until its max_age expires. Ah, thanks. I got confused by the DNS spoofing, but of course since the policy is protected by PKI TLS even if the attacker can redirect the MTA to evil.com/policy.txt the cert won't match, and the email recipient domain is already in the email to be delivered. (And if that can be spoofed, then the mail contents are also very likely accessible.)
- ddtaylor 5y agoThat comcast.net email address is only good for storing all the DMCA complaints that pile up and get automatically deleted.
- _huayra_ 5y agoEveryone can easily move to email at their own domain. A com domain is about $14/year and you can get free email hosting from your domain on zoho. Google can cut off your account on a whim with no recourse and God forbid you want to escape your isp but keep your email address with them!
- agwa 5y agoRight, but now you're on the hook for setting up MTA-STS yourself, which as this post shows is no easy task.
- rovr138 5y agoIf you're using a provider, you're not on the hook for it. Unless I'm mistaken in how this is implemented (not just server side)
- agwa 5y agoYour provider is on the hook for properly configuring TLS on their mail server, but you still need to publish an MTA-STS policy saying who your provider is. Unless your provider provides MTA-STS endpoints that you can CNAME your _mta-sts and mta-sts subdomains to. I haven't seen anyone do this yet.
- Byteflux 5y agoFor sure. It's not trivial but I also wouldn't say it's difficult either, maybe more of a moderate difficulty? The policy file format is straight forward, as are the necessary TXT and CNAME records. In my case, I use Cloudflare for my domain so I set up a Cloudflare Pages project to generate and host the policy file at the mta-sts subdomain which is nice because I don't have to worry about hosting or SSL/TLS. As long as your provider supports MTA-STS, you're only on the hook for setting up the records and hosting the policy file.
- agwa 5y agoIf you never make changes, that will work once it's set up. But if you ever decide to change your mail provider, you'll have to remember to correctly update your MTA-STS policy. If your change to Cloudflare Pages doesn't propagate through the Cloudflare CDN right away, you could end up in the situation described in the blog post, with the TXT record updated before the text file, and other mail servers unable to send you email.
- zokier 5y agoDoes anyone know why SNI style approach is not used for email? The client could indicate in the TLS handshake (or in STARTTLS command) that it wants to send email to comcast.com and the server would use comcast.com certificate to prove that it is authentic destination for those emails. That way you'd avoid this whole mess of needing to fetch another file using another protocol based on another DNS record etc. SNI seems to work well for web, so why not for email too?
- meepmorp 5y agoYou're not always talking directly to the destination mailserver when sending a message via SMTP.
- zokier 5y agoI don't see how that makes a difference? Sure, the cert issuance/distribution can be bit of hassle but it doesn't seem insurmountable
- rblatz 5y agoI’ve heard this, and I believe it is technically true. But in practice never happens anymore. Are there any situations today where that happens?
- deleted 5y ago[deleted]
- agwa 5y agoThis is a good question. It's a bit more complicated than just using SNI. First is certificate management. There is currently no automated way for mail providers to obtain TLS certificates for domains whose MX records point to them - i.e. there is no SMTP/MX equivalent to the Baseline Requirement's "Agreed-Upon Change to Website" domain validation method, or ACME's HTTP-01 challenge type. And in general domain operators don't want their mail providers to have certificates which could be used to impersonate their websites. The certificate problem could be solved with SRVName certificates, which permit certificates to be scoped to a particular service on a host. Combined with an automated SMTP/MX-based validation method, this would permit mail providers to easily and securely get certificates for the domains whose mail they host. But there are thorny security-critical compatibility problems which are blocking the deployment of SRVName certificates. I wrote about the problem near the end of this blog post: https://www.agwa.name/blog/post/avoiding_dnssec https://www.agwa.name/blog/post/avoiding_dnssec Second is that you need a mechanism for servers to remember that they should use TLS with a particular recipient domain, similar to HSTS. This could probably be accomplished with a simple SMTP extension that's included in the EHLO response.
- RedShift1 5y agoI think the real problem is (emphasis mine): > the MX record is vulnerable to manipulation unless DNSSEC is used, but people don't like DNSSEC which is why MTA-STS was invented This speaks volumes about the way these protocols are developed and subsequently implemented in software. People are not going to like complex solutions and that should be kept in mind when those protocols are being developed and implemented, otherwise its deployment is hampered just like we're seeing now with DNSSEC. In fact, now we've got MTA-STS because "people don't like DNSSEC", involving a whole HTTPS stack on both sides with so much more failure points just to send an email.
- AndyMcConachie 5y agocomcast.net is DNSSEC signed and Comcast was one of the first major ISPs to implement DNSSEC validation. Comcast's MX records are DNSSEC signed. > dig +dnssec comcast.net MX This has nothing to do with DNSSEC.
- Dylan16807 5y agoIt has nothing to do with Comcast not liking DNSSEC. But if the ecosystem as a whole was happy about DNSSEC and DANE, it's somewhat doubtful we'd have both specs and the involvement of https.
- tptacek 5y agoWe would not. There'd be no reason to have MTA-STS if there was universal DNSSEC deployment. Of course, outages like this would be vastly more common, because DNSSEC includes a large superset of the failure modes of MTA-STS.
- KennyBlanken 5y agoIt's not that people don't "like" DNSSEC. The author themselves states in a different post on their blog that DNSSEC is avoided like the plague because it is a royal pain in the ass to set up and outages at major corporations (coughSlackcough) that clearly have talented, knowledgeable, experienced ops people show that the standard was not designed with practical implementation in mind. "Security that is difficult to implement and can cause catastrophic-level outages not used by majority of internet; film at 11." Instead of lecturing everyone on how they should be using DNSSEC, what we should be doing is working on "DNSSEC 2.0", from lessons learned in the 2-3 decades people have been trying to use DNSSEC.
- LinuxBender 5y agoWouldn't the simplest work around be to just remove mx2h1.comcast.net and mx1a1.comcast.net from the MX records for at least 2592000 seconds and instead add the IP addresses of those mail servers vips to other existing DNS records that were properly in the STS file? e.g. dig +short mx2h1.comcast.net 96.102.157.179 dig +short mx1a1.comcast.net 96.103.145.165 So just add 96.102.157.179 to mx2c1.comcast.net and add 96.103.145.165 to mx1h1.comcast.net or something along that line? Their MX TTL is only 300 seconds.
- agwa 5y agoComcast just needs to rotate the ID in the _mta-sts.comcast.net TXT record. That will cause sending mail servers to download the new policy.
- LinuxBender 5y agoWell that's way too easy. I supposed I improperly assumed that the results would be cached longer than the TTL of the TXT record based on the expiration in the well-known text file. In that case they should have only seen a tiny window of time that emails remained in queues. I will have to read up on the RFC for MTA-STS a bit more. In that case I would suggest they did not really shoot themselves in the feet. Rather they may have stubbed their toe.
- ocdtrekkie 5y agoGood Lord, someone took the dumpster fire that was HSTS and made it for mail servers? Heaven help us all and have mercy on our souls. I wanted to then follow up with an insightful or interesting comment but I am at a loss.
- Macha 5y agoHSTS itself is fine? It just means "please only contact me over TLS for X amount of time". The only complaint I heard about was adtech companies using it for fingerprinting. Are you sure you're not thinking of HPKP?
- Karunamon 5y agoProblem is "X amount of time" is usually "months-to-years" and there's no override.
- Macha 5y agoBut as for as stipulations go, "Must use TLS" seems pretty tame.
- ocdtrekkie 5y agoUntil an application chooses to enable HSTS, and their only documented way to update the certificate is the web interface... and there's HSTS. I lost a weekend to HSTS not long ago. Essentially the problem with HSTS is it's a malicious choice to remove user agency because the spec intentionally prohibits web browsers from allowing a bypass option, despite the browser being an agent of the user, not the server. It also tells you to talk to your administrator, which is unhelpful when you are the administrator but the browser refuses to allow the administrator to continue. Every time I've encountered HSTS in practice, it's been because of a perfectly normal certificate error I'm aware of the reason for (usually expiration) and have no real concerns about bypassing, except my own web browser won't let me. The problem with HSTS (and MTA-STS) is that these are specs designed for internet centralization, where Google or Microsoft controls everything and manages everything[1], and a certificate never lapses. It's not designed for the real world, and neither of these specs would've made it except for that big tech has complete capture of web standards. [1] Didn't know who the authors were when writing this comment, but the authors of the MTA-STS RFC are in fact, Google, Google, Microsoft, Comcast, and Yahoo.
- wolverine876 5y ago> MTA-STS is a relatively new standard that allows domain owners such as Comcast to opt in to authenticated encryption for their mail servers. (By default, SMTP traffic between mail servers uses opportunistic encryption, which can be defeated by active attackers to intercept email.) MTA-STS requires the domain owner to duplicate their MX record (the DNS record that lists a domain's mail servers) in a text file served over HTTPS. Sending mail servers, like Alex's, refuse to contact mail servers that aren't listed in the MTA-STS text file. Since HTTPS uses authenticated encryption, the text file can't be altered by active attackers. (In contrast, the MX record is vulnerable to manipulation unless DNSSEC is used, but people don't like DNSSEC which is why MTA-STS was invented.) Is this just a way of validating the MX record? That is, the public MX record in DNS can't be trusted, but because a cert verifies the identity of the sending mail server, the one they send can be trusted? If someone is successfully impersonating the sender, what stops them from sending a false MX record. If they aren't impersonators, why not just trust them regardless - why do they need the MX record? Obviously, I'm missing something.
- tedunangst 5y agoThis allows you to verify the mx record of the domain you're sending to, not the sender.
- wolverine876 5y agoThanks. Without trusting the MX record, how do you know from what address ask for the MX over HTTPS? Maybe I should just look it up myself!
- tedunangst 5y agoIn theory, you cache a good copy from sometime before the attack.
- aaronmdjones 5y agoThis is for the receiver, not the sender.