10 ms·
Upcoming changes to Let's Encrypt and how they affect XMPP server operators
- PunchyHamster 8mo agoShame LE didn't give people option to generate client and client+server auth certs
- forty 8mo agoYes, but then the lack of pragmatism shown by the XMPP community is a bit disconcerting
- superkuh 8mo agoIt is not pragmatic to design your protocol for web use cases when it's not the web.
- bawolff 8mo agoUnless im missing something, this is a poor design full stop. How are they validating SAN on these client certificates?
- agwa 8mo agoXMPP identifiers have domain names, so the XMPP server can check that the DNS SAN matches the domain name of the identifiers in incoming XMPP messages. I've seen non-XMPP systems where you configure the DNS name to require in the client certificate. It's possible to do this securely, but I agree entirely with your other comment that using a public PKI with client certs is a recipe for disaster because it's so easy and common to screw up.
- account42 8mo agoOn the contrary, setting up a separate PKI for XMPP would be what is not pragmatic or even feasible at all. The pragmatic choice is to make use of the options available even if some people find them icky.
- SahAssar 8mo agoWhat is the lack of pragmatism you are talking about?
- forty 8mo agoThe refusal to accept server only certificate as client certificate for server
- kokx 8mo agoThe article literally talks about how one of the server implementations does exactly that: > Does this affect Prosody? > Not directly. Let’s Encrypt is not the first CA to issue server-only certificates. Many years ago, we incorporated changes into Prosody which allow server-only certificates to be used for server-to-server connections, regardless of which server started the connection. [...]
- MattJ100 8mo agoThere might be some confusion here, as there is no refusal at all. As stated in the blog post, we (Prosody) have been accepting (only) serverAuth certificates for a long time. However this is technically in violation of the relevant RFCs, and not the default behaviour of TLS libraries, so it's far from natural for software to be implementing this. There was only one implementation discovered so far which was not accepting certificates unless they included the clientAuth purpose, and that was already updated 6+ months ago. This blog post is intended to alert our users, and the broader XMPP community, about the issue that many were unaware of, and particularly to nudge server operators to upgrade their software if necessary, to avoid any federation issues on the network.
- forty 8mo agoSorry, I probably misunderstood, I thought there were resistance to update the corresponding specs. I understand that non XMPP specs might refuse to be updated, but at least this behavior could be standardized for XMPP specifically.
- jammcq 8mo agoI like how the article describes how certificates work for both client and server. I know a little bit about it but what I read helps to reinforce what I already know and it taught me something new. I appreciate it when someone takes the time to explain things like this.
- MattJ100 8mo agoThanks! I didn't intentionally write this for a broader audience (I didn't expect to see it while casually opening HN!). Our user base is quite diverse, so I try to find the balance between being too technical and over-explanatory. Glad it was helpful!
- RobotToaster 8mo agoWhy did LE make this change? It feels like a rather deliberate attack on the decentralised web.
- duskwuff 8mo agoNot precisely an answer, but there's some related discussion here: https://cabforum.org/2025/06/11/minutes-of-the-f2f-65-meeting-in-toronto-canada-scwg-june-11-2025/#discussion-about-removal-of-clientauth-keypurposeid https://cabforum.org/2025/06/11/minutes-of-the-f2f-65-meetin... The real takeaway is that there's never been a lot of real thought put into supporting client authentication - e.g. there's no root CA program for client certificates. To use a term from that discussion, it's usually just "piggybacked" on server authentication.
- mhurron 8mo agoNo, it feels like the standard 'group/engineer/PM' didn't think anyone did anything different from their own implementation. Lets Encrypt is just used for like, webservers right, why do this other stuff webservers never use. Which does appear to be the thinking, though they blame Google, which also seems to have taken the 'webservers in general don't do this, it's not important' - https://letsencrypt.org/2025/05/14/ending-tls-client-authentication https://letsencrypt.org/2025/05/14/ending-tls-client-authent...
- pseudalopex 8mo agoGoogle forced separate client and server PKIs.[1] [1] https://letsencrypt.org/2025/05/14/ending-tls-client-authentication https://letsencrypt.org/2025/05/14/ending-tls-client-authent...
- ameliaquining 8mo agoGoogle has recently imposed a rule that CA roots trusted by Chrome must be used solely for the core server-authentication use case, and can't also be used for other stuff. They laid out the rationale here: https://googlechrome.github.io/chromerootprogram/moving-forward-together/#phase-out-multi-purpose-roots-from-the-chrome-root-store https://googlechrome.github.io/chromerootprogram/moving-forw... It's a little vague, but my understanding reading between the lines is that sometimes, when attempts were made to push through security-enhancing changes to the Web PKI, CAs would push back on the grounds that there'd be collateral damage to non-Web-PKI use cases with different cost-benefit profiles on security vs. availability, and the browser vendors want that to stop happening. Let's Encrypt could of course continue offering client certificates if they wanted to, but they'd need to set up a separate root for those certificates to chain up to, and they don't think there's enough demand for that to be worth it.
- everfrustrated 8mo agoFrom https://letsencrypt.org/2025/05/14/ending-tls-client-authentication https://letsencrypt.org/2025/05/14/ending-tls-client-authent... "This change is prompted by changes to Google Chrome’s root program requirements, which impose a June 2026 deadline to split TLS Client and Server Authentication into separate PKIs. Many uses of client authentication are better served by a private certificate authority, and so Let’s Encrypt is discontinuing support for TLS Client Authentication ahead of this deadline." TL;DR blame Google
- bawolff 8mo agoGoogle didn't force lets encrypt to totally get out of the client cert business, they just decided it wasn't worth the effort anymore.
- josephcsible 8mo ago> they just decided it wasn't worth the effort anymore That seems disingenuous. Doesn't being in the client cert business now require a lot of extra effort that it didn't before, due entirely to Google's new rule?
- everfrustrated 8mo agoFeel free to start your own non-profit to issue client certs signed by a public authority. As LE says, most users of client certs are doing mtls and so self-signed is fine.
- _tdeb 8mo agoIs there any reason why things gravitate towards being web-centric, especially Google-centric? Seeing that Google's browser policies triggered the LE change and the fact that most CAs are really just focusing on what websites need rather than non-web services isn't helpful considering that browsers now are terribly inefficient (I mean come on, 1GB of RAM for 3 tabs of Firefox whilst still buffering?!) yet XMPP is significantly more lightweight and yet more featureful compared to say Discord.
- xg15 8mo ago> Is there any reason why things gravitate towards being web-centric, especially Google-centric? Yes, the reason is called "Chrome" and "90% market share"...
- account42 8mo agoAnd the reason for that is called "regulatory capture".
- nickf 8mo agoGoogle dominate the space because they have an active, robust trust-store program that they manage well. Apple the same. Mozilla and Microsoft too (though to a lesser extent). If any ecosystem - such as XMPP - wishes to, they could start their own root-program, but many simply copy what Chrome or Mozilla do and then are surprised when things change.
- bawolff 8mo agoI feel like using web pki for client authentication doesn't really make sense in the first place. How do you verify the common name/subject alt name actually matches when using a client cert. Using web pki for client certs seems like a recipe for disaster. Where servers would just verify they are signed but since anyone can sign then anyone can spoof. And this isn't just hypothetical. I remember xmlsec (a library for validating xml signature, primarily saml) used to use web pki for signature validation in addition to specified cert, which resulted in lot SAML bypasses where you could pass validation by signing the SAML response with any certificate from lets encrypt including the attackers.
- nickf 8mo agoYou are correct, and the answer is - no-one using publicly-trusted TLS certs for client authentication is actually doing any authentication. At best, they're verifying the other party has an internet connection and perhaps the ability to read. It was only ever used because other options are harder to implement.
- xg15 8mo agoIt seems reasonable for server-to-server auth though? Suppose my server xmpp.foo.com already trusts the other server xmpp.bar.com. Now I get some random incoming connection. How would I verify that this connection indeed originates from xmpp.bar.com? LE-assigned client certs sound like a good solution to that problem.
- benjojo12 8mo agoFor those wondering if ejabberd Debian systems will be impacted, it seems like for now there no fix, the issue is being tracked here: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1127369 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1127369
- Avamander 8mo agoCode can just ignore the EKU. Especially if the ecosystem consists of things that are already using certificates in odd ways, as it shouldn't be making outgoing connections without it in the first place.
- nickf 8mo agoClient authentication with publicly-trusted (i.e. chaining to roots in one of the major 4 or 5 trust-store programs) is bad. It doesn't actually authenticate anything at all, and never has. No-one that uses it is authenticating anything more than the other party has an internet connection and the ability, perhaps, to read. No part of the Subject DN or SAN is checked. It's just that it's 'easy' to rely on an existing trust-store rather than implement something secure using private PKI. Some providers who 'require' public TLS certs for mTLS even specify specific products and CAs (OV, EV from specific CAs) not realising that both the CAs and the roots are going to rotate more frequently in future.
- ajross 8mo agoA client cert can be stored, so it provides at least a little bit of identification certainty. It's very hard to steal or impersonate a specific client cert, so the site has a high likelihood of knowing you're the same person you were when you connected before (even though the initial connection may very well not have ID'd the correct person!). That has value. But it also doesn't involve any particular trust in the CA either. Lets Encrypt has nothing to offer here so there's no reason for them to try to make promises.
- nickf 8mo agoEh, it's pretty easy to impersonate if the values in the certificate aren't checked, and you could get one from any of a list of public CAs. If you're relying on a certificate for authentication - issue it yourself.
- ajross 8mo agoPoint being that if you get a valid TLS connection from a client cert, and then you get another valid connection from the same cert tomorrow, you can be very certain that the entity connecting is either the same software environment that connected earlier, or an attacker that has compromised it. You can be cryptographically certain that it is not an attacker that hasn't effected a full compromise of your client. And there's value there, if you're a server. It's why XMPP wants federated servers to authenticate themselves with certificates in the first place.
- deleted 8mo ago[deleted]
- agwa 8mo agoIs there a reason why dialback isn't the answer? I would think it's more secure than clientAuth certs because if an attacker gets a misissued cert they'd have to actually execute a MitM attack to use it. In contrast, with a misissued clientAuth cert they can just connect to the server and present it. Another fun fact: the Mozilla root store, which I'd guess the vast majority of XMPP servers are using as their trust store, has ZERO rules governing clientAuth issuance[1]. CAs are allowed to issue clientAuth-only certificates under a technically-constrained non-TLS sub CA to anyone they want without any validation (as long as the check clears ;-). It has never been secure to accept the clientAuth EKU when using the Mozilla root store. [1] https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/policy/#11-scope https://www.mozilla.org/en-US/about/governance/policies/secu...
- MattJ100 8mo ago> Is there a reason why dialback isn't the answer? There are some advantages to using TLS for authentication as well as encryption, which is already a standard across the internet. For example, unlike an XMPP server, CAs typically perform checks from multiple vantage points ( https://letsencrypt.org/2020/02/19/multi-perspective-validation.html https://letsencrypt.org/2020/02/19/multi-perspective-validat... ). There is also a lot of tooling around TLS, ACME, CT logs, and such, which we stand to gain from. In comparison, dialback is a 20-year-old homegrown auth mechanism, which is more vulnerable to MITM. Nevertheless, there are some experiments to combine dialback with TLS. For example, checking that you get the same cert (or at least public key) when connecting back. But this is not really standardized, and can pose problems for multi-server deployments. > It has never been secure to accept the clientAuth EKU when using the Mozilla root store. Good job we haven't been doing this for a very long time by now :)
- jauntywundrkind 8mo agoI really fail to understand or sympathize with Let's Encrypt limiting their certs so. What is gained by slamming the door on other applications than servers being able to get certs? In this case I do think it makes sense for servers to accept certs even as marked by servers, since it's for a s2s use case. But this just feels like such an unnecessary clamping down. To have made certs finally plentiful, & available for use... Then to take that away? Bother!
- sam_lowry_ 8mo agoAFAIU Let's Encrypt (ISRG) gets most of its money from Google, both directly and indirectly via Mozilla. They do what Google says.
- denus 8mo agoI wonder if issues like this couldn't be a use case for DANE.
- MattJ100 8mo agoYes, definitely. Prosody supports DANE, but DNSSEC deployment continues to be an issue when talking about the public XMPP network at large. Ironically the .im TLD our own site is on still doesn't support it at all.
- account42 8mo agoUsing ccTLDs from small countries for domain hacks has always been a risky choice.
- tkel 8mo agoProsody is also the base of Snikket[1], a popular recent XMPP server. Snikket is basically just a Prosody config.[2] [1] https://snikket.org/service/quickstart/ https://snikket.org/service/quickstart/ [2] https://github.com/snikket-im/snikket-server/blob/master/ansible/files/prosody.cfg.lua https://github.com/snikket-im/snikket-server/blob/master/ans...
- syntheticnature 8mo agoA minor quibble: I'd describe Snikket as more a batteries-included distribution of the Prosody config plus validated client apps for mobile devices.
- MattJ100 8mo agoAs the founder of both projects, explaining the difference between the two projects is roughly 20% of my working day (okay, not quite 20% but sometimes it feels that way). Your description is great :)
- thayne 8mo agoI can think of a of other ways that client certificates could work, but they have problems too: 1. Use DANE to verify the client certificate. But that requires DNSSEC, which isn't widely used. Would probably require new implemntations of the handshake to check the client cert, and would add latency since the server has to do a DNS call to verify the clients cer. 2. When the server receives a request it makes an https request to a well known enpdpoint on the domain in the client-cert's subject that contains a CA, it then checks that the client cert is signed by that CA. And the client generates the client cert with that CA (or even uses the same self-signed cert for both). This way the authenticity of the client CA is verified using the web PKI cert. But the implementation is kind of complicated, and has an even worse latency problem than 1. 3. The server has an endpoint where a client can request a client certificate from that server, probably with a fairly short expiration, for a domain, with a csr, or equivalent. The server then responds by making an https POST operation to a well known enpdpoint on the requested domain containing a certificate signed by the servers own CA. But for that to work, the registration request needs to be unauthenticated, and could possibly be vulnerable to DoS attacks. It also requires state on the client side, to connect the secret key with the final cert (unless the server generated a new secret key for the client, which probably isn't ideal). And the client should probably cache the cert until it expires. And AFAIK, all of these would require changes to how XMPP and other federated protocols work.
- MattJ100 8mo agoOf these, (1) and (2) are already implemented in XMPP. (1) just isn't that widely deployed due to low DNSSEC adoption and setup complexity, but there is a push to get server operators to use it if they can. (2) is defined in RFC 7711: https://www.rfc-editor.org/rfc/rfc7711 https://www.rfc-editor.org/rfc/rfc7711 however it has more latency and complexity compared to just using a valid certificate directly in the XMPP connection's TLS handshake. Its main use is for XMPP hosting providers that don't have access to a domain's HTTPS.
- thayne 8mo ago2 isn't quite the same as my idea, it uses a list of fingerprints for valid certs instead of a CA itself, but it is roughly equivalent.
- nilslindemann 8mo ago> The current CA ecosystem is *heavily* driven by web browser vendors (i.e. Google, Apple, Microsoft and Mozilla), and they are increasingly hostile towards non-browser applications using certificates from CAs that they say only provide certificates for consumption by web browsers. Let's translate and simplify: > The current CA ecosystem is Google. They want that only Google-applications get certificates from CAs.
- mmsc 8mo agoNo. HTTPS certificates are being abused for non-https purposes. CAs want to sell certificates for everything under the sun, and want to force those in the ecosystem to support their business, even though https certificates are not designed to be used for other things (mail servers for example). If CAs don't want hostility from browser companies for using https certificate for non-http/browser applications, they should build their own thing.
- sam_lowry_ 8mo agoNo. These are just certificates that happen to be used predominantly in HTTPS context and Google tries to tie them exclusively to the HTTPS context.
- dijit 8mo agoYou’re like, so wrong. Are we really at an age where people don’t remember that SSL was intended for many protocols, including MAIL?! Do you think email works on web technology because you use a web-client to access your mailbox? Jesus christ, formal education needs to come quickly to our industry.
- bux93 8mo agoPKI certificates weren't even intended for SSL, it predates even that. X.509 was published in November 25, 1988 ; version 3 added support for "the web" as it was known at the time. One obvious use was for X.400 e-mail systems in the 1980s. Novell Netware adopted x.509. It was originally intended to use with X.511 "Directory Access Protocol", which LDAP was based on. You can still find X.500 heritage in Microsft Exchange and Active Directory, although it's getting less over time and e.g. EntraID only has some affordances for backward compatibility.
- deleted 8mo ago[deleted]
- greatgib 8mo agoBasically they will break all MTLS usage with the certificates.
- rnhmjoj 8mo agoWhat is the point of restricting a certificate to "server" or "client" use, anyway?
- direwolf20 8mo agoTrust chains. Some implementations would accept an LE certificate for foo.com as a valid login for foo.com or something like that, because they treated all trusted certs the same, whether issued by the service being authenticated to, or some other CA. It might be possible to relay communications between two servers and have one of them act as a client without knowing. Handshake verification prevents that in TLS, but there could be similar attacks.
- csense 8mo agoThe problem here is that when alice@chat.example.com and bob@xmpp.example2.com talk to each other, chat.example.com asks "Are you xmpp.example2.com?" and xmpp.example2.com asks "Are you chat.example.com?" If you strictly require the side that opens the TCP connection to only use client certs and require the side that gets the TCP connection to only use server certs, then workflows where both sides validate each other become impossible with a single connection. You could have each server open a TCP connection to the other, but then you have a single conversation spread across multiple connections. It gets messy fast, especially if you try to scale beyond a single server -- the side that initiates the first outgoing connection has to receive the second incoming connection, so you have to somehow get your load balancer to match the second connection with the first and route it to the same box. Then at the protocol level, you'd have to essentially have each connection's server send a random number challenge to the client saying "I can't authenticate clients because they don't have certs. So please echo this back on the other connection where you're the server and I can authenticate you." The complexity and subtlety of this coordination dance seems like you're just asking security issues. If I was implementing XMPP I would be very tempted to say, "Don't be strict about client vs. server certs, let a client use a server cert to demonstrate ownership of a domain -- even if it's forbidden by RFC and even if we have to patch our TLS library to do it."