6 ms·

Regular reminder that the best ACME clients will fall back to other CAs if one is down. For example caddy does this. (Disclosure yada yada)

by mholt 3y ago

yjftsjthsd-h 3y ago
> the best ACME clients will fall back Are there others that do it, or are you just saying that yours is the best?
- xen2xen1 3y ago
  I used Caddy and was just wondering if my stuff would be broke, so the GP comment was useful to me, and snarky as your reply was.
  - yjftsjthsd-h 3y ago
    Sure, there are a handful of useful points there (off the top of my head: it's possible to work around this failure by using multiple CAs, multiple free ACME CAs exist, caddy implements this solution). I'm just 1. slightly frustrated that caddy's author never seems to miss a chance for self-promotion (at least he's started alluding to the fact that it's his project), and 2. actually curious whether any other ACME clients are implementing that fallback.
    - mholt 3y ago
      > I'm just 1. slightly frustrated that caddy's author never seems to miss a chance for self-promotion (at least he's started alluding to the fact that it's his project), Sorry that I happen to be the author. It's really not about that though -- it just matters that an ACME-native HTTPS server exists. We need more integrated fully-native ACME clients. > actually curious whether any other ACME clients are implementing that fallback. There are at least one or two others. I don't recall which ones at the moment but I think Certify the Web may be one. Edit: mod_md is another apparently!
      - tialaramex 3y ago
        Right, it's astounding to me that outfits like Microsoft didn't just immediately ship decent ACME implementations. People seem to have settled for third party bolt-on solutions. It's like you wake up in an alternate world where yeah, no cars come with seat belts, but of course everybody buys seatbelts for the car, there's usually a store next to the car dealer which sells them. Um. What? Most of the "popular" software in this space is garbage. I spent the entire day today (aside from meetings and helping other people debug problems) wrestling with the fact Apache seems to be designed so heavily with a C programmer mindset that even the idea of reporting problems has never occurred to them. Just blunder on, it'll be fine, don't think about it. You can sprinkle complete nonsense into Apache configuration files and, until you trip an actual syntactical error and blow up their parser, Apache just presses on anyway with the nonsense values you provided, and if that doesn't work, no reason to report it just do whatever was the default and hope that's OK. As far as I can tell, in the wild the result is a lot of Apache configuration is complete nonsense, but hey no errors are reported, so, copy, paste, move on.
    - toast0 3y ago
      Apache mod_md has fallback too, https://github.com/icing/mod_md#acme-failover https://github.com/icing/mod_md#acme-failover I'm just a user, not the author, and I didn't try the fallback. I'm more worried about stuff breaking if I switch issuers than certs expiring without me noticing. I've got some embedded junk that hits my website and has weak cert validation, so better to stick with something that works.
    - schoolornot 3y ago
      Promoting went from HTTP headers to HN comments.
      - francislavoie 3y ago
        How would you feel if most of the times you post online, someone shows up and mentions a mistake you made years ago? Not cool.
        yjftsjthsd-h 3y ago
        Does mholt consider it a mistake? I'm aware that it was reverted (https://github.com/caddyserver/caddy/pull/1866 https://github.com/caddyserver/caddy/pull/1866) and that mholt found the whole thing difficult (which is my attempt to neutrally summarize https://caddy.community/t/the-realities-of-being-a-foss-maintainer/2728?u=matt https://caddy.community/t/the-realities-of-being-a-foss-main... accurately), but that is a somewhat different statement. If so, then yes, it's unkind and unhelpful to keep bringing it up, but if no then it's useful to keep previous behavior in mind when evaluating the product.
        francislavoie 3y ago
        Yes, it was a mistake. (Source: I'm a Caddy maintainer, and it comes up in our discussions from time to time.) The reason it was done was because Caddy needed some source of revenue since Matt made it his full time job, and he assumed the sponsors thought they would appreciate the extra promotion (and as you can see at the bottom of that github link, he tried to alert them but received no feedback). Of course, thinking that no feedback was an implicit "sure" was a lapse in judgment, but we're all human. Remember, this was six years ago. That's an eternity in this industry. Caddy is a very different project than it was then, and Matt has a different and more stable revenue stream than he did then. We can promise we'll never attempt the same thing again. But seriously, this comes up in like one in ten HN threads where Matt comments, it's exhausting to keep telling people "okay can you please forget what you remember from 6 years ago and look at the project for what it is now?"
    - xen2xen1 3y ago
      For self promotion that was pretty light. You ever seen the Sourcehut guy on here?
  - agwa 3y ago
    If your certificate was issued after the start of the incident but before Let's Encrypt suspended issuance, then the certificate is currently not working in Chrome or Safari. This incident wasn't just about downtime, it was also about issuing non-functional/non-compliant certificates.
    - mholt 3y ago
      Is this because the certs were revoked? (Revocation is broken ;P) Caddy staples Valid OCSP responses to all certificates that have an OCSP responder, so if browsers aren't accepting that, then arguably the clients are broken, because that response is valid until a few days from now. But before the 100% valid and trusted OCSP staple expires, Caddy will get a new staple that presumably says Revoked, and replace them right away before browsers would ever see a Revoked status. (Revocation is broken ;P)
      - agwa 3y ago
        No, it's because the SCTs in the certificate have invalid signatures.
        mholt 3y ago
        Ah, that makes sense! I wonder if we should be doing some basic sanity checks on newly obtained certificates in Caddy, and treat this as a failure, and try the next configured CA instead. (Obviously SCT signatures will require some external resource so we would have to weigh that a bit more, maybe make it configurable...) Issue opened here to discuss, though it does sound troublesome/tedious: https://github.com/caddyserver/certmagic/issues/240 https://github.com/caddyserver/certmagic/issues/240
        agwa 3y ago
        Yeah, the question is how far you want to go. To be safe against every possible CA screwup, you basically have to re-implement every browser's entire certificate validation engine and run certificates through each one. That would obviously be very hard, and could do more harm than good if it falls out-of-sync with browsers.
- mholt 3y ago
  There are others that do it. While we're on the topic, I will say that Caddy has independently and repeatedly been cited as the gold standard of ACME clients, "the best client experience," and "we hope to see other servers follow Caddy's lead." [0] [1] [and others I don't have links to currently]. [0]: https://www.youtube.com/watch?v=OE5UhQGg_Fo https://www.youtube.com/watch?v=OE5UhQGg_Fo [1]: https://jhalderm.com/pub/papers/letsencrypt-ccs19.pdf https://jhalderm.com/pub/papers/letsencrypt-ccs19.pdf
Arnavion 3y ago
That wouldn't work for caddy if you also follow the best practice to have a CAA record pointing to the issuer and account URL, unless caddy is also managing DNS records in addition to being an HTTP server. (I don't know if it is, but I would think it's a layering violation for an HTTP server to also be a DNS server.)
- mholt 3y ago
  This is true, if you manually configure a CAA limited to just one CA, then you lose that benefit of redundancy. I recommend trusting multiple CAs (but not too many): https://matt.life/writing/the-acme-protocol-in-practice-and-reality https://matt.life/writing/the-acme-protocol-in-practice-and-... > (I don't know if it is, but I would think it's a layering violation for an HTTP server to also be a DNS server.) Caddy 2 is, at its core, a server of servers. The HTTP server is just an "app module" for Caddy. There are other servers; I don't know of a DNS server app yet. (CoreDNS is a fork of Caddy v1, though.)
neurostimulant 3y ago
What other CAs you recommend aside from letsencrypt? I'm a bit wary of trying some random CAs that offers free certificates aside from letsencrypt.
- mholt 3y ago
  It's good to be wary. You can trust the ACME CAs listed on this site: https://www.acmeisuptime.com/ https://www.acmeisuptime.com/ (Although, I think that list could use some updating. I'll ping the author.) Personally I would use Let's Encrypt, ZeroSSL (Sectigo) and Google Trust Services. There are, of course, others. But which ones you choose depend on your requirements and such. (Some offer business support, for example.) SSL.com and Sectigo also offer ACME but I am not sure how performant their CA software is.
remram 3y ago
The missing disclosure is "I'm the author of the Caddy web server". I'm not sure why you would do it halfway.
- mholt 3y ago
  Because it's distracting, honestly. Anyone could say "Caddy does this" and they would objectively be correct. The point is about ACME client features and giving an objectively true example.
  - remram 3y ago
    Then don't put the disclosure at all, it's just weird to do the "distracting" part but not give the information.
    - BonsaiAU 3y ago
      Especially if their next comment is a reply to someone else and states "Mine is the gold standard, here's proof". Just make the disclosure clear but short if the comment is a throwaway, or don't mention your own service, otherwise it's a race to the bottom.

Regular reminder that the best ACME clients will fall back to other CAs if one is down. For example caddy does this. (Disclosure yada yada)

by mholt 3y ago

Regular reminder that the best ACME clients will fall back to other CAs if one is down. For example caddy does this. (Disclosure yada yada)

yjftsjthsd-h 3y ago

> the best ACME clients will fall back Are there others that do it, or are you just saying that yours is the best?

xen2xen1 3y ago
I used Caddy and was just wondering if my stuff would be broke, so the GP comment was useful to me, and snarky as your reply was.
- yjftsjthsd-h 3y ago
  Sure, there are a handful of useful points there (off the top of my head: it's possible to work around this failure by using multiple CAs, multiple free ACME CAs exist, caddy implements this solution). I'm just 1. slightly frustrated that caddy's author never seems to miss a chance for self-promotion (at least he's started alluding to the fact that it's his project), and 2. actually curious whether any other ACME clients are implementing that fallback.
  - mholt 3y ago
    > I'm just 1. slightly frustrated that caddy's author never seems to miss a chance for self-promotion (at least he's started alluding to the fact that it's his project), Sorry that I happen to be the author. It's really not about that though -- it just matters that an ACME-native HTTPS server exists. We need more integrated fully-native ACME clients. > actually curious whether any other ACME clients are implementing that fallback. There are at least one or two others. I don't recall which ones at the moment but I think Certify the Web may be one. Edit: mod_md is another apparently!
    - tialaramex 3y ago
      Right, it's astounding to me that outfits like Microsoft didn't just immediately ship decent ACME implementations. People seem to have settled for third party bolt-on solutions. It's like you wake up in an alternate world where yeah, no cars come with seat belts, but of course everybody buys seatbelts for the car, there's usually a store next to the car dealer which sells them. Um. What? Most of the "popular" software in this space is garbage. I spent the entire day today (aside from meetings and helping other people debug problems) wrestling with the fact Apache seems to be designed so heavily with a C programmer mindset that even the idea of reporting problems has never occurred to them. Just blunder on, it'll be fine, don't think about it. You can sprinkle complete nonsense into Apache configuration files and, until you trip an actual syntactical error and blow up their parser, Apache just presses on anyway with the nonsense values you provided, and if that doesn't work, no reason to report it just do whatever was the default and hope that's OK. As far as I can tell, in the wild the result is a lot of Apache configuration is complete nonsense, but hey no errors are reported, so, copy, paste, move on.
  - toast0 3y ago
    Apache mod_md has fallback too, https://github.com/icing/mod_md#acme-failover https://github.com/icing/mod_md#acme-failover I'm just a user, not the author, and I didn't try the fallback. I'm more worried about stuff breaking if I switch issuers than certs expiring without me noticing. I've got some embedded junk that hits my website and has weak cert validation, so better to stick with something that works.
  - schoolornot 3y ago
    Promoting went from HTTP headers to HN comments.
    - francislavoie 3y ago
      How would you feel if most of the times you post online, someone shows up and mentions a mistake you made years ago? Not cool.
      - yjftsjthsd-h 3y ago
        Does mholt consider it a mistake? I'm aware that it was reverted (https://github.com/caddyserver/caddy/pull/1866 https://github.com/caddyserver/caddy/pull/1866) and that mholt found the whole thing difficult (which is my attempt to neutrally summarize https://caddy.community/t/the-realities-of-being-a-foss-maintainer/2728?u=matt https://caddy.community/t/the-realities-of-being-a-foss-main... accurately), but that is a somewhat different statement. If so, then yes, it's unkind and unhelpful to keep bringing it up, but if no then it's useful to keep previous behavior in mind when evaluating the product.
        francislavoie 3y ago
        Yes, it was a mistake. (Source: I'm a Caddy maintainer, and it comes up in our discussions from time to time.) The reason it was done was because Caddy needed some source of revenue since Matt made it his full time job, and he assumed the sponsors thought they would appreciate the extra promotion (and as you can see at the bottom of that github link, he tried to alert them but received no feedback). Of course, thinking that no feedback was an implicit "sure" was a lapse in judgment, but we're all human. Remember, this was six years ago. That's an eternity in this industry. Caddy is a very different project than it was then, and Matt has a different and more stable revenue stream than he did then. We can promise we'll never attempt the same thing again. But seriously, this comes up in like one in ten HN threads where Matt comments, it's exhausting to keep telling people "okay can you please forget what you remember from 6 years ago and look at the project for what it is now?"
  - xen2xen1 3y ago
    For self promotion that was pretty light. You ever seen the Sourcehut guy on here?
- agwa 3y ago
  If your certificate was issued after the start of the incident but before Let's Encrypt suspended issuance, then the certificate is currently not working in Chrome or Safari. This incident wasn't just about downtime, it was also about issuing non-functional/non-compliant certificates.
  - mholt 3y ago
    Is this because the certs were revoked? (Revocation is broken ;P) Caddy staples Valid OCSP responses to all certificates that have an OCSP responder, so if browsers aren't accepting that, then arguably the clients are broken, because that response is valid until a few days from now. But before the 100% valid and trusted OCSP staple expires, Caddy will get a new staple that presumably says Revoked, and replace them right away before browsers would ever see a Revoked status. (Revocation is broken ;P)
    - agwa 3y ago
      No, it's because the SCTs in the certificate have invalid signatures.
      - mholt 3y ago
        Ah, that makes sense! I wonder if we should be doing some basic sanity checks on newly obtained certificates in Caddy, and treat this as a failure, and try the next configured CA instead. (Obviously SCT signatures will require some external resource so we would have to weigh that a bit more, maybe make it configurable...) Issue opened here to discuss, though it does sound troublesome/tedious: https://github.com/caddyserver/certmagic/issues/240 https://github.com/caddyserver/certmagic/issues/240
        agwa 3y ago
        Yeah, the question is how far you want to go. To be safe against every possible CA screwup, you basically have to re-implement every browser's entire certificate validation engine and run certificates through each one. That would obviously be very hard, and could do more harm than good if it falls out-of-sync with browsers.
mholt 3y ago
There are others that do it. While we're on the topic, I will say that Caddy has independently and repeatedly been cited as the gold standard of ACME clients, "the best client experience," and "we hope to see other servers follow Caddy's lead." [0] [1] [and others I don't have links to currently]. [0]: https://www.youtube.com/watch?v=OE5UhQGg_Fo https://www.youtube.com/watch?v=OE5UhQGg_Fo [1]: https://jhalderm.com/pub/papers/letsencrypt-ccs19.pdf https://jhalderm.com/pub/papers/letsencrypt-ccs19.pdf

Arnavion 3y ago

That wouldn't work for caddy if you also follow the best practice to have a CAA record pointing to the issuer and account URL, unless caddy is also managing DNS records in addition to being an HTTP server. (I don't know if it is, but I would think it's a layering violation for an HTTP server to also be a DNS server.)

mholt 3y ago
This is true, if you manually configure a CAA limited to just one CA, then you lose that benefit of redundancy. I recommend trusting multiple CAs (but not too many): https://matt.life/writing/the-acme-protocol-in-practice-and-reality https://matt.life/writing/the-acme-protocol-in-practice-and-... > (I don't know if it is, but I would think it's a layering violation for an HTTP server to also be a DNS server.) Caddy 2 is, at its core, a server of servers. The HTTP server is just an "app module" for Caddy. There are other servers; I don't know of a DNS server app yet. (CoreDNS is a fork of Caddy v1, though.)

neurostimulant 3y ago

What other CAs you recommend aside from letsencrypt? I'm a bit wary of trying some random CAs that offers free certificates aside from letsencrypt.

mholt 3y ago
It's good to be wary. You can trust the ACME CAs listed on this site: https://www.acmeisuptime.com/ https://www.acmeisuptime.com/ (Although, I think that list could use some updating. I'll ping the author.) Personally I would use Let's Encrypt, ZeroSSL (Sectigo) and Google Trust Services. There are, of course, others. But which ones you choose depend on your requirements and such. (Some offer business support, for example.) SSL.com and Sectigo also offer ACME but I am not sure how performant their CA software is.

remram 3y ago

The missing disclosure is "I'm the author of the Caddy web server". I'm not sure why you would do it halfway.

mholt 3y ago
Because it's distracting, honestly. Anyone could say "Caddy does this" and they would objectively be correct. The point is about ACME client features and giving an objectively true example.
- remram 3y ago
  Then don't put the disclosure at all, it's just weird to do the "distracting" part but not give the information.
  - BonsaiAU 3y ago
    Especially if their next comment is a reply to someone else and states "Mine is the gold standard, here's proof". Just make the disclosure clear but short if the comment is a throwaway, or don't mention your own service, otherwise it's a race to the bottom.