7 ms·

I used Caddy and was just wondering if my stuff would be broke, so the GP comment was useful to me, and snarky as your reply was.

yjftsjthsd-h 3y ago
Sure, there are a handful of useful points there (off the top of my head: it's possible to work around this failure by using multiple CAs, multiple free ACME CAs exist, caddy implements this solution). I'm just 1. slightly frustrated that caddy's author never seems to miss a chance for self-promotion (at least he's started alluding to the fact that it's his project), and 2. actually curious whether any other ACME clients are implementing that fallback.
- mholt 3y ago
  > I'm just 1. slightly frustrated that caddy's author never seems to miss a chance for self-promotion (at least he's started alluding to the fact that it's his project), Sorry that I happen to be the author. It's really not about that though -- it just matters that an ACME-native HTTPS server exists. We need more integrated fully-native ACME clients. > actually curious whether any other ACME clients are implementing that fallback. There are at least one or two others. I don't recall which ones at the moment but I think Certify the Web may be one. Edit: mod_md is another apparently!
  - tialaramex 3y ago
    Right, it's astounding to me that outfits like Microsoft didn't just immediately ship decent ACME implementations. People seem to have settled for third party bolt-on solutions. It's like you wake up in an alternate world where yeah, no cars come with seat belts, but of course everybody buys seatbelts for the car, there's usually a store next to the car dealer which sells them. Um. What? Most of the "popular" software in this space is garbage. I spent the entire day today (aside from meetings and helping other people debug problems) wrestling with the fact Apache seems to be designed so heavily with a C programmer mindset that even the idea of reporting problems has never occurred to them. Just blunder on, it'll be fine, don't think about it. You can sprinkle complete nonsense into Apache configuration files and, until you trip an actual syntactical error and blow up their parser, Apache just presses on anyway with the nonsense values you provided, and if that doesn't work, no reason to report it just do whatever was the default and hope that's OK. As far as I can tell, in the wild the result is a lot of Apache configuration is complete nonsense, but hey no errors are reported, so, copy, paste, move on.
- toast0 3y ago
  Apache mod_md has fallback too, https://github.com/icing/mod_md#acme-failover https://github.com/icing/mod_md#acme-failover I'm just a user, not the author, and I didn't try the fallback. I'm more worried about stuff breaking if I switch issuers than certs expiring without me noticing. I've got some embedded junk that hits my website and has weak cert validation, so better to stick with something that works.
- schoolornot 3y ago
  Promoting went from HTTP headers to HN comments.
  - francislavoie 3y ago
    How would you feel if most of the times you post online, someone shows up and mentions a mistake you made years ago? Not cool.
    - yjftsjthsd-h 3y ago
      Does mholt consider it a mistake? I'm aware that it was reverted (https://github.com/caddyserver/caddy/pull/1866 https://github.com/caddyserver/caddy/pull/1866) and that mholt found the whole thing difficult (which is my attempt to neutrally summarize https://caddy.community/t/the-realities-of-being-a-foss-maintainer/2728?u=matt https://caddy.community/t/the-realities-of-being-a-foss-main... accurately), but that is a somewhat different statement. If so, then yes, it's unkind and unhelpful to keep bringing it up, but if no then it's useful to keep previous behavior in mind when evaluating the product.
      - francislavoie 3y ago
        Yes, it was a mistake. (Source: I'm a Caddy maintainer, and it comes up in our discussions from time to time.) The reason it was done was because Caddy needed some source of revenue since Matt made it his full time job, and he assumed the sponsors thought they would appreciate the extra promotion (and as you can see at the bottom of that github link, he tried to alert them but received no feedback). Of course, thinking that no feedback was an implicit "sure" was a lapse in judgment, but we're all human. Remember, this was six years ago. That's an eternity in this industry. Caddy is a very different project than it was then, and Matt has a different and more stable revenue stream than he did then. We can promise we'll never attempt the same thing again. But seriously, this comes up in like one in ten HN threads where Matt comments, it's exhausting to keep telling people "okay can you please forget what you remember from 6 years ago and look at the project for what it is now?"
- xen2xen1 3y ago
  For self promotion that was pretty light. You ever seen the Sourcehut guy on here?
agwa 3y ago
If your certificate was issued after the start of the incident but before Let's Encrypt suspended issuance, then the certificate is currently not working in Chrome or Safari. This incident wasn't just about downtime, it was also about issuing non-functional/non-compliant certificates.
- mholt 3y ago
  Is this because the certs were revoked? (Revocation is broken ;P) Caddy staples Valid OCSP responses to all certificates that have an OCSP responder, so if browsers aren't accepting that, then arguably the clients are broken, because that response is valid until a few days from now. But before the 100% valid and trusted OCSP staple expires, Caddy will get a new staple that presumably says Revoked, and replace them right away before browsers would ever see a Revoked status. (Revocation is broken ;P)
  - agwa 3y ago
    No, it's because the SCTs in the certificate have invalid signatures.
    - mholt 3y ago
      Ah, that makes sense! I wonder if we should be doing some basic sanity checks on newly obtained certificates in Caddy, and treat this as a failure, and try the next configured CA instead. (Obviously SCT signatures will require some external resource so we would have to weigh that a bit more, maybe make it configurable...) Issue opened here to discuss, though it does sound troublesome/tedious: https://github.com/caddyserver/certmagic/issues/240 https://github.com/caddyserver/certmagic/issues/240
      - agwa 3y ago
        Yeah, the question is how far you want to go. To be safe against every possible CA screwup, you basically have to re-implement every browser's entire certificate validation engine and run certificates through each one. That would obviously be very hard, and could do more harm than good if it falls out-of-sync with browsers.

yjftsjthsd-h 3y ago

Sure, there are a handful of useful points there (off the top of my head: it's possible to work around this failure by using multiple CAs, multiple free ACME CAs exist, caddy implements this solution). I'm just 1. slightly frustrated that caddy's author never seems to miss a chance for self-promotion (at least he's started alluding to the fact that it's his project), and 2. actually curious whether any other ACME clients are implementing that fallback.

mholt 3y ago
> I'm just 1. slightly frustrated that caddy's author never seems to miss a chance for self-promotion (at least he's started alluding to the fact that it's his project), Sorry that I happen to be the author. It's really not about that though -- it just matters that an ACME-native HTTPS server exists. We need more integrated fully-native ACME clients. > actually curious whether any other ACME clients are implementing that fallback. There are at least one or two others. I don't recall which ones at the moment but I think Certify the Web may be one. Edit: mod_md is another apparently!
- tialaramex 3y ago
  Right, it's astounding to me that outfits like Microsoft didn't just immediately ship decent ACME implementations. People seem to have settled for third party bolt-on solutions. It's like you wake up in an alternate world where yeah, no cars come with seat belts, but of course everybody buys seatbelts for the car, there's usually a store next to the car dealer which sells them. Um. What? Most of the "popular" software in this space is garbage. I spent the entire day today (aside from meetings and helping other people debug problems) wrestling with the fact Apache seems to be designed so heavily with a C programmer mindset that even the idea of reporting problems has never occurred to them. Just blunder on, it'll be fine, don't think about it. You can sprinkle complete nonsense into Apache configuration files and, until you trip an actual syntactical error and blow up their parser, Apache just presses on anyway with the nonsense values you provided, and if that doesn't work, no reason to report it just do whatever was the default and hope that's OK. As far as I can tell, in the wild the result is a lot of Apache configuration is complete nonsense, but hey no errors are reported, so, copy, paste, move on.
toast0 3y ago
Apache mod_md has fallback too, https://github.com/icing/mod_md#acme-failover https://github.com/icing/mod_md#acme-failover I'm just a user, not the author, and I didn't try the fallback. I'm more worried about stuff breaking if I switch issuers than certs expiring without me noticing. I've got some embedded junk that hits my website and has weak cert validation, so better to stick with something that works.
schoolornot 3y ago
Promoting went from HTTP headers to HN comments.
- francislavoie 3y ago
  How would you feel if most of the times you post online, someone shows up and mentions a mistake you made years ago? Not cool.
  - yjftsjthsd-h 3y ago
    Does mholt consider it a mistake? I'm aware that it was reverted (https://github.com/caddyserver/caddy/pull/1866 https://github.com/caddyserver/caddy/pull/1866) and that mholt found the whole thing difficult (which is my attempt to neutrally summarize https://caddy.community/t/the-realities-of-being-a-foss-maintainer/2728?u=matt https://caddy.community/t/the-realities-of-being-a-foss-main... accurately), but that is a somewhat different statement. If so, then yes, it's unkind and unhelpful to keep bringing it up, but if no then it's useful to keep previous behavior in mind when evaluating the product.
    - francislavoie 3y ago
      Yes, it was a mistake. (Source: I'm a Caddy maintainer, and it comes up in our discussions from time to time.) The reason it was done was because Caddy needed some source of revenue since Matt made it his full time job, and he assumed the sponsors thought they would appreciate the extra promotion (and as you can see at the bottom of that github link, he tried to alert them but received no feedback). Of course, thinking that no feedback was an implicit "sure" was a lapse in judgment, but we're all human. Remember, this was six years ago. That's an eternity in this industry. Caddy is a very different project than it was then, and Matt has a different and more stable revenue stream than he did then. We can promise we'll never attempt the same thing again. But seriously, this comes up in like one in ten HN threads where Matt comments, it's exhausting to keep telling people "okay can you please forget what you remember from 6 years ago and look at the project for what it is now?"
xen2xen1 3y ago
For self promotion that was pretty light. You ever seen the Sourcehut guy on here?

agwa 3y ago

If your certificate was issued after the start of the incident but before Let's Encrypt suspended issuance, then the certificate is currently not working in Chrome or Safari. This incident wasn't just about downtime, it was also about issuing non-functional/non-compliant certificates.

mholt 3y ago
Is this because the certs were revoked? (Revocation is broken ;P) Caddy staples Valid OCSP responses to all certificates that have an OCSP responder, so if browsers aren't accepting that, then arguably the clients are broken, because that response is valid until a few days from now. But before the 100% valid and trusted OCSP staple expires, Caddy will get a new staple that presumably says Revoked, and replace them right away before browsers would ever see a Revoked status. (Revocation is broken ;P)
- agwa 3y ago
  No, it's because the SCTs in the certificate have invalid signatures.
  - mholt 3y ago
    Ah, that makes sense! I wonder if we should be doing some basic sanity checks on newly obtained certificates in Caddy, and treat this as a failure, and try the next configured CA instead. (Obviously SCT signatures will require some external resource so we would have to weigh that a bit more, maybe make it configurable...) Issue opened here to discuss, though it does sound troublesome/tedious: https://github.com/caddyserver/certmagic/issues/240 https://github.com/caddyserver/certmagic/issues/240
    - agwa 3y ago
      Yeah, the question is how far you want to go. To be safe against every possible CA screwup, you basically have to re-implement every browser's entire certificate validation engine and run certificates through each one. That would obviously be very hard, and could do more harm than good if it falls out-of-sync with browsers.