6 ms·

I don't understand the tone of aggression against ACME and their plethora of clients. I know it isn't a skill issue because of who the author is. So I can only

I don't understand the tone of aggression against ACME and their plethora of clients. I know it isn't a skill issue because of who the author is. So I can only imagine it is some sort of personal opinion that they dislike ACME as a concept or the tooling around ACME in general. We've been using LE for a while (since 2019 I think) for handful of sites, and the best nonsense client _for us_ was https://github.com/do-know/Crypt-LE/releases https://github.com/do-know/Crypt-LE/releases. Then this year we've done another piece of work this time against the Sectigo ACME server and le64 wasn't quite good enough. So we ended up trying:- - https://github.com/certbot/certbot https://github.com/certbot/certbot on GitHub Actions, it was fine but didn't quite like the locked down environment - https://github.com/go-acme/lego https://github.com/go-acme/lego huge binary, cli was interestingly designed and the maintainer was quite rude when raising an issue - https://github.com/rmbolger/Posh-ACME https://github.com/rmbolger/Posh-ACME our favourite, but we ended up going with certbot on GHA once we fixed the weird issues around permissions Edit* Re-read it. The tone isn't aimed at the ACME or the clients. It's the spec itself. ACME idea good, ACME implementation bad.

immibis 1y ago
Some people don't want to be forced to run a bunch of stuff they don't understand on the server, and I agree with them. Sadly, security is a cat and mouse game, which means it's always evolving and you're forced to keep up - and it's inherent by the nature of the field, so we can't really blame anyone (unlike, say, being forced to integrate with the latest Google services to be allowed on the Play Store). At least you get to write your own ACME client if you want to. You don't have to use certbot, and there's no TPM-like behaviour locking you out of your own stuff.
- spockz 1y ago
  Given that keys probably need to be shared between multiple gateway/ingresses, how common is it to just use some HSM or another mechanism of exchanging the keys with all the instances? The acme client doesn’t have to run on the servers itself.
  - tialaramex 1y ago
    > The acme client doesn’t have to run on the servers itself. This is really important to understand if you care about either: Actually engineering security at some scale or knowing what's actually going on in order to model it properly in your head. If you just want to make a web site so you can put up a blog about your new kitten, any of the tools is fine, you don't care, click click click, done. For somebody like Rachel or many HN readers, knowing enough of the technology to understand that the ACME client needn't run on your web servers is crucial. It also means you know that when some particular client you're evaluating needs to run on the web server that it's a limitation of that client not of the protocol - birds can't all fly, but flying is totally one of the options for birds, we should try an eagle not an emu if we want flying.
  - immibis 1y ago
    You could if your domain was that valuable. Most aren't.
- hannob 1y ago
  > Some people don't want to be forced to run a bunch of stuff they > don't understand on the server, and I agree with them. Honest question: * Do you understand OS syscalls in detail? * Do you understand how your BIOS initializes your hardware? * Do you understand how modern filesystems work? * Do you understand the finer details of HTTP or TCP? Because... I don't. But I know enough about them that I'm quite convinced each of them is a lot more difficult to understand than ACME. And all of them and a lot more stuff are required if you want to run a web server.
  - frogsRnice 1y ago
    Sure - but people are still free to decide where they draw the line. Each extra bit of software is an additional attack surface after all
  - sussmannbaka 1y ago
    This point is so tired. I don’t understand how a thought forms in my neurons, eventually matures into a decision and how the wires in my head translate this into electrical pulses to my finger muscles to type this post so I guess I can’t have opinions about complexity.
  - kjs3 1y ago
    I hear some variation of this line of 'reasoning' about once a week, and it's always followed by some variation of "...and that's why we shouldn't have to do all this security stuff you want us to do".
  - fc417fc802 1y ago
    An OS is (at least generally) a prerequisite. If minimalism is your goal then you'd want to eliminate tangentially related things that aren't part of the underlying requirements. If you're a fan of left-pad I won't judge but don't expect me to partake without bitter complaints.
  - snowwrestler 1y ago
    I get where you’re going with this, but in this particular case it might not be relevant because there’s a decent chance that Rachel By The Bay does actually understand all those things.
- g-b-r 1y ago
  > Some people don't want to be forced to run a bunch of stuff they don't understand on the server It's not just about not understanding, it's that more complex stuff is inherently more prone to security vulnerabilities, however well you think you reviewed its code.
  - Avamander 1y ago
    > It's that more complex stuff is inherently more prone to security vulnerabilities That's overly simplifying it and ignores the part where the simple stuff is not secure to begin with. In the current context you could take a HTTP client with a formally verified TLS stack, would you really say it's inherently more vulnerable than a barebones HTTP client talking to a server over an unencrypted connection? I'd say there's a lot more exposed in that barebones client.
    - g-b-r 1y ago
      The alternative of the article was ACME vs other ways of getting TLS certificates, not https vs http. Of course plain http would be, generally, much more dangerous than a however complex encrypted connection
      - g-b-r 1y ago
        Why the heck was this downvoted
- tptacek 1y ago
  Non-ACME certs are basically over. The writing has been on the wall for a long time. I understand people being squeamish about it; we fear change. But I think it's a hopeful thing: the Web PKI is evolving. This is what that looks like: you can't evolve and retain everyone's prior workflows, and that has been a pathology across basically all Internet security standards work for decades.
  - ipdashc 1y ago
    ACME is cool (compared to what came before it), but I'm kind of sad that EV certs never seemed to pan out at all. I feel like they're a neat concept, and had the potential to mitigate a lot of scams or phishing websites in an ideal world. (That said, discriminating between "big companies" and "everyone else who can't afford it" would definitely have some obvious downsides.) Does anyone know why they never took off?
    - johannes1234321 1y ago
      > Does anyone know why they never took off? Browser vendors at some point claimed it confused users and removed the highlight (I think the same browser vendors who try to remove the "confusing" URL bar ...) Aside from that EV certificates are slow to issue and phishers got similar enough EV certs making the whole thing moot.
    - tialaramex 1y ago
      EV can't actually work. It was always about branding for the for-profit CAs so that they have a premium product which helps the line go up. Let me give you a brief history - you did ask. In about 2005, the browser vendors and the Certificate Authorities began meeting to see if they could reach some agreement as neither had what they wanted and both might benefit from changes. This is the creation of the CA/Browser Forum aka CA/B Forum which still exists today. From the dawn of SSL the CAs had been on a race to the bottom on quality and price. Initially maybe somebody from a huge global audit firm that owns a CA turn up on a plane and talks to your VP of New Technology about this exciting new "World Wide Web" product, maybe somebody signs a $1M deal over ten years, and they issue a certificate for "Huge Corporation, Inc" with all the HQ address details, etc. and oh yeah, "www.hugecorp.example" should be on there because of that whole web thing, whatever that's about. Nerd stuff! By 2005 your web designer clicks a web page owned by some bozo in a country you've never heard of, types in the company credit card details, company gets charged $15 because it's "on sale" for a 3 year cert for www.mycompany.example and mail.mycompany.com is thrown in for free, so that's nice. Is it secure? Maybe? I dunno, I think it checked my email address? Whatever. The "real world address" field in this certificate now says "Not verified / Not verified / None" which is weird, but maybe that's normal? The CAs can see that if this keeps up in another decade they'll be charging $1 each for 10 year certificates, they need a better product and the browser vendors can make that happen. On the other hand the browser vendors have noticed that whereas auditors arriving by aeroplane was a bit much, "Our software checked their email address matched in the From line" is kinda crap as an "assurance" of "identity". So, the CA/B Baseline Requirements aka BRs are one result. Every CA agreed they'd do at least what the "baseline" required and in practice that's basically all they do because it'd cost extra to do more so why bother. The BRs started out pretty modest - but it's amazing what you find people were doing when you begin writing down the basics of what obviously they shouldn't do. For example, how about "No issuing certificates for names which don't exist" ? Sounds easy enough right? When "something.example" first comes into existence there shouldn't already be certificates for "something.example" because it didn't exist... right? Oops, lots of CAs had been issuing those certificates, reasoning that it's probably fine and hey, free money. Gradually the BRs got stricter, improving the quality of this baseline product in both terms of the technology and the business processes, this has been an enormous boon, because it's an agreement for the industry this ratchets things for everybody, so there's no race to the bottom on quality because your competitors aren't allowed to do worse than the baseline. On price, the same can't be said, zero cost certificates are what Let's Encrypt is most famous for after all. The other side of the deal is what the CAs wanted, they wanted UI for their new premium product. That's EV. Unlike many of the baseline requirements, this is very product focused (although to avoid being an illegal cartel it is forbidden for CA/B Forum to discuss products, pricing etc.) and so it doesn't make much technical sense. The EV documents basically say you get all of the Baseline, plus we're going to check the name of your business - here's how we'll check - and then the web browser is going to display that name. It's implied that these extra checks cost more money (they do, and so this product is much more expensive). So improvements to that baseline do help still, but they also help everybody who didn't buy the premium EV product. Now, why doesn't this work in practice? The DNS name or IP address in an "ordinary" certificate can be compared to reality automatically by the web browser. This site says it is news.ycombinator.com, it has a certificate for news.ycombinator.com, that's the same OK. Your browser performs this check, automatically and seamlessly, for every single HTTP transaction. Here on HN that's per page load, but on many sites you're doing transactions as you click UI or scroll the page, each is checked. With EV the checks must be done by a human, is this site really "Bob's Burgers" ? Actually wait, is it really "Bob's Burgers of Ohio, US" ? Worse, probably although you know them as Bob's Burgers, legally, as you'd see on their papers of incorporation they are "Smith Restaurant Holdings Inc." and they're registered in Delaware because of course they are. So now you're staring at the formal company name of a busines and trying to guess whether that's legitimate or a scam. But remember you can't just do this check once, scammers might find a way to redirect some traffic and so you need to check every individual transaction like your web browser does. Of course it's a tireless machine and you are not. So in practice this isn't effective.
- throw0101b 1y ago
  > Some people don't want to be forced to run a bunch of stuff they don't understand on the server, and I agree with them. There are a number of shell-based ACME clients whose prerequisites are: OpenSSL and cURL. You're probably already relying on OpenSSL and cURL for a bunch of things already. If you can read shell code you can step through the logic and understand what they're doing. Some of them (e.g., acme.sh) often run as a service user (e.g., default install from FreeBSD ports) so the code runs unprivileged: just add a sudo (or doas) config to allow it to restart Apache/nginx.
diggan 1y ago
> I don't understand the tone of aggression against ACME and their plethora of clients. The older posts on the same website provided a bit more context for me to understand today's post better: - "Why I still have an old-school cert on my https site" - January 3, 2023 - https://rachelbythebay.com/w/2023/01/03/ssl/ https://rachelbythebay.com/w/2023/01/03/ssl/ - "Another look at the steps for issuing a cert" - January 4, 2023 - https://rachelbythebay.com/w/2023/01/04/cert/ https://rachelbythebay.com/w/2023/01/04/cert/
lucideer 1y ago
> I don't understand the tone of aggression against ACME and their plethora of clients. > ACME idea good, ACME implementation bad. Maybe I'm misreading but it sounds like you're on a similar page to the author. As they said at the top of the article: > Many of the existing clients are also scary code, and I was not about to run any of them on my machines. They haven't earned the right to run with privileges for my private keys and/or ability to frob the web server (as root!) with their careless ways. This might seem harsh but when I think it's a pretty fair perspective to have when running security-sensitive processes.
- giancarlostoro 1y ago
  Im not a container guru by any means (at least not yet?) but would docker not suffice these concerns?
  - TheNewsIsHere 1y ago
    My reading of the article suggested to me that the author took exception to the code that touched the keying material. Docker is immaterial to that problem. I won’t deign to speak for Rachel By The Bay (mother didn’t raise a fool, after all), but I expect Docker would be met with a similar regard. Which I do understand. Although I use Docker, I mainly use it personally for things I don’t want to spend much time on. I don’t really like it over other alternatives, but it makes standing up a lab service stupidly easy.
  - fpoling 1y ago
    The issue is that the client needs to access the private key, tell web server where various temporary files are during the certificate generation (unless the client uses DNS mode) and tell the web server about a new certificate to reload. To implement that many clients run as a root. Even if that root is in a docket container, this is needlessly elevated privileges especially given the complexity (again, needless) of many clients. The sad part is that it is trivial to run most of the clients with an account with no privileges that can access very few files and use a unix socket to tell the web server to reload the certificate. But this is not done. And then ideally at this point the web servers should if not implement then at least facilitate ACME protocol implementations, like, for example, redirect traffic requests from acme servers to another port with one-liner in config. But this is not the case.

immibis 1y ago

Some people don't want to be forced to run a bunch of stuff they don't understand on the server, and I agree with them. Sadly, security is a cat and mouse game, which means it's always evolving and you're forced to keep up - and it's inherent by the nature of the field, so we can't really blame anyone (unlike, say, being forced to integrate with the latest Google services to be allowed on the Play Store). At least you get to write your own ACME client if you want to. You don't have to use certbot, and there's no TPM-like behaviour locking you out of your own stuff.

spockz 1y ago
Given that keys probably need to be shared between multiple gateway/ingresses, how common is it to just use some HSM or another mechanism of exchanging the keys with all the instances? The acme client doesn’t have to run on the servers itself.
- tialaramex 1y ago
  > The acme client doesn’t have to run on the servers itself. This is really important to understand if you care about either: Actually engineering security at some scale or knowing what's actually going on in order to model it properly in your head. If you just want to make a web site so you can put up a blog about your new kitten, any of the tools is fine, you don't care, click click click, done. For somebody like Rachel or many HN readers, knowing enough of the technology to understand that the ACME client needn't run on your web servers is crucial. It also means you know that when some particular client you're evaluating needs to run on the web server that it's a limitation of that client not of the protocol - birds can't all fly, but flying is totally one of the options for birds, we should try an eagle not an emu if we want flying.
- immibis 1y ago
  You could if your domain was that valuable. Most aren't.
hannob 1y ago
> Some people don't want to be forced to run a bunch of stuff they > don't understand on the server, and I agree with them. Honest question: * Do you understand OS syscalls in detail? * Do you understand how your BIOS initializes your hardware? * Do you understand how modern filesystems work? * Do you understand the finer details of HTTP or TCP? Because... I don't. But I know enough about them that I'm quite convinced each of them is a lot more difficult to understand than ACME. And all of them and a lot more stuff are required if you want to run a web server.
- frogsRnice 1y ago
  Sure - but people are still free to decide where they draw the line. Each extra bit of software is an additional attack surface after all
- sussmannbaka 1y ago
  This point is so tired. I don’t understand how a thought forms in my neurons, eventually matures into a decision and how the wires in my head translate this into electrical pulses to my finger muscles to type this post so I guess I can’t have opinions about complexity.
- kjs3 1y ago
  I hear some variation of this line of 'reasoning' about once a week, and it's always followed by some variation of "...and that's why we shouldn't have to do all this security stuff you want us to do".
- fc417fc802 1y ago
  An OS is (at least generally) a prerequisite. If minimalism is your goal then you'd want to eliminate tangentially related things that aren't part of the underlying requirements. If you're a fan of left-pad I won't judge but don't expect me to partake without bitter complaints.
- snowwrestler 1y ago
  I get where you’re going with this, but in this particular case it might not be relevant because there’s a decent chance that Rachel By The Bay does actually understand all those things.
g-b-r 1y ago
> Some people don't want to be forced to run a bunch of stuff they don't understand on the server It's not just about not understanding, it's that more complex stuff is inherently more prone to security vulnerabilities, however well you think you reviewed its code.
- Avamander 1y ago
  > It's that more complex stuff is inherently more prone to security vulnerabilities That's overly simplifying it and ignores the part where the simple stuff is not secure to begin with. In the current context you could take a HTTP client with a formally verified TLS stack, would you really say it's inherently more vulnerable than a barebones HTTP client talking to a server over an unencrypted connection? I'd say there's a lot more exposed in that barebones client.
  - g-b-r 1y ago
    The alternative of the article was ACME vs other ways of getting TLS certificates, not https vs http. Of course plain http would be, generally, much more dangerous than a however complex encrypted connection
    - g-b-r 1y ago
      Why the heck was this downvoted
tptacek 1y ago
Non-ACME certs are basically over. The writing has been on the wall for a long time. I understand people being squeamish about it; we fear change. But I think it's a hopeful thing: the Web PKI is evolving. This is what that looks like: you can't evolve and retain everyone's prior workflows, and that has been a pathology across basically all Internet security standards work for decades.
- ipdashc 1y ago
  ACME is cool (compared to what came before it), but I'm kind of sad that EV certs never seemed to pan out at all. I feel like they're a neat concept, and had the potential to mitigate a lot of scams or phishing websites in an ideal world. (That said, discriminating between "big companies" and "everyone else who can't afford it" would definitely have some obvious downsides.) Does anyone know why they never took off?
  - johannes1234321 1y ago
    > Does anyone know why they never took off? Browser vendors at some point claimed it confused users and removed the highlight (I think the same browser vendors who try to remove the "confusing" URL bar ...) Aside from that EV certificates are slow to issue and phishers got similar enough EV certs making the whole thing moot.
  - tialaramex 1y ago
    EV can't actually work. It was always about branding for the for-profit CAs so that they have a premium product which helps the line go up. Let me give you a brief history - you did ask. In about 2005, the browser vendors and the Certificate Authorities began meeting to see if they could reach some agreement as neither had what they wanted and both might benefit from changes. This is the creation of the CA/Browser Forum aka CA/B Forum which still exists today. From the dawn of SSL the CAs had been on a race to the bottom on quality and price. Initially maybe somebody from a huge global audit firm that owns a CA turn up on a plane and talks to your VP of New Technology about this exciting new "World Wide Web" product, maybe somebody signs a $1M deal over ten years, and they issue a certificate for "Huge Corporation, Inc" with all the HQ address details, etc. and oh yeah, "www.hugecorp.example" should be on there because of that whole web thing, whatever that's about. Nerd stuff! By 2005 your web designer clicks a web page owned by some bozo in a country you've never heard of, types in the company credit card details, company gets charged $15 because it's "on sale" for a 3 year cert for www.mycompany.example and mail.mycompany.com is thrown in for free, so that's nice. Is it secure? Maybe? I dunno, I think it checked my email address? Whatever. The "real world address" field in this certificate now says "Not verified / Not verified / None" which is weird, but maybe that's normal? The CAs can see that if this keeps up in another decade they'll be charging $1 each for 10 year certificates, they need a better product and the browser vendors can make that happen. On the other hand the browser vendors have noticed that whereas auditors arriving by aeroplane was a bit much, "Our software checked their email address matched in the From line" is kinda crap as an "assurance" of "identity". So, the CA/B Baseline Requirements aka BRs are one result. Every CA agreed they'd do at least what the "baseline" required and in practice that's basically all they do because it'd cost extra to do more so why bother. The BRs started out pretty modest - but it's amazing what you find people were doing when you begin writing down the basics of what obviously they shouldn't do. For example, how about "No issuing certificates for names which don't exist" ? Sounds easy enough right? When "something.example" first comes into existence there shouldn't already be certificates for "something.example" because it didn't exist... right? Oops, lots of CAs had been issuing those certificates, reasoning that it's probably fine and hey, free money. Gradually the BRs got stricter, improving the quality of this baseline product in both terms of the technology and the business processes, this has been an enormous boon, because it's an agreement for the industry this ratchets things for everybody, so there's no race to the bottom on quality because your competitors aren't allowed to do worse than the baseline. On price, the same can't be said, zero cost certificates are what Let's Encrypt is most famous for after all. The other side of the deal is what the CAs wanted, they wanted UI for their new premium product. That's EV. Unlike many of the baseline requirements, this is very product focused (although to avoid being an illegal cartel it is forbidden for CA/B Forum to discuss products, pricing etc.) and so it doesn't make much technical sense. The EV documents basically say you get all of the Baseline, plus we're going to check the name of your business - here's how we'll check - and then the web browser is going to display that name. It's implied that these extra checks cost more money (they do, and so this product is much more expensive). So improvements to that baseline do help still, but they also help everybody who didn't buy the premium EV product. Now, why doesn't this work in practice? The DNS name or IP address in an "ordinary" certificate can be compared to reality automatically by the web browser. This site says it is news.ycombinator.com, it has a certificate for news.ycombinator.com, that's the same OK. Your browser performs this check, automatically and seamlessly, for every single HTTP transaction. Here on HN that's per page load, but on many sites you're doing transactions as you click UI or scroll the page, each is checked. With EV the checks must be done by a human, is this site really "Bob's Burgers" ? Actually wait, is it really "Bob's Burgers of Ohio, US" ? Worse, probably although you know them as Bob's Burgers, legally, as you'd see on their papers of incorporation they are "Smith Restaurant Holdings Inc." and they're registered in Delaware because of course they are. So now you're staring at the formal company name of a busines and trying to guess whether that's legitimate or a scam. But remember you can't just do this check once, scammers might find a way to redirect some traffic and so you need to check every individual transaction like your web browser does. Of course it's a tireless machine and you are not. So in practice this isn't effective.
throw0101b 1y ago
> Some people don't want to be forced to run a bunch of stuff they don't understand on the server, and I agree with them. There are a number of shell-based ACME clients whose prerequisites are: OpenSSL and cURL. You're probably already relying on OpenSSL and cURL for a bunch of things already. If you can read shell code you can step through the logic and understand what they're doing. Some of them (e.g., acme.sh) often run as a service user (e.g., default install from FreeBSD ports) so the code runs unprivileged: just add a sudo (or doas) config to allow it to restart Apache/nginx.

diggan 1y ago

> I don't understand the tone of aggression against ACME and their plethora of clients. The older posts on the same website provided a bit more context for me to understand today's post better: - "Why I still have an old-school cert on my https site" - January 3, 2023 - https://rachelbythebay.com/w/2023/01/03/ssl/ https://rachelbythebay.com/w/2023/01/03/ssl/ - "Another look at the steps for issuing a cert" - January 4, 2023 - https://rachelbythebay.com/w/2023/01/04/cert/ https://rachelbythebay.com/w/2023/01/04/cert/

lucideer 1y ago

> I don't understand the tone of aggression against ACME and their plethora of clients. > ACME idea good, ACME implementation bad. Maybe I'm misreading but it sounds like you're on a similar page to the author. As they said at the top of the article: > Many of the existing clients are also scary code, and I was not about to run any of them on my machines. They haven't earned the right to run with privileges for my private keys and/or ability to frob the web server (as root!) with their careless ways. This might seem harsh but when I think it's a pretty fair perspective to have when running security-sensitive processes.

giancarlostoro 1y ago
Im not a container guru by any means (at least not yet?) but would docker not suffice these concerns?
- TheNewsIsHere 1y ago
  My reading of the article suggested to me that the author took exception to the code that touched the keying material. Docker is immaterial to that problem. I won’t deign to speak for Rachel By The Bay (mother didn’t raise a fool, after all), but I expect Docker would be met with a similar regard. Which I do understand. Although I use Docker, I mainly use it personally for things I don’t want to spend much time on. I don’t really like it over other alternatives, but it makes standing up a lab service stupidly easy.
- fpoling 1y ago
  The issue is that the client needs to access the private key, tell web server where various temporary files are during the certificate generation (unless the client uses DNS mode) and tell the web server about a new certificate to reload. To implement that many clients run as a root. Even if that root is in a docket container, this is needlessly elevated privileges especially given the complexity (again, needless) of many clients. The sad part is that it is trivial to run most of the clients with an account with no privileges that can access very few files and use a unix socket to tell the web server to reload the certificate. But this is not done. And then ideally at this point the web servers should if not implement then at least facilitate ACME protocol implementations, like, for example, redirect traffic requests from acme servers to another port with one-liner in config. But this is not the case.