5 ms·

Let's Encrypt AUTHZ Reuse and Eternal Account Key

107 points by dc352 10y ago

TheDong 10y ago
In general, you'll automate lets-encrypt. That means you'll have some file or group of files (a cronjob, systemd timer, or what have you) that's capable of producing new keys. If an attacker can read files on the filesystem as root, then they can simply read whatever automation scripts you have and issue new certs as well. This isn't always true (e.g. if it depends on an IAM role, they'll have to get data from the instance metadata service as well), but the point is generally true. If an attacker has root filesystem access on a box which can issue certs, it makes miniscule difference whether they need to grab an account key or bash script; either way you've probably lost. You could also, however, have more locked-down boxes which issue certs, encrypt them for their intended hosts, and transport them over, at which point all of your host boxes no longer have account keys to worry about. You could also encrypt-and-ship the account key from the box to elsewhere, and then remove it. Frankly, you're making a lot of fuss about absolutely nothing.
- dc352 10y ago
  The difference is that I need to access your file system only once to get your account key. I need permanent access to exploit your automation. Attackers can also scale revenues, potentially, by selling account keys to third parties.
  - TheDong 10y ago
    For most types of automation they only need to access your filesystem once as well, but accessing the filesystem as root once also implies they can again (albeit at increased risk of detection). Your second sentence is pure FUD imo.
- zaroth 10y ago
  If you have to authenticate the domain every time, there's a good chance you capture that specific HTTP request traffic and log it to another server which isn't compromised. You now have an audit trail of the cert being issued and you have a chance of catching it. If the account_key "walks away" and within 90 days they can get a cert without anything hitting your server, you have no audit log of the certificate being issued and are less likely to know your TLS is owned. So if I understand correctly, the account_key is an unnecessary risk with very little return. (see my other post). I just immediately delete it every time, I don't see the point of keeping it. Even key revocation does not require the account key, just the certificate private key. So really, what is the point? Unless there's something you actually need the account_key for... I feel like this was a failure to KISS. EDIT: Mentioned downthread, there are also Certificate Transparency logs which can help you catch an unauthorized issuance, but that's not going to let you wire up an instant SMS through Twilio on every domain validation.
  - pfg 10y ago
    With the stolen account key scenario, you still get an audit trail through Certificate Transparency, as Let's Encrypt logs all certificates they issue.
  - lftl 10y ago
    > Even key revocation does not require the account key, just the certificate private key. So really, what is the point? Unless there's something you actually need the account_key for... I feel like this was a failure to KISS. Can you revoke with either one? If someone steals my account key, and creates a new certificate private key do I have no way of revoking that since I don't have the new certificate private key?
    - pfg 10y ago
      Yes, both keys can be used to revoke, so as long as you still have either the account key or all certificate private keys, you're good.
veeti 10y ago
> The surprising aspect is that Authz has a validity of 300 days (which is likely to be increased) This is not true. Authorization lifetimes are only going to get shorter and shorter [1], and have already been reduced to 90 days. [1] https://community.letsencrypt.org/t/upcoming-api-changes/17947 https://community.letsencrypt.org/t/upcoming-api-changes/179...
- dc352 10y ago
  I've updated the blog post. I don't think, though, it has any material impact on the rest of the text.
  - pfg 10y ago
    Why not? Right now, compromised account keys give you the ability to issue certificates that are valid about as long as certificates are valid (or, in the worst case, +90 days from the original expiration date). Just about any scenario where the account key is compromised is one where the certificate's private key is compromised as well (not to mention typically one where you can just solve another challenge), so you're not really in a much worse position. Even less of a concern once authorizations are valid for only ~7 days.
  - lol768 10y ago
    > I've updated the blog post You still state in your updated post that: "The surprising aspect is that Authz has a validity of 300 days (which is likely to be increased)" [emphasis mine]. This would appear to be incorrect based on the source cited above , where it is stated at "Eventually, we'd like to make authorization objects much shorter than certificate lifetimes, probably 7 days." Perhaps it's worth updating the post again in light of this?
    - dc352 10y ago
      It is an interesting one - here is the original source (and there were many more mentioning 300 days). https://community.letsencrypt.org/t/dns-authorization-lifetime-on-le-servers/17007/2 https://community.letsencrypt.org/t/dns-authorization-lifeti... It is from 14 June 2016 (a few months back), @pfg states "The CA/B Forum is currently developing new rules for domain validation and is probably going to settle on a validation period that is significantly longer than the 300 days currently in use ..." Is there an authority to say which way it will go?
chad-autry 10y ago
Or just destroy the auth key once a certificate is issued. No need to keep it around. Can go through the expected validation process for a new one when renewing.
- TheDong 10y ago
  Absolutely not. You need it for revocation. Destroying it is thus dangerous and reduces your security. Edit: I'm wrong, see pfg's comment below. See my top-level comment for why I don't think it matters still.
  - pfg 10y ago
    Revocation is also possible using the certificate's private key, but I agree that the risk of keeping the account key is not worth the hassle of deleting it straight away. As it stands, authorizations are valid as long as certificates, so the worst case is you're compromised on the 89th day of validity and get another certificate valid for the next 90 days. Soon, that'll be ~7 days, so even less of a concern.
    - dc352 10y ago
      So do you imply that the account key is created from scratch for every new certificate? Why we were surprised (and we don't say the implementation is necessarily wrong!) is that I can use the account key anywhere. If the genuine user keeps refreshing authz's, it will keep the stolen account key operational as well. That's my understanding. I may be wrong, but if so, I don't quite yet understand the logic behind authz.
      - pfg 10y ago
        No, account keys are typically persisted and re-used, though specifics depend on the client implementation. I still think this is largely a non-issue. If you compromise the target system, you can just keep stealing private keys as they are being renewed. This would even decrease the chances of being detected, because using the stolen account key to issue new certificates would lead to those certificates popping up on various Certificate Transparency log servers, which might be noticed by the domain owner. You're already in a game-over scenario if you don't detect the compromise, the account key doesn't make things significantly worse. Think of it like the credentials for a hosting provider where you bought your traditional SSL certificate being stolen. Not really a new threat.
pfg 10y ago
> I did expect that each request includes a fresh verification of the domain control. And I also expect that substantial majority of people out there believe in this process / data flow as that’s how certificates have always been verified – for each new certificate, you either have to send an email from a given domain or put a file to a web server’s folder. This is not correct. Certificate Authorities are allowed to reuse authorizations for (I believe) up to 36 months, so Let's Encrypt is already significantly below that (with 90 days right now, and probably closer to something like ~7 days soon). Many CAs I've used in the past worked like this.
- 0x0 10y ago
  This is interesting in the context of buying and selling domain names. Let's say you pay big $$$ for a fancy .com because you're launching a new product with that name or whatever. The seller could actually possess certificates (or authorizations) that allow them to MITM your fancy expensive domain for up to, what, 3 years. Never really thought of that. CT logs could prove on the date of purchase that your new domain no longer has any valid certificates in the hands of third parties, but if the third party still holds an authorization, they could sneakily obtain new certificates long after the sale? Or - are authorizations also logged somehow to CT-like logs?
  - pfg 10y ago
    Yeah, it's definitely something I'm hoping the CA/B Forum will address at some point. Something like a month seems like a sweet spot between not being annoying for new customers when they're in the process of getting multiple certs and not being a huge deal when domains are transferred. I'm not aware of any kind of "Authorization Transparency" effort. Let's Encrypt were thinking about a way to share logs at some point (IIRC), so it might get included in those, but that's a maybe for the future and not a general solution for all CAs.
zaroth 10y ago
I think it's also important to note... you don't need the account key at all if you can re-do the domain verification every time. In most cases I delete the account key right after verifying the domain and getting a cert, and let certbot generate a new one every time. We route the '/.well-known/acme-challenge' request to the standalone server (we use haproxy ACLs 'path_beg' to route it to a dedicated port on localhost) so it hasn't been a problem running the validation every time for every new cert. And per TFA, the timeout for validation on their side (what they call Authz or authorization token) is about the same as the timeout for renewal, so you can't really depend on not needing to validate the domain even with the account key. I'd much rather have the code path be identical every time, rather than different for new cert vs renewal cert, and then also different every other time you renew due to authz expiry.
- schoen 10y ago
  > In most cases I delete the account key right after verifying the domain and getting a cert, and let certbot generate a new one every time. An alternative would be deactivating the authorizations.
pg_is_a_butt 10y ago
uh... who reuses anything with letsencrypt? your renew process should install letsencrypt clean every time... ideally in a clean VM. you're all idiots.
lightedman 10y ago
"This is not true any more with re-usable AUTHZ tokens" I must re-iterate my own 10+ year experiences with Token auth - once authorized, end that token, period, or you're asking for problems.
- lightedman 10y ago
  Feel free to downvote - I retired because I was right and proved it AND GOT PAID FOR IT. Come retire at 28 years of age, guys, if you bother to listen.
fomojola 10y ago
My recommendation is to use the DNS challenge implementation with an appropriate DNS provider (I use Route 53): you can do all your certificate generation on a single locked-down machine and then just distribute the certs out to your machines. No need to deal with path_beg or check which host in the fleet is responding to the acme-challenge request.
hlandau 10y ago
>And I also expect that substantial majority of people out there believe in this process / data flow as that’s how certificates have always been verified – for each new certificate, you either have to send an email from a given domain or put a file to a web server’s folder. Err... what? I don't think any CA validation process has ever involved sending e. mail, and I'd be very concerned if it did, since the origin of an e. mail is not in general authenticated. Unless it's in response to a CA e. mail which includes a token, of course. 'Obtain reusable authorization' followed by 'issue certificate' is hardly unprecedented, and it's a sensible way to structure things if you don't have a commercial interest in applying the friction of a financial transaction to the act of issuance. Startcom, for example, used this model (there are many issues with them, this wasn't one of them). I think CAcert may also use this model, unrecognised though they may be. The ACME protocol is also a standard. It's right there, I'm surprised people aren't reading it, especially when they're rigging up custom solutions. https://tools.ietf.org/html/draft-ietf-acme-acme-02 https://tools.ietf.org/html/draft-ietf-acme-acme-02
packetized 10y ago
If you're upset about the expected operation of a service based on open-source software (github.com/letsencrypt/boulder), then I'd strongly recommend reading the code before use. Additionally: "I also expect that our Enigma Bridge Cloud HSM service (check our PKI product with hardware key security) can securely control storage and use of the account key to prevent thefts of the keys and their subsequent misuse." Using this as a way to promote your product and not reading the code first is a serious failure of tradecraft on your part.

107 points by dc352 10y ago

In general, you'll automate lets-encrypt. That means you'll have some file or group of files (a cronjob, systemd timer, or what have you) that's capable of producing new keys. If an attacker can read files on the filesystem as root, then they can simply read whatever automation scripts you have and issue new certs as well. This isn't always true (e.g. if it depends on an IAM role, they'll have to get data from the instance metadata service as well), but the point is generally true. If an attacker has root filesystem access on a box which can issue certs, it makes miniscule difference whether they need to grab an account key or bash script; either way you've probably lost. You could also, however, have more locked-down boxes which issue certs, encrypt them for their intended hosts, and transport them over, at which point all of your host boxes no longer have account keys to worry about. You could also encrypt-and-ship the account key from the box to elsewhere, and then remove it. Frankly, you're making a lot of fuss about absolutely nothing.

dc352 10y ago
The difference is that I need to access your file system only once to get your account key. I need permanent access to exploit your automation. Attackers can also scale revenues, potentially, by selling account keys to third parties.
- TheDong 10y ago
  For most types of automation they only need to access your filesystem once as well, but accessing the filesystem as root once also implies they can again (albeit at increased risk of detection). Your second sentence is pure FUD imo.
zaroth 10y ago
If you have to authenticate the domain every time, there's a good chance you capture that specific HTTP request traffic and log it to another server which isn't compromised. You now have an audit trail of the cert being issued and you have a chance of catching it. If the account_key "walks away" and within 90 days they can get a cert without anything hitting your server, you have no audit log of the certificate being issued and are less likely to know your TLS is owned. So if I understand correctly, the account_key is an unnecessary risk with very little return. (see my other post). I just immediately delete it every time, I don't see the point of keeping it. Even key revocation does not require the account key, just the certificate private key. So really, what is the point? Unless there's something you actually need the account_key for... I feel like this was a failure to KISS. EDIT: Mentioned downthread, there are also Certificate Transparency logs which can help you catch an unauthorized issuance, but that's not going to let you wire up an instant SMS through Twilio on every domain validation.
- pfg 10y ago
  With the stolen account key scenario, you still get an audit trail through Certificate Transparency, as Let's Encrypt logs all certificates they issue.
- lftl 10y ago
  > Even key revocation does not require the account key, just the certificate private key. So really, what is the point? Unless there's something you actually need the account_key for... I feel like this was a failure to KISS. Can you revoke with either one? If someone steals my account key, and creates a new certificate private key do I have no way of revoking that since I don't have the new certificate private key?
  - pfg 10y ago
    Yes, both keys can be used to revoke, so as long as you still have either the account key or all certificate private keys, you're good.

veeti 10y ago

> The surprising aspect is that Authz has a validity of 300 days (which is likely to be increased) This is not true. Authorization lifetimes are only going to get shorter and shorter [1], and have already been reduced to 90 days. [1] https://community.letsencrypt.org/t/upcoming-api-changes/17947 https://community.letsencrypt.org/t/upcoming-api-changes/179...

dc352 10y ago
I've updated the blog post. I don't think, though, it has any material impact on the rest of the text.
- pfg 10y ago
  Why not? Right now, compromised account keys give you the ability to issue certificates that are valid about as long as certificates are valid (or, in the worst case, +90 days from the original expiration date). Just about any scenario where the account key is compromised is one where the certificate's private key is compromised as well (not to mention typically one where you can just solve another challenge), so you're not really in a much worse position. Even less of a concern once authorizations are valid for only ~7 days.
- lol768 10y ago
  > I've updated the blog post You still state in your updated post that: "The surprising aspect is that Authz has a validity of 300 days (which is likely to be increased)" [emphasis mine]. This would appear to be incorrect based on the source cited above , where it is stated at "Eventually, we'd like to make authorization objects much shorter than certificate lifetimes, probably 7 days." Perhaps it's worth updating the post again in light of this?
  - dc352 10y ago
    It is an interesting one - here is the original source (and there were many more mentioning 300 days). https://community.letsencrypt.org/t/dns-authorization-lifetime-on-le-servers/17007/2 https://community.letsencrypt.org/t/dns-authorization-lifeti... It is from 14 June 2016 (a few months back), @pfg states "The CA/B Forum is currently developing new rules for domain validation and is probably going to settle on a validation period that is significantly longer than the 300 days currently in use ..." Is there an authority to say which way it will go?

chad-autry 10y ago

Or just destroy the auth key once a certificate is issued. No need to keep it around. Can go through the expected validation process for a new one when renewing.

TheDong 10y ago
Absolutely not. You need it for revocation. Destroying it is thus dangerous and reduces your security. Edit: I'm wrong, see pfg's comment below. See my top-level comment for why I don't think it matters still.
- pfg 10y ago
  Revocation is also possible using the certificate's private key, but I agree that the risk of keeping the account key is not worth the hassle of deleting it straight away. As it stands, authorizations are valid as long as certificates, so the worst case is you're compromised on the 89th day of validity and get another certificate valid for the next 90 days. Soon, that'll be ~7 days, so even less of a concern.
  - dc352 10y ago
    So do you imply that the account key is created from scratch for every new certificate? Why we were surprised (and we don't say the implementation is necessarily wrong!) is that I can use the account key anywhere. If the genuine user keeps refreshing authz's, it will keep the stolen account key operational as well. That's my understanding. I may be wrong, but if so, I don't quite yet understand the logic behind authz.
    - pfg 10y ago
      No, account keys are typically persisted and re-used, though specifics depend on the client implementation. I still think this is largely a non-issue. If you compromise the target system, you can just keep stealing private keys as they are being renewed. This would even decrease the chances of being detected, because using the stolen account key to issue new certificates would lead to those certificates popping up on various Certificate Transparency log servers, which might be noticed by the domain owner. You're already in a game-over scenario if you don't detect the compromise, the account key doesn't make things significantly worse. Think of it like the credentials for a hosting provider where you bought your traditional SSL certificate being stolen. Not really a new threat.

pfg 10y ago

> I did expect that each request includes a fresh verification of the domain control. And I also expect that substantial majority of people out there believe in this process / data flow as that’s how certificates have always been verified – for each new certificate, you either have to send an email from a given domain or put a file to a web server’s folder. This is not correct. Certificate Authorities are allowed to reuse authorizations for (I believe) up to 36 months, so Let's Encrypt is already significantly below that (with 90 days right now, and probably closer to something like ~7 days soon). Many CAs I've used in the past worked like this.

0x0 10y ago
This is interesting in the context of buying and selling domain names. Let's say you pay big $$$ for a fancy .com because you're launching a new product with that name or whatever. The seller could actually possess certificates (or authorizations) that allow them to MITM your fancy expensive domain for up to, what, 3 years. Never really thought of that. CT logs could prove on the date of purchase that your new domain no longer has any valid certificates in the hands of third parties, but if the third party still holds an authorization, they could sneakily obtain new certificates long after the sale? Or - are authorizations also logged somehow to CT-like logs?
- pfg 10y ago
  Yeah, it's definitely something I'm hoping the CA/B Forum will address at some point. Something like a month seems like a sweet spot between not being annoying for new customers when they're in the process of getting multiple certs and not being a huge deal when domains are transferred. I'm not aware of any kind of "Authorization Transparency" effort. Let's Encrypt were thinking about a way to share logs at some point (IIRC), so it might get included in those, but that's a maybe for the future and not a general solution for all CAs.

zaroth 10y ago

I think it's also important to note... you don't need the account key at all if you can re-do the domain verification every time. In most cases I delete the account key right after verifying the domain and getting a cert, and let certbot generate a new one every time. We route the '/.well-known/acme-challenge' request to the standalone server (we use haproxy ACLs 'path_beg' to route it to a dedicated port on localhost) so it hasn't been a problem running the validation every time for every new cert. And per TFA, the timeout for validation on their side (what they call Authz or authorization token) is about the same as the timeout for renewal, so you can't really depend on not needing to validate the domain even with the account key. I'd much rather have the code path be identical every time, rather than different for new cert vs renewal cert, and then also different every other time you renew due to authz expiry.

schoen 10y ago
> In most cases I delete the account key right after verifying the domain and getting a cert, and let certbot generate a new one every time. An alternative would be deactivating the authorizations.

pg_is_a_butt 10y ago

uh... who reuses anything with letsencrypt? your renew process should install letsencrypt clean every time... ideally in a clean VM. you're all idiots.

lightedman 10y ago

"This is not true any more with re-usable AUTHZ tokens" I must re-iterate my own 10+ year experiences with Token auth - once authorized, end that token, period, or you're asking for problems.

lightedman 10y ago
Feel free to downvote - I retired because I was right and proved it AND GOT PAID FOR IT. Come retire at 28 years of age, guys, if you bother to listen.

fomojola 10y ago

My recommendation is to use the DNS challenge implementation with an appropriate DNS provider (I use Route 53): you can do all your certificate generation on a single locked-down machine and then just distribute the certs out to your machines. No need to deal with path_beg or check which host in the fleet is responding to the acme-challenge request.

hlandau 10y ago

>And I also expect that substantial majority of people out there believe in this process / data flow as that’s how certificates have always been verified – for each new certificate, you either have to send an email from a given domain or put a file to a web server’s folder. Err... what? I don't think any CA validation process has ever involved sending e. mail, and I'd be very concerned if it did, since the origin of an e. mail is not in general authenticated. Unless it's in response to a CA e. mail which includes a token, of course. 'Obtain reusable authorization' followed by 'issue certificate' is hardly unprecedented, and it's a sensible way to structure things if you don't have a commercial interest in applying the friction of a financial transaction to the act of issuance. Startcom, for example, used this model (there are many issues with them, this wasn't one of them). I think CAcert may also use this model, unrecognised though they may be. The ACME protocol is also a standard. It's right there, I'm surprised people aren't reading it, especially when they're rigging up custom solutions. https://tools.ietf.org/html/draft-ietf-acme-acme-02 https://tools.ietf.org/html/draft-ietf-acme-acme-02

packetized 10y ago

If you're upset about the expected operation of a service based on open-source software (github.com/letsencrypt/boulder), then I'd strongly recommend reading the code before use. Additionally: "I also expect that our Enigma Bridge Cloud HSM service (check our PKI product with hardware key security) can securely control storage and use of the account key to prevent thefts of the keys and their subsequent misuse." Using this as a way to promote your product and not reading the code first is a serious failure of tradecraft on your part.