5 ms·

Based on their stated 225M sites and a renewal period of 90 days, they're probably averaging around 40 certificates per second. That's only an order of magnitu

Based on their stated 225M sites and a renewal period of 90 days, they're probably averaging around 40 certificates per second. That's only an order of magnitude higher than bitcoin; I wouldn't call it an indication of an ability to scale to a particularly large amount of traffic.

RL_Quine 6y ago
Bitcoin is ECDSA verification, letsencrypt is generating RSA signatures, the two aren't even remotely comparable.
- schoen 6y ago
  But the cryptographic bottleneck is in HSMs, not in database servers (database servers don't generate the digital signatures, they just have to store them after they've been generated).
- SulfurHexaFluri 6y ago
  Bitcoin is actually not limited by compute power at all. Its an artificial cap on transaction rate to prevent the blockchain from expanding too large and preventing normal users from hosting the whole thing. You can see the blockchain size was growing exponentially but then switches to linear as we hit the transaction cap and it now sits at about 350GB
tyingq 6y ago
Does "certbot renew" talk to the mothership at all if no certs are ready for renewal? If it does, most setups I've seen run the renewal once or twice a day since it only does the renew when you're down to 30 days left. There may also be some OCSP related traffic.
- mike_d 6y ago
  Certbot will look at the expiration timestamp on your local certs without talking to Lets Encrypt.
  - flatiron 6y ago
    There is a —force though. I had to do it once because they thought I got a bad cert (don’t remember the details)
PeterCorless 6y ago
Yes. They are not doing a very heavy computational workload. Typical heavy-duty servers these days can do 100k's or millions of TPS. 40 TPS is a really, really, really light load. Further, I was looking at those new server specs. There's an error I think? The server config on the Dell site shows 2x 8 GB DRIMMs, for 16 GB RAM per sever, whereas the article says 2 TB! With only 16GB of RAM, but 153.6 TB of NVMe storage, the real issue here is memory limitation for a general-purpose SQL database or a typical high-availability NoSQL database. Check my math: 153600 GB storage / 16 GB memory = 9600:1 ratio Consider, by comparison that a high data volume AWS i3en.24xlarge has 60TB of NVMe storage but 768 GB of RAM. A 78:1 ratio. If the article is correct, and the error is in the config on the Dell page (not the blog), and this server is actually 2 TB RAM, then that's another story. That'd make it a ratio of 153600 / 2000 = ~77:1. Quite in line with the AWS I3en. But then it would baffle me why you would only get 40 TPS out of such a beast. Check my logic. Did I miss something?
- edoceo 6y ago
  They don't only GET 40 TPS, that value is an estimate what they serve (above it was site/90 days). They have capacity for much, much more with that hardware
- z77dj3kl 6y ago
  Yes, you can customize the config on Dell's site to upgrade the memory.
- cm2187 6y ago
  Don’t these certificates have long RSA keys which are more expensive computationally? Though I guess that doesn’t have to happen on the database server.
  - tialaramex 6y ago
    The only RSA computations Let's Encrypt need to do are: * Signature by their RSA Intermediate (currently R3, with R4 on hot standby) - which will be a dedicated piece of hardware - to issue a subscriber's certificate. In practice this happens twice, as a poisoned pre-certificate to obtain proof of logging from public log servers, and then the real certificate with the proofs baked inside it. * Signatures by their OCSP signer periodically on an OCSP response for each certificate saying it's still trustworthy for a fixed period. Again this will be inside an HSM. * Signature verification on a subscriber's CSR. To "complete the circuit" it's helpful that Let's Encrypt actually confirms you know the private key corresponding to the public key you wanted a certificate for, the signature on your CSR does this. Some people don't think this is necessary, but I believe Let's Encrypt do it anyway. You're correct that none of this happens on the database servers. I guess it's possible their servers use TLS to secure the MariaDB connections, in which case a small amount of either ECDSA or RSA computation happens each time such a connection is set up or torn down like at any outfit using TLS, but those database connections are cached in a sane system so that wouldn't be very often.
dylan604 6y ago
They might average that, but we all know averages only work on paper. For example, AWS has tutorials that provide instructions on how to setup TLS in a LAMP stack running on Linux 2 EC2s. As part of the Let's Encrypt setup, they provide a crontab entry that runs twice a day with a copy option to paste. How many EC2s all hit the Let's Encrypt server at that exact time? Since EC2s default to UTC time, that means that servers are not offsetting those requests by timezones, so that means an even bigger spike.
- Denvercoder9 6y ago
  Certbot has a random sleep before it does anything when run non-interactively.
- SulfurHexaFluri 6y ago
  Its worth noting that running certbot / LE twice a day doesn't actually hit the LE server twice a day. It just checks the certificate dates locally and if they have been renewed in the last month it does nothing. I guess you still have a peak time of 00:00 UTC every day though unless people are using servers set to their local time.
  - tialaramex 6y ago
    I believe modern Certbot also does OCSP. The intent here is, if for any reason your certificate was revoked it makes sense to try to obtain a new certificate even if it hasn't expired. Even if it can't, perhaps the operator will notice a logged reason from Certbot when they did not yet notice their certificate was revoked. Examples of reasons your certificate might have been revoked: * You re-used a private key from somewhere, perhaps because you don't understand what "private" means, and other copies of that key leaked * You didn't re-use the private key but your "clever" backup strategy involves putting the private key file in a public directory named /backup/server/ on your web server and somebody found it * You use the same certificate for dozens of domain names in your SEO business and yesterday you sold a name for $10k. Hooray. The new owner immediately revoked all certificates for that name which they're entitled to do. * Your tooling is busted and the "random" numbers it picked aren't very random. (e.g. Debian OpenSSL bug) * A bug at Let's Encrypt means their Ten Blessed Methods implementation was inadequate for some subset of issuances and rather than cross their fingers the team decided to revoke all certificates issued with the inadequate control. * Let's Encrypt discovers you're actually a country sanctioned by the US State Department, perhaps in some thin disguise such as an "independent" TV station in a country that doesn't have any independent media whatsoever. It is illegal for them to provide you with services and you were supposed to already know that. So that is a network connection, but not to the Let's Encrypt servers described in this story. In practice OCSP is done by a big CA by periodically computing OCSP responses for every single unexpired certificate (either saying it's still valid, or not), and then providing those to a CDN and the CDN acts as "OCSP server" returning the appropriate OCSP response when asked without itself having possession of any cryptographic materials.

RL_Quine 6y ago

Bitcoin is ECDSA verification, letsencrypt is generating RSA signatures, the two aren't even remotely comparable.

schoen 6y ago
But the cryptographic bottleneck is in HSMs, not in database servers (database servers don't generate the digital signatures, they just have to store them after they've been generated).
SulfurHexaFluri 6y ago
Bitcoin is actually not limited by compute power at all. Its an artificial cap on transaction rate to prevent the blockchain from expanding too large and preventing normal users from hosting the whole thing. You can see the blockchain size was growing exponentially but then switches to linear as we hit the transaction cap and it now sits at about 350GB

tyingq 6y ago

Does "certbot renew" talk to the mothership at all if no certs are ready for renewal? If it does, most setups I've seen run the renewal once or twice a day since it only does the renew when you're down to 30 days left. There may also be some OCSP related traffic.

mike_d 6y ago
Certbot will look at the expiration timestamp on your local certs without talking to Lets Encrypt.
- flatiron 6y ago
  There is a —force though. I had to do it once because they thought I got a bad cert (don’t remember the details)

PeterCorless 6y ago

Yes. They are not doing a very heavy computational workload. Typical heavy-duty servers these days can do 100k's or millions of TPS. 40 TPS is a really, really, really light load. Further, I was looking at those new server specs. There's an error I think? The server config on the Dell site shows 2x 8 GB DRIMMs, for 16 GB RAM per sever, whereas the article says 2 TB! With only 16GB of RAM, but 153.6 TB of NVMe storage, the real issue here is memory limitation for a general-purpose SQL database or a typical high-availability NoSQL database. Check my math: 153600 GB storage / 16 GB memory = 9600:1 ratio Consider, by comparison that a high data volume AWS i3en.24xlarge has 60TB of NVMe storage but 768 GB of RAM. A 78:1 ratio. If the article is correct, and the error is in the config on the Dell page (not the blog), and this server is actually 2 TB RAM, then that's another story. That'd make it a ratio of 153600 / 2000 = ~77:1. Quite in line with the AWS I3en. But then it would baffle me why you would only get 40 TPS out of such a beast. Check my logic. Did I miss something?

edoceo 6y ago
They don't only GET 40 TPS, that value is an estimate what they serve (above it was site/90 days). They have capacity for much, much more with that hardware
z77dj3kl 6y ago
Yes, you can customize the config on Dell's site to upgrade the memory.
cm2187 6y ago
Don’t these certificates have long RSA keys which are more expensive computationally? Though I guess that doesn’t have to happen on the database server.
- tialaramex 6y ago
  The only RSA computations Let's Encrypt need to do are: * Signature by their RSA Intermediate (currently R3, with R4 on hot standby) - which will be a dedicated piece of hardware - to issue a subscriber's certificate. In practice this happens twice, as a poisoned pre-certificate to obtain proof of logging from public log servers, and then the real certificate with the proofs baked inside it. * Signatures by their OCSP signer periodically on an OCSP response for each certificate saying it's still trustworthy for a fixed period. Again this will be inside an HSM. * Signature verification on a subscriber's CSR. To "complete the circuit" it's helpful that Let's Encrypt actually confirms you know the private key corresponding to the public key you wanted a certificate for, the signature on your CSR does this. Some people don't think this is necessary, but I believe Let's Encrypt do it anyway. You're correct that none of this happens on the database servers. I guess it's possible their servers use TLS to secure the MariaDB connections, in which case a small amount of either ECDSA or RSA computation happens each time such a connection is set up or torn down like at any outfit using TLS, but those database connections are cached in a sane system so that wouldn't be very often.

dylan604 6y ago

They might average that, but we all know averages only work on paper. For example, AWS has tutorials that provide instructions on how to setup TLS in a LAMP stack running on Linux 2 EC2s. As part of the Let's Encrypt setup, they provide a crontab entry that runs twice a day with a copy option to paste. How many EC2s all hit the Let's Encrypt server at that exact time? Since EC2s default to UTC time, that means that servers are not offsetting those requests by timezones, so that means an even bigger spike.

Denvercoder9 6y ago
Certbot has a random sleep before it does anything when run non-interactively.
SulfurHexaFluri 6y ago
Its worth noting that running certbot / LE twice a day doesn't actually hit the LE server twice a day. It just checks the certificate dates locally and if they have been renewed in the last month it does nothing. I guess you still have a peak time of 00:00 UTC every day though unless people are using servers set to their local time.
- tialaramex 6y ago
  I believe modern Certbot also does OCSP. The intent here is, if for any reason your certificate was revoked it makes sense to try to obtain a new certificate even if it hasn't expired. Even if it can't, perhaps the operator will notice a logged reason from Certbot when they did not yet notice their certificate was revoked. Examples of reasons your certificate might have been revoked: * You re-used a private key from somewhere, perhaps because you don't understand what "private" means, and other copies of that key leaked * You didn't re-use the private key but your "clever" backup strategy involves putting the private key file in a public directory named /backup/server/ on your web server and somebody found it * You use the same certificate for dozens of domain names in your SEO business and yesterday you sold a name for $10k. Hooray. The new owner immediately revoked all certificates for that name which they're entitled to do. * Your tooling is busted and the "random" numbers it picked aren't very random. (e.g. Debian OpenSSL bug) * A bug at Let's Encrypt means their Ten Blessed Methods implementation was inadequate for some subset of issuances and rather than cross their fingers the team decided to revoke all certificates issued with the inadequate control. * Let's Encrypt discovers you're actually a country sanctioned by the US State Department, perhaps in some thin disguise such as an "independent" TV station in a country that doesn't have any independent media whatsoever. It is illegal for them to provide you with services and you were supposed to already know that. So that is a network connection, but not to the Let's Encrypt servers described in this story. In practice OCSP is done by a big CA by periodically computing OCSP responses for every single unexpired certificate (either saying it's still valid, or not), and then providing those to a CDN and the CDN acts as "OCSP server" returning the appropriate OCSP response when asked without itself having possession of any cryptographic materials.