5 ms·
What are they storing on this server that requires 150Tb of storage and millions of IOPS?
by ryanworl 6y ago
What are they storing on this server that requires 150Tb of storage and millions of IOPS?
- dewey 6y ago> What exactly are we doing with these servers? Our CA software, Boulder, uses MySQL-style schemas and queries to manage subscriber accounts and the entire certificate issuance process.
- jeffbee 6y agoThere's nothing in that sentence that implies they'd need even 100 IOPS, much less 20 million.
- gbrown_ 6y agoThe post doesn't specify requirements or application level targets for performance. They show a couple of good latency improvements but don't describe the business or technical impact. The closest we get is this. > If this database isn’t performing well enough, it can cause API errors and timeouts for our subscribers. What are the SLO's? How was this being met (or not) before vs after the hardware upgrade? There's a lot of additional context that could have been added in this post. It's not a bad post but instead it simply reduces down to this new hardware is faster than our old hardware.
- stefan_ 6y agoWhat exactly needs to be stored once the certificate is created and published in the hash tree? It seems like the kind of data that possibly needn't be stored at all or onto something like Glacier for archival.
- Conclusionist 6y agoGoing to guess it's for OCSP responses.
- stefan_ 6y agoI'm not sure, e.g. Chrome doesn't do OCSP by default, lots of embedded clients like curl won't either. Unless the protocol is terribly broken, that also seems like the kind of use case where 99% of queries just come out of cache and should never hit a database.
- tialaramex 6y agoLet's Encrypt still has to publish OCSP responses for every non-expired leaf certificate, at least in time that you can always get a new OCSP response before the previous one expires. In practice they have a tighter schedule so that there's a period between "We are not meeting our self-imposed deadline" and "The Internet broke, oops" in which staff can figure out the problem and fix it. To do this they automatically generate and sign OCSP responses (the vast majority of which will just say the certificate is still good) on a periodic cycle, and then they deliver them in bulk to a CDN. The CDN is who your client (or server if you do OCSP stapling, which you ideally should) talks to when checking OCSP. To generate those responses they need a way (hey, a database) to get the set of all certificates which have not yet expired and whether those certificates are revoked or not.
- siwyd 6y agoFYI, the intermediate CA's signed by their new Root X2 certificate won't have OCSP URLs anymore. Source: https://letsencrypt.org/2020/09/17/new-root-and-intermediates.html#why-we-didn-t-issue-an-ocsp-responder https://letsencrypt.org/2020/09/17/new-root-and-intermediate...
- cipherboy 6y agoAFAIK, nobody has suggested removal of OCSP from end-entity certificates. This article you linked (and the comment you wrote) is purely about removal from intermediate CA certificates. The majority of OCSP traffic will probably be for end-entity certificates; most OCSP validation (in browsers and cryptographic libraries) is end-entity validation, not leaf-and-chain. Removal of intermediate CA's OCSP is probably not really relevant to their overall OCSP performance numbers (and if it was, it was likely cached already).
- abrookewood 6y agoMy thoughts exactly ... They are creating and storing text. Only thing I can think is that they don't actually need the storage, but just want the lowest possible latency by having a large number of drives.
- ksec 6y agoWell if Latency were an issue or something they like to lower they should have chosen Optane.
- ptrwis 6y agoLet's say they maxed out their former storage capacity of 24x3.8[TB]. For these 235millions of websites as they claim it's around 400kB per website.