13 ms·
Preparing to Issue 200M Certificates in 24 Hours
- Proven 6y agoAgain that nonsense about AMD CPUs and PCI lanes. It's completely unclear why the old h/w wasn't enough (and whether the new servers are enough). 200 million certs / 86,400 means 2,315 certs per second. Mid-range server with SSDs can do many 000's of TPS. Even if they need 5 tx/cert, that shouldn't be a problem with 3-4 year old servers.
- ivoras 6y agoOh single points of failure, where art thou... I hope everyone realises that Let's Encrypt is by now an essential part of the Internet, somewhat like DNS is, just massively centralised.
- RL_Quine 6y agoThey aren't the only no-cost distributor, their downtime doesn't really matter so long as it's less than a month long. What exactly is the issue with centralization here?
- acct776 6y ago> They aren't the only no-cost distributor I couldn't name a second, which means they're probably not getting a huge %.
- cholmon 6y agoHow would Netlify feel about not being able to issue or renew any certs for any sites for a month? Plenty of platforms rely on LE exclusively for one-click/automatic HTTPS for their customers sites.
- Denvercoder9 6y agoLet's Encrypt isn't the only provider supporting ACME either. Sectigo (under the ZeroSSL name) is a notable alternative.
- mvolfik 6y agoThe thing is that we're not talking downtime here, but rather something in the CA compromised, which would mean that pretty much ANY website could be impersonated, as the attacker could issue a Let's Encrypt CA valid certificate for it. That is mitigated by invalidating this CA, but that also invalidates all legit certificates previously issued by them. So they need to reissue them all
- warkdarrior 6y agoLet's Encrypt publishes Certificate Transparency logs: https://letsencrypt.org/docs/ct-logs/ https://letsencrypt.org/docs/ct-logs/ You can both block certs that do not appear in the logs, and decide which certs not to trust ("everything after Friday the 13th at midnight is not trusted"), once you know the date/time of the intrusion.
- jaywalk 6y agoIf Let's Encrypt went away tomorrow, I'd Google "free SSL certificate" and find somebody else. Worst case, I'd either go back to the old way and buy a cheap certificate or go without SSL until a new free option comes along.
- notafraudster 6y agoHonest question (I know just enough about SSL to be dangerous): Does certificate pinning throw a wrench in this plan?
- rntksi 6y agoIf proper revoking procedures are being followed, no. You can start reading more info here: https://developer.mozilla.org/en-US/docs/Web/Security/Certificate_Transparency https://developer.mozilla.org/en-US/docs/Web/Security/Certif... (HPKP is considered obsolete, if by cert pinning you meant that. If you're confusing HSTS with HPKP, just know that HSTS makes it much harder to mistakenly access the site via http, while HPKP [now deprecated] is the practice of ensuring some hashes is found in the cert that the server sends to mitigate some attacks)
- NovemberWhiskey 6y agoAt this point, I don't think certificate pinning in the general internet/web environment is a thing.
- LinuxBender 6y agoAgreed. I MITM all my own traffic and I only have to exclude a handful of domains or A records on domains to not MITM.
- leesalminen 6y agoHonestly curious, why do you MITM all your own traffic? For detailed logging?
- toomuchtodo 6y agoA lesson in recognizing, supporting, and defending public goods.
- marcosdumay 6y agoThe TLS CA system is one of those places where a single failure point is absolutely better than distributed possibilities. That is the case because any failure anywhere compromises the entire system. Of course, the other failure points didn't completely go away yet. But I do expect their number to reduce a lot in the future.
- nodesocket 6y agoSounds like they have read-replicas, why can’t they fail over to a read replica as the new master?
- gregwebs 6y agoThis is why they created the ACME protocol. There are actually other providers of this protocol now. So there is a possible future that is much more like DNS.
- coding123 6y agoBut it's a single point of failure that has about 60 days to come back online (if all actors are working fairly normally) from a full crash.
- gerdesj 6y agoQuite. I'd hate to test it but 60 days is long enough to deploy a new CA and push the cert out to trust stores. In an emergency. It would take quite some coordination but I suspect the current pandemic might provide some models and examples as to the way to do things at scale with JiT decision making from wonks who normally drag feet by default.
- M2Ys4U 6y agoAgreed. I'd love to see, say, another charitable CA set up in a different jurisdiction[0] that can hold a similar reputation to the ISRG. [0] Probably somewhere in Europe, whether that's in the EU or elsewhere like Switzerland or the UK. What matters most is that it's somewhere with strong commitment to the Rule of Law and not beholden to the US.
- sandGorgon 6y ago>There is no viable hardware RAID for NVME, so we’ve switched to ZFS to provide the data protection we need. This is Linux right ? would this be the largest deployment of ZFS-on-Linux then ?
- RL_Quine 6y agoI doubt it? 150TB of NVMe storage is big, but I've walked past racks with many orders of magnitude more in it. (edit: units)
- dlkmp 6y agoIt's TB though, per unit.
- dragontamer 6y ago> 150GB of NVMe storage is big Your age is showing :-) Every few years, I gotta get used to the 100s MBs is big!! -> 100s GBs is big!! -> 100s TBs is big!! Seems like we're entering the age of PBs, and then we stopped caring about capacity and more about the speed of our TB+ sized archives.
- sandGorgon 6y agofor linux ZFS ? im specifically asking about zfs-on-linux
- mnw21cam 6y agoNot by a long shot. I just assembled two servers with 168 12TB drives each, giving a bit over 1.5PB available space on each server. And I'm pretty confident that this is also not the largest ZFS-on-Linux deployment either.
- tpxl 6y agoDamn, what is the use case for that?
- 6y ago
- zanecodes 6y agoVery probably I'm missing something, and I love Let's Encrypt and the service that they provide, but... the point of Let's Encrypt is to bring SSL/TLS to more websites, right (and not necessarily to provide identity verification, since the automated renewal process doesn't really require any proof of identity for the entity requesting the certificate)? Why couldn't that have been accomplished using self-signed certificates, and having browser vendors remove their big scary warning pages for sites using self-signed certificates for SSL/TLS? Do certificates from Let's Encrypt provide any security benefits over a self-signed certificate?
- gsich 6y ago>Do certificates from Let's Encrypt provide any security benefits over a self-signed certificate? Depends. The encryption is the same and only dependent on your client/server. Could have been solved by DNS ... maybe. Self signed certs don't validate that you at least own the domain.
- zanecodes 6y agoAh yes, that makes sense. Let's Encrypt requires proof of domain ownership, which at least ensures that the entity you're connecting to is the entity that owns the domain. Encryption without authentication wouldn't be very helpful, since a man-in-the-middle could just present their own self-signed certificate during the handshake...
- tialaramex 6y agoYou'd be protected from a passive attack and thus you could always (with enough effort) detect an attack. Someone who is snooping (e.g. fibre taps) is potentially undetectable (yes in theory there are quantum physics tricks you could do to detect this, but nobody much is really doing that) whereas an active attack is always potentially detectable. So it's not nothing, but it isn't very much without the Certificate Authority role.
- VoidWhisperer 6y agoThe issue with browsers just allowing self-signed certs is that you cant verify their authenticity - ie were they correctly issued for the domain, or is it someone acting as the website using an invalidly issued cert. Having certs come from a recognized certificate authority helps with this because it provides a point for the certificate to be verified for authenticity
- TorKlingberg 6y agoThis seems like really fun project. For some reason it makes me happy that someone has good reason to run their own servers and networking, rather than rent everything from cloud providers.
- jtchang 6y agoAlways nice to see some private companies stepping up. Specifically Cisco, Luna, Thales, and Fortinet. I'm sure there are a bunch others that donate their resources to Lets Encrypt.
- tialaramex 6y ago> Normally ACME clients renew their certificates when one third of their lifetime is remaining, and don’t contact our servers otherwise. At least newer versions of Certbot, and I believe some other ACME clients, will also try to discern if the certificate is Revoked when considering it. So if you have a daily cron job running Certbot, and your certificate with 60 days left on it has been revoked since yesterday, Certbot ought to notice that and attempt to replace it as if it had expired. If you are doing OCSP stapling, and if your stapling implementation is good (sadly last I looked neither Apache nor nginx were) this ought to be enough to make a mass revocation event survivable for you. Your server will notice the latest OCSP answers now say it's revoked and continue to hand out the last GOOD answer it knew, some time later before that OCSP answer expires your Certbot should replace the certificate with a good one. Seamless. The new ACME feature is welcome, not least because there are a tremendous number of those bad Apache servers out there, but (unless I misunderstand) I think it's already possible to survive this sort of catastrophe without service interruption.
- hannob 6y agoFWIW Apache has a new stapling implementation that is not suffering from all the major problems the old one did. Can be activated with "MDStapling on".
- tialaramex 6y agoThanks, that's good to know. I also read their documentation explaining why they couldn't (or at least didn't) fix the old one. I will try to publicise this rather than simply saying the Apache httpd OCSP stapling is garbage in future. It explicitly mentions two big flaws with the old one, but not the one most relevant here - out of the box Apache's old OCSP stapling would merrily staple BAD answers simply because they're newer, which makes no sense. I assume that's corrected, but if you know this'd be a good place to say.
- Apofis 6y agoShouldn't that be the default config option then?
- 6y ago
- cbhl 6y agoIf there's one thing that always surprises me about the internet, it's that vertical scaling (bigger/faster machines, as opposed to horizontal scaling) can take a well-written service to "Internet Scale".
- hinkley 6y agoAm I the only one having flashbacks to Rainbow's End (Vernor Vinge)?
- tialaramex 6y agoThe events in Rainbows End depend upon revocation being effective, in particular revocation of a root CA, and very rapidly. For one thing, "revoking" a root CA isn't a thing, the root's signature is mostly a convenience (it's self-signed) and perhaps in another world roots would exist as distinct documents not as X509 certificates signed by themselves. So there isn't anybody to "revoke" it like other certificates. You can distrust them though. In the real world lots of systems would never become aware of the revocation/ distrust at all, and there are gatekeepers for many other systems that could become aware (e.g your copy of Chrome or Firefox can learn that a root CA is no longer trusted but it would not do so without humans at Google or Mozilla deciding this was a necessary course of action). It's necessary to the plot of Rainbows End that this happens unrealistically quickly, Rabbit must be disabled or it will certainly interfere with subsequent events, and it seems Vinge (unlike me) isn't sure exactly what Rabbit is, so this vague technical intervention seems like an effective way to stall Rabbit without thinking too hard about that question.
- hinkley 6y agoYou don't revoke the root cert, no. You revoke a cert that signed a lot of certs, which is often one degree of separation from the root cert. I don't recall if he stated it wrong and I glossed over it because I knew what he meant, or the character was dumbing it down for the rest of them. Revoking a signing cert, breaking the cert chain, would indeed make a mess for everyone using those certs.
- tialaramex 6y agoOK, consider revoking, since we're talking about them, Let's Encrypt's R3 issuer. In a sense only they (ISRG, the organisation behind Let's Encrypt) can do that, since the revocation would need to come in the form of an updated CRL (Certificate Revocation List) from ISRG's root. So you'll be making a late night phone call to key people from ISRG to demand (persuade?) an immediate revocation. How often do you suppose most systems examine that CRL? If you guessed anything other than "never" you're wrong. Almost nothing you use will ever notice. But that's not enough anyway - R3 is also trusted via a cross signature from Identrust's DST Root CA X3. So their organisation also needs to be woken and persuaded to perform an unprecedented middle-of-the-night revocation of some certificate they've seen no evidence is a problem. Now, you've disturbed all these nice people, with your very urgent problem of... you want to stop something you can't discuss happening that involves some confidential things and a bunch of other confidential things and they must never speak of it to anybody - in the morning they're going to be doorstopped by a thousand tech journalists wondering why they broke everything - or they could just hang up. But we'll suppose you did that, as I said it doesn't have any effect. Oops. Rabbit will notice, no doubt, but it isn't disabled. OK, so what can you do that will actually have some impact? Well, you can get Microsoft, Google and Mozilla to use their out-of-band "kill switch" functionality. Each works differently. Microsoft's has the advantage that it's entirely a closed door process, if you can get the ear of the right person you can make any change you want. Unfortunately the latency is one calendar month. Rabbit is causing you a problem today? By April we can fix that. Oh, you need something sooner? Too bad. Google are more promising, they can tell most Chrome installs to distrust R3 without any independent confirmation and the updates will typically take only a day or so to have effect. Rabbit will be out of your hair before tomorrow's evening news can run the story about your resignation or arrest. They will probably tell everybody why though... Mozilla likewise can react quickly, perhaps in just a single day. However unfortunately Mozilla deliberately makes these decisions in public. You're going to need to tell a bunch of random people who don't even work for Mozilla about your urgent need to shut down this issuing CA. They're going to have questions. You probably don't even want to read the questions, never mind answer them. Oh dear.
- amaccuish 6y agoI hope all that traffic passing Cisco switches is encrypted...
- pgporada 6y agoAll of the boulder grpc communications use mutual TLS authentication.
- Dylan16807 6y ago> the really interesting thing about these machines is that the EPYC CPUs provide 128 PCIe4 lanes each Not really. In a single socket setup, an EPYC gives you 128 lanes. In a dual socket setup, 64 lanes from each CPU are repurposed to connect them together instead of doing PCIe. So just like single socket, you end up with 128 lanes total.
- virgulino 6y ago128 or 160 total, configurable. https://www.servethehome.com/dell-and-amd-showcase-future-of-servers-160-pcie-lane-design/ https://www.servethehome.com/dell-and-amd-showcase-future-of...
- 120bits 6y agoOn a side note. It always nice to see them include stuff like, Internal networking and Hardware specs of server. It shows you how much scalable they are or how they deal with large amount of data. I always enjoy reading them.
- jiggawatts 6y agoSomething I would like the community do is force vendors such as Azure and AWS to support the ACME protocol to provide the option of free certificates to users. HTTPS isn't an add-on any more, and it shouldn't come with a yearly tax. Unfortunately, a yearly tax has a margin, so the vendors have been dragging their feet. DigiCert and GoDaddy have bribed the Azure team with kickbacks and wholesale discounts, so Azure is now refusing to implement anything else. Why would they? They get a margin on every issued certificate! So please: If you have a presence in AWS or Azure, call your sales representative and pressure them. Submit requests through the feedback portal, and vote up existing requests. Raise this regularly, or it will never happen by itself. The incentives just aren't there! Pressure must be applied by a large fraction of the customers. You are that customer. Apply pressure.
- est31 6y agoIf you wanted to save money, you wouldn't use Azure or AWS anyways.
- wkcmp 6y agoWhat would you use then?
- HotVector 6y agoLinode, DigitalOcean, or a $200 used server from EBay. Both Linode and DO also have really good Kubernetes services. AWS and alike force users to stick to their ecosystem of products, and come with all sorts of weird pricing models. And if you really need some sort of serverless features or BaaS, you can just self-host that on a VPS.
- ehwhyreally 6y agoBinary lane for Australia is fantastic.
- ggm 6y agoEcho this. great pricepoint, I'm on a BSD box and their console/backend has been flawless. And, its full dualstack v4/v6
- zertrin 6y agoThey discussed a lot about their own internal bottlenecks. I'm wondering about the external bottlenecks they might encounter, such as the requirement for sending all certificate requests to CT logs prior to issuing a certificate. Could it be that the amount of data and requests per seconds sent to external entities to fulfil CT obligations is deemed negligible or already manageable?
- M2Ys4U 6y agoThey also run their own CT log called Oak: https://letsencrypt.org/docs/ct-logs/ https://letsencrypt.org/docs/ct-logs/