14 ms·
Let's Encrypt's performance is currently degraded due to a DDoS attack
- timvisee 6y agoI wonder what their motivation is?
- jamescun 6y agoSome people just want to watch the world burn.
- unixhero 6y agoThis
- mysterydip 6y ago"no one is buying our expensive SSL certs. If we show businesses how unreliable a free service is, CTOs will make their admins buy from us." or "domain xyz's certificate is expiring. If we pay for a ddos, their site won't be able to renew and (customers wont go to the site due to expired cert/API people use wont work/we can take advantage of a compromised cert longer)" Just some possible but implausible scenarios.
- foepys 6y ago> domain xyz's certificate is expiring That's why it's so important not to wait until the end of the 90 day expiration period but to renew it every other week or so.
- hmaxwell 6y agois that Let's Encrypt's default behavior?
- Symbiote 6y agoThe default is to renew when less than 30 days remain, and to check that every day or every week.
- tutfbhuf 6y agoSeems like a sane default to me.
- philihp 6y agoVery much so. In my experience with nightly jobs in a corporate setting, the more often something happens, the more likely you are to catch an upstream dependency that breaks it. The sooner you catch that breakage, the easier it is to get the resources (either from that team, or from your own team) to fix it. It’s a matter of “Oh we changed that API 2 months ago, everything is fine for us, all of our people have moved on to other tasks” versus “Oh our change broke you? We can revert it until we have a workaround”. 2 months, in most orgs, is enough time to figure something out before your entire business goes offline.
- hftplot 6y agoIt seems unworkable for the majority of smaller sites who are increasingly forced to use letsencrypt. Unless you want that automatic update tool on your server, which I find a bit sketchy.
- abhinav22 6y agoWhy are they forced to use letsencrypt? Cloud flare gives me free SSL and I imagine AWS or GCE would also
- yjftsjthsd-h 6y ago> Unless you want that automatic update tool on your server, which I find a bit sketchy. Where else would you put it? You could put an ACME client somewhere else but it still needs to connect to place the updated certs.
- tgragnato 6y ago> CTOs will make their admins buy from us Or, those admins can switch to zerossl.com until the DDoS ends (you basically just need to change the domain in certbot). Yielding the DDoS.. wasted money.
- imhoguy 6y agoAren't OCSP servers affected too? That would cause issues for page visitors too.
- josefx 6y agoGiven the complete absence of information could it just be an accident? I think Wikipedia recently had performance issues where it turned out that a popular app was just pulling an image in the background and the app developers fixed it once they were notified.
- scoot 6y agoA distraction from the real intrusion?
- christophilus 6y agoMy thoughts exactly. Hacking letsencrypt would be a massive deal.
- tomxor 6y agoNot sure. Unless this is sustained for a long time it shouldn't affect autorenewals which are done well in advance of expiry. So it _shouldn't_ affect cert expiry unless people are still manually renewing and leaving it to last minute. [edit] Unless the attackers identified a bug in certbot (commonly used autorenewal scripts), e.g what happens when LE is unavailable when autorenew is triggered - you'd hope it would retry periodically until LE is restored, but perhaps not. If not you could time the DDoS just right to ensure a specific cert does not get renewed even after the DDoS stops, then maybe a couple weeks later it would expire... But that's relying on such a bug existing and the site owners not noticing it (LE will also email the registered email address eventually regardless of autorenewal scripts), so maybe this is too much of a stretch.
- tgsovlerkhgsel 6y agoPossibly testing or demonstrating a botnet. "For bragging rights" is a thing, as is advertising ("my botnet took down critical infrastructure, wanna buy my DDoS service?").
- iso1210 6y agoPointing out the issues of a single point of failure for the internet?
- Chilinot 6y agoIs it really a single point of failure though? Certificates are renewed well in advance, and there are several free alternatives with ACME support to LetsEncrypt today. Switching to a new provider in case LetsEncrypt goes down is as simple as updating your scripts.
- deleted 6y ago[deleted]
- gsich 6y agoBuypass, ZeroSSL also provide free certificates with ACME.
- beermonster 6y agoThe keys to the kingdom are ever more being placed in the hands of relatively few internet custodians. Figuratively here of course, since the private keys are generated locally and never transmitted to LE.
- francislavoie 6y agoCaddy mitigates this by falling back to ZeroSSL if it couldn't issue a cert from LE: https://caddyserver.com/docs/automatic-https#errors https://caddyserver.com/docs/automatic-https#errors
- chrisandchris 6y agoFun? Just last week our shared hosting provider was attacked and the attacker tried to brute-force it‘s way into a management API. I cannot image another reason as „fun“ and „just because we can“ because there‘s nothing to get [besides money after encrypting all data]. So I think the attacker just attacks LE because it‘s in the internet and he can.
- stjohnswarts 6y agoIt as a practice run before they try it on a bigger entity
- breakingcups 6y agoThey could be aiming for credentials to use in credential stuffing attack, a place to put malware, a place to distribute malware, servers to add to their botnet, a proxy to use for shady stuff, the list goes on. I see plenty of reasons to attack a hosting provider and its infrastructure? Or am I missing something?
- mirekrusin 6y agoit's better to assume that people will simply do everything that is possible, you can't open umbrella in your butt so that can be assumed not to be done, but everything possible should be assumed done/to be done sonner or later; motivation of "because I can" is simply enough.
- bombcar 6y agoCould be a "test fire" of a paid service - showing it works and will do what the customer wants.
- beermonster 6y agoOften a DDoS is used as a smokescreen/cover for an actual compromise. I guess the hope is that it gets unnoticed in all the noise and whilst all hands are busy at the pumps. Hope not! I see in their status page that OCSP endpoints are also impacted. There could be any number of motivations including interfering with someone's ability to check if a certificate has been revoked.
- mjthompson 6y agoThis had me scratching my head earlier today when I was debugging why renewal was taking so long. I've taken Let's Encrypt's reliability for granted. Didn't even cross my mind that it might be a service issue.
- FreeCodeFreak 6y agoWhy would anyone want to attack lets encrypt?
- oconnor663 6y agoAbout how expensive is it to rent a botnet and pull off an attack like this?
- polycaster 6y agoAsking for a friend?
- malikNF 6y agoWe used to run a game server for a small community of around 400-500 people and DDos attacks were something we had to face almost every week, whenever someone got upset with the admin team, the go to solution was was to DDos, you get scammed by another player? DDos. Got banned for saying racist things ingame? DDos. You figured out a new way to cheat in game and the admins fixed it? DDos. We were kids back then and those were kids that were attacking us with just a 5-10usd budget. Yes they were relatively small (ranging from 10-60Gbps) attacks compared to the Tbps attacks that are happening to some companies, but good god it was so annoying when all it took was just 5 usd from some idiot to take down your server. We moved to gcp got null routed (or reduced network bandwitch to the node under attack) every-time there was an attack. Bought azure's 3000usd a month anti DDos protection, was worthless for a tcp/udp service. Tried to have a network load balancer in the cloud that auto-scaled, still some players got effected when an attack came in. Finally we moved over to OVH and placed a few really powerful servers in-front of the game server and applied some ipfilter rules to reduce common attacks. That ended up being the cheapest option out of all the options. When you have a very small community its not like you have the biggest budget to work with. But it was really fun and taught all of us a lot. Looking back its kinna sad we had to end things. But it was a lot fun. DDos attacks are one of those things that really makes me worried about the future of the internet. The only way to win it is to throw money at it and cross your fingers that the attacker will run out of resources before you do. Definitely companies like cloudflare does an incredibly good job of stopping some insanely big attacks when it comes to http/https (I recently saw they were supporting udp and tcp based services now, never tried it). But one thing that's weird is having to rely on some 3rd party company. Yes cloudflare so far has been a company I can trust, but, I once loved and trusted a company that said "Don't be evil". If you are a developer for some IOT device manufacturer please do your best to makesure someone wont turn your light bulb in to a part of a botnet. When you guys fuck-up the rest of us have to suffer.
- EGreg 6y agoAnd this is why we need MaidSAFE instead of the Web. They don’t have DDOS attacks, instead you make money every time someone accesses a chunk of a resource, and the kademlia tree hides the hosts’ IP after one hop so the network and hosts can’t be taken down easily. Very different from Tor. https://maidsafe.net https://maidsafe.net is the best project to come out of the “Web3” space. If you heard of freenet, this is like freenet 2.0 PS: Why the massive silent downvotes? This platform actually solves the problem and many others HN constantly correctly complain about. But when posted, you prefer to ignore it. (Disclaimer: I am not affiliated with them in any way. In some ways they are a competitor to Qbix and Intercoin but I give credit where it is due.)
- Tenpenny99 6y agoHeroes really, https and certificate centralization should end as soon as possible. Maybe along with DNS.
- superkuh 6y agoTLS cert authorities shouldn't end, but more importantly, HTTP shouldn't end. HTTP+HTTPS together are great. HTTPS only, as being pushed in modern times is quite bad. LetsEncrypt is great and I am really glad someone stepped up to create a mostly not evil non-profit cert authority. But everyone using LE is very bad for the health of the internet. It provides nearly a single point of failure for government/political interefence, technical failure, and failure due to corruption from money and scale internally.
- sakisv 6y ago> HTTPS only, as being pushed in modern times is quite bad. I'd be interested to hear more about this, care to elaborate?
- superkuh 6y agoWeb devs have been cargo-culting really hard lately and adopting practices like completely disabling HTTP and only doing 304 redirects to HTTPS on the HTTP interface. They say they need to protect their users from MITM and downgrade attacks if they say anything at all, but realistically this isn't even in the threat model for 99% of sites. So now we have sites abandoning HTTP entirely and only having HTTPS. So this encourages browsers like Firefox to start enabling things like HTTPS only in their browsers by default. It encourages putting up scaremongering warnings of danger on HTTP sites like HTTPS self-signed certs get (which killed off self signed sites). So now browsers are beginning to refuse to show HTTP and the web admins are putting up servers that refuse to serve HTTP. That means in the near future unless you can get a cert authority approval (forever) you'll be unable to host a visitable website (ie, get a TLS cert from an authority) and unable to visit most websites that don't play the cert game unless you modify your browser. Human people cannot be cert authorities. Only corporations can. These two trends towards HTTPS only, on client and server, lead inevitably towards a situation where everywhere is in a handful of cert authority chains and things become easily controlled, or accidentally broken, due to that centralization.
- manishsharan 6y agoHere is something to think about. If you get ddos'd in middle of trial run by an enterprise customer and its the end of your startup. AZ , AWS , OVH almost all hosting providers will all start dropping your connections. And DDOS protection services are expensive. Pay as you go models are awful for this as when you do get ddos'd , you bill could be quite high.
- cpncrunch 6y agoNo, OVH has DDoS protection built into their service, and it's free, and your connections will not get dropped. I moved to OVH after getting a few DDoS attacks, and since then there have been no problems. I've had a few emails from OVH notifying me about attacks in progress and that they are automatically mitigating it. When it happens, it has zero effect on our service.
- manishsharan 6y agoThnak you . I did not know this. I would love to hear more about it -- is this protecion available for their VPS or Public cloud ? I already have stringent firewall rules for all VMs in my ansible roles (I can port over my stuff from one cloud to next as my deployment is based on asible). what else do I need to do to protect my servers?
- cpncrunch 6y agoAll their services come with DDoS protection. https://www.ovh.com/ca/en/anti-ddos/faq.xml?lsdDoc=faq.xml https://www.ovh.com/ca/en/anti-ddos/faq.xml?lsdDoc=faq.xml
- malikNF 6y agoOVH actually does a really good job when it comes to DDoS protection. They can take some really big attacks and slow it down just enough so you firewall rules can take care of the rest. From all the hosting providers I have used, they are the only ones who don't null route you as soon as you get a attacked, and considering the cost of their service OVH is a real life saver when you really need the help. ---- Just realized this sounds like an advert for OVH, lol I have no affiliation with them whatsoever, just a really happy old customer.
- benlivengood 6y agoI wonder how well common acme tools implement exponential backoff on retries. With a ton of clients you can inadvertently make a DDoS longer/worse. Let's Encrypt asks for max 1 request per day per certificate: https://letsencrypt.org/docs/integration-guide/ https://letsencrypt.org/docs/integration-guide/
- francislavoie 6y agoCaddy does a lot of it: https://caddyserver.com/docs/automatic-https#errors https://caddyserver.com/docs/automatic-https#errors, and Caddy also falls back to ZeroSSL if it couldn't issue with Let's Encrypt. Caddy is without a doubt the most robust ACME client implementation to date.
- bobberkarl 6y agoI completely dropped nginx for caddy some months ago. The only thing missing is an ingress implementation of caddy
- 0xbadcafebee 6y agoThis is a very morbid thought, but I wonder if the people who run LE ever travel via the same means. If somebody took them out all at once, would the web's security essentially crumble? This is the danger of centralized services, but moreso the crap design of web PKI. All "usable" HTTPS depends on certs, right? And "usable" certs require a domain, right? And that cert for that domain needs to have been generated by a CA, right? But it's tied to a domain, and IP space. You have to prove to a CA that you both control a domain record and some IP space it points to. Nobody has designed anything to straightforwardly prove that in an unhackable way. We have shitty hacks, like "serve this unique file on this web server that this domain record is pointing to", or "answer an e-mail on one of 20 addresses at this domain", etc. But none of those address what we actually want to do, which is just to prove that we own/control a domain record. That's the only meaningful thing in having a cert: proving that you actually own the domain record this cert is assigned to. And we have no actual way to do this. Literally the only way to prove definitively that you own a domain is to talk to the registrar, and the only way to prove that you control a domain record is to talk to the nameserver that the registrar is pointing to. The former we don't handle at all, and the latter is highly susceptible to various attacks. You could remove the reliance on CAs entirely with a different model. You tie a private key to domain ownership, and a private key to a domain record. Then you only have to trust registrars' keys/certs, and you can walk backward along a cryptographically-signed web of trust. Your browser trusts the registrar's key X. The registrar signs your domain key Y. The domain key Y signs a domain record key Z. Your web server generates a cert using domain key Z. For a client to verify the web server cert, they verify it was created by key Z, and verify that key Z was signed by key Y, and that key Y was signed by key X. Then any webserver can generate its own cert for any domain record, we don't need CAs to generate certs, and we have a solid web of trust that goes back to the actual owner of the domain, but also allows split trust via the domain owner assigning keys to domain records.
- yebyen 6y ago> This is a very morbid thought, but This is such a well-understood problem in fact that it has a name and Wikipedia entry, called "bus factor". According to: > The "bus factor" is the minimum number of team members that have to suddenly disappear from a project before the project stalls due to lack of knowledgeable or competent personnel As for proving that you own a domain, I think the DNS-01 challenge that is used to grant Star-certificates does a pretty good approximation, if you can create and update TXT records in the root zone, you have at least functionally "owned" a domain even if you don't legally own the domain.
- noncoml 6y agoWhy the duck would anyone DDoS Let’s Encrypt?
- deleted 6y ago[deleted]
- francislavoie 6y agoJust a reminder that users of Caddy (v2.3.0 or higher) are not at risk when LE gets hit like this, because it will fallback to having a certificate issued from ZeroSSL. Both issuers would need to be down for the whole last 30 days of the certificate's 90 day lifetime before Caddy would be stuck with expired certificates. https://caddyserver.com/docs/automatic-https#errors https://caddyserver.com/docs/automatic-https#errors https://github.com/caddyserver/caddy/releases/tag/v2.3.0 https://github.com/caddyserver/caddy/releases/tag/v2.3.0
- twirlock 6y agoBut at least my funny dog photo blog has perfect end to end secrecy.