13 ms·
O2 outage due to expired Ericsson certificate
- cameldrv 8y agoWhy do certificates expire? How is it acceptable to have a piece of data somewhere that contains a timebomb that must be periodically defused? I do not see how this helps security, and it is particularly ridiculous that normal behavior is that one day the system works normally and is deemed secure, and the next day it is so insecure and dangerous that communication simply fails.
- jstanley 8y agoI agree with you, but I suspect the intention is to prevent another type of timebomb. One where one day the system works and everything is considered secure, and then 25 years later nobody has given it a second thought.
- derefr 8y agoYes. Even then, a few years is too long, because you forget about the need for renewal. I would say that the "proper" use of certificate expiry is the way LetsEncrypt and other ACME providers do it: it's set so low that you need an automated renewal process in order to make the certificate at-all useful.
- jstanley 8y agoBut if it's automated, aren't we back to "forget about it for 25 years"?
- deleted 8y ago[deleted]
- zimbatm 8y agoBut then you need to make sure that the automated cert renewal system is still working...
- derefr 8y agoBut when automated cert-renewal breaks, it immediately breaks, and you start getting "cert renewal failed" messages in your email prompting you to action—but at that point, you still have time remaining to fix it before the cert expires, since ACME impls tend not to renew the cert exactly one second before it expires, but rather more like a day or two before it would expire. Unlike cert expiry, where the first you hear about it is when your production system stops working.
- wcoenen 8y agoBut of course, the failure emails will go to an email address that was deactivated a couple years ago because an employee left/everybody fired in reorg/unused address cleanup/domain migration/technical error/... Just this week I had someone complain to me that he was no longer getting build failure emails, and it turned out IT had disappeared our old <username>@<olddomain> addresses. That reminds me about an IT ticket I need to write.
- kodablah 8y ago> Why do certificates expire? I always assumed one reason is so revocation lists don't become huge. Also, digital properties such as domains change hands.
- bluGill 8y agoThe problem is what is someone screws up and compromises the key? By having an expiration we ensure that in the worst case they get just a few years to do damage.
- noobermin 8y ago>just a few years of damage. Something in there is telling you that you're doing the worst of both worlds, by allowing such a huge gap between renewals that people merely forget leading to outages, and such a huge gap that if a key is to expire, someone else could use it for years before someone notice and actually revoke the old key. derefr above has the correct response.
- modernpacifist 8y agoSeriously? Any half decent administrator knows that for every SSL certificate you install, you set a date some time in the future where you need to change it out for a new one. To your comment of "one day the system works normally and is deemed secure, and the next day it is so insecure and dangerous", this is working as intended. The certificate is to establish trust and identity along with encrypting the data in transit. The identity described by the certificate is only valid up until the expiry date after which it ceases to be valid for that purpose. You now have an encrypted connection to something that can't prove its identity, which is certainly a lower level of "secure" than what it was before the certificate expired. Ignoring expiry dates would mean any keys that were compromised ever could be used for MITM attacks and no one would be the wiser.
- admax88q 8y agoExpiry dates are in years, if a key is compromised, then an adversary has _years_ to exploit a MITM. However we already mitigate this with revocation lists. But if we can revoke certificates why do we have expiration dates? Seems to me expiration dates are rent seeking behaviour by certificate vendors.
- jstanley 8y agoOne good reason is that if you buy a domain name that somebody else has used in the past, they don't have an infinite valid SSL certificate for your domain.
- AltF4me 8y agoWould it not be possible to expire the cert if the domain expires?
- toast0 8y agoNo. The certificate's expiration is fixed at time of issuance. You could set the expiration of the certificate to the expiration date of the domain, but the domain could be transferred, cancelled, or revoked before the expiration.
- jcranmer 8y agoCryptography is basically a computational treadmill: you want to make it cheap enough that it's not burdensome for the actual users, but that reversing the information without the key is computationally expensive. Processing power, especially for the highly parallelizable task of grinding through potential keys, follows an exponential curve; ergo, even the present exponential gap is not a long-term protection mechanism. The expiry date of a certificate was present in the original implementation of X.509 certificates; the infrastructure for certificate revocation lists was added later. Furthermore, a certificate revocation list includes every certificate ever revoked that has yet to expire. If you relied on certifcate revocation for expiring old certificates as well, that would be a nasty long file you'd have to download before you can start connecting to any website.
- hamandcheese 8y agoI think the much more important reason for expiring certificates is to minimize the damage in the case that your private keys are stolen, which is far more likely than them being brute forced.
- geofft 8y agoAttestation of identity should be seen as a service, not a product. Revocation exists. The fact that certificates are pieces of data is a compromise, not an ideal design. Most CAs already support OCSP, which is effectively one-day-long certificates: the client can contact the OCSP server for a signed response saying "yes, it's still unrevoked", or the server can include ("staple") such a response along with its certs. Some certs have the MustStaple extension, indicating that they should not be treated as valid unless a recent OCSP response is stapled to it. If we had the computational resources to just not have certificates at all and have every client check the CA's current belief that a public key belongs to a name at each use (and magically avoid the associated privacy problems), that would be ideal. Certificates are an approximation. One neat thing about 3-month certificates is that it's a meaningfully different human lifescale from a year (or multiple years): operators may not still be around in a year but will certainly be around in 3 months. Operators may just plan to manually renew in a few years, but generally decide they need to automate the process if it's every 3 months. (For extremely boring reasons, I manually update the certificate on my personal website every 3 months and it's a pain, and I think I am one of very few people who do manual updates to their Let's Encrypt certs.) So it forces people to think of certificates as running their end of a service with ongoing operational work, not a one-time transfer of data.
- NeedMoreTea 8y agoIgnoring letting it expire in the first place. The surprising part is it took over 24 hours to restore service. I currently still have 3G only, and that's struggling (apparently 4G will follow).
- cnorthwood 8y agoI suspect it was a root CA (they possibly had a private PKI) that expired - I saw a tweet suggesting that they had to manually apply a fix to every deployment of the software, which is consistent with having to change the trust chain.
- jwdunne 8y agoStill only 4G here too. Apparently we can expect it to return tomorrow morning. O2 were asked if we'll be compensated. They said they'll "apologise in an O2 way" but couldn't confirm what an "O2 way" is.
- ksec 8y agoApologise in an O2 way? Gosh are we back to BT era. See what we can do. O2.
- mikeash 8y agoIf O2 is like most phone companies, my reaction to this threat would be to beg them not to.
- matthewmacleod 8y agoI assume that means they’ll apologise by ramping up your monthly rate.
- setquk 8y agoProbably 10% off some ticketing scam they have running.
- umichguy 8y agoThey are a bunch of O2 thieves.
- Bokanovsky 8y agoThis was a major outage in the UK causing millions of people not to have data access on their phones. https://www.bbc.co.uk/news/business-46464730 https://www.bbc.co.uk/news/business-46464730
- walshemj 8y agoAnd some major major emergency ass pulling I was told by insiders.
- bonaldi 8y ago"Ass pulling"?
- Thev00d00 8y agoAn Ass Pull is a moment when the writers pull something out of thin air in a less-than-graceful narrative development, violating the Law of Conservation of Detail by dropping a Plot-critical detail in the middle, or near the end of their narrative without Foreshadowing or dropping a Chekhov's Gun earlier on.
- walshemj 8y agoChanged all the Cell tower id's en mass
- Crosseye_Jack 8y agoNot just data but also calls and text. Mother is on o2 and until about 6pm I was unable to call, was able to text her from about 4pm. Between 4 and 6 the calls were just failing with “called failed” and not even going to Voicemail, but texts was go though and I would even get delivery reports, just that calls wouldn’t connect.
- gambiting 8y agoYeah, O2 kept saying that calls and texts were not affected, but that was utter nonsense. I couldn't place a call most of the day, and texts would go out, but my phone would insist that sending failed. And then there was just no signal at all most of the afternoon.
- lifeisstillgood 8y agoOoh - I was wondering what the hell was going on. I assumed I had dropped the darn thing one too many times. Glad to see the HN grapevine works on wifi. It is a reminder of just how fragile this digital world still is - we are taking technology designed to survive nuclear war, and adding single points of failure. Let's look at mesh networking again.
- stillworks 8y agoIt wasn't just the phones or just data on phones. GWR ticket kiosks went offline. Bus stop digital timetables went offline. On the phone, SMS delievery went wonky i.e. same SMS being delivered more than once but on the sender's phone it showed up as not delivered !
- deleted 8y ago[deleted]
- RowanH 8y agoWould be interesting to know how many people who've managed footprints for a reasonable period of time (say 5-10years) who haven't had a cert expire on them. Wouldn't be surprised if it's single digit %ages. So many human & tech error factors lead to this occurring and they're all the same old things. Staffing changes, spam filters, ignored warnings, skipped emails...
- stdplaceholder 8y agoI know it's happened to every company where I've worked. It happens so rarely, though, that people don't have enough opportunity to learn from it. Even at Google they were on their Nth such outage for a large value of N before it became apparent that no certificate should ever expire at 23:59:59 on December 31, or otherwise outside of normal operating hours. Seriously 20 years of organizational knowledge required to get the company to understand that certs should expire at noon on a Wednesday to minimize time-to-repair in the inevitable event that one is allowed to lapse.
- jtl999 8y agoInstead of waiting last minute, you'd think a large company would have planning to renew certificates X amount of time before they expire. Alas I understand it's not that simple.
- stdplaceholder 8y agoYou'd certainly think so. If you have frontend probers that exercise your accessible endpoints (HTTP or whatever) then those probes should fail when the certificate expires in less than 30 days. I couldn't comment on whether an organization like Ericsson or O2 would be expected to have such probers.
- jtl999 8y agoI should get on that for my own infrastructure. Thanks for the reminder.
- 8y ago
- felideon 8y agoWe use Erlang for a high availability of nine nines, so long as we remember to renew our certs.
- deleted 8y ago[deleted]
- seibelj 8y agoAn interesting new project is Handshake which is attempting to use decentralization to remove centralized certificate authorities. Maybe it will help stop these and similar situations in the future. https://handshake.org/ https://handshake.org/
- aaaaaaaaaab 8y agoWhy is it better than GPG? (“Blockchain” is not a valid answer)
- nickdothutton 8y agoReminds me of the 1990 network-wide outage AT&T suffered, aka "the big one of nine-oh". Caused by a bad update which took a couple of weeks to eventually bork the entire network.
- mhandley 8y agoMaybe it would be a good idea for certificates to expire slowly and randomly over 24 or 48 hours. In other words, if the cert has an expiry date of 12:00 UTC, Dec 6th 2018, then start to randomly fail connections at that time with low probability. The probability increases progressively during the next 24 hours until 100% of connections fail at 12:00 UTC, Dec 7th 2018. It's not like the cert is 100% trustworthy one minute and 100% untrustworthy the next minute. Having the failure rate ramp up slowly would give advance warning before everything has gone completely pear-shaped. In the case of Ericsson, this might have allowed an emergency certificate update before all the O2 systems could no longer be automatically updated. Once your network is completely down, bringing it back up remotely is hard.
- deleted 8y ago[deleted]
- codingminds 8y agoCertificate monitoring would be a good idea.
- gred 8y agoI think this could also be a good idea for phasing out public APIs -- instead of just taking an API offline, start to fail requests early at low probability, ramping the probability up to 100% over the course of a month or so.
- hinkley 8y agoAdding delays to responses can also work too. Years ago, during the tail end of the Age of XML, I had a conversation with one of the admins over at W3C. To make your app work you're supposed to cache all schemas locally but an inordinate amount of people don't. Their apps will be slow because of that, but work anyway. So tons of traffic goes to w3.org. They can't remove those schemas because then nobody can use them the right way (download them 1x and keep the result). But how do you get people's attention? I pointed out to him that a human downloading the schema isn't going to notice if the request takes .3 seconds or 3 seconds, but someone loading that schema 100x in a loop is sure going to be incentivized to figure out where the extra 5 minutes of processing time came from. He allowed that I might be onto something there but I never heard back about whether he tried it. You want to kill off an API, assign it resources (hardware, bandwidth) that get progressively smaller, or add artificial waits that get progressively bigger. Depending on how horizontally scalable your app is (is it easier to add a sleep or to pull servers out of the cluster?).
- Zenst 8y agoIf only the mechanisms that check certificates could provide warnings of impending expiry - 1,3,7,30 days would be prudent. Though companies should be doing at least a yearly audit of certificates and calendering any that will need renewing. As I'm sure they do with domain names already.
- Jolter 8y agoPresumably this certificate was protecting some machine-to-machine connection. Then, no human would ever have seen those warnings.
- wil421 8y agoCertificates can be hard to manage across enterprises. I have a project coming across my desk next year specifically to manage expiring certs and track on going changes. The company has 20,000+ certs to manage for us and our customers.
- karambahh 8y agoI can't tell if you're sitting on a gold mine or about to get very, very depressed very, very quickly. I am, however, pretty sure there's no middle ground. Wish you all the best but I would never trade my job for yours, I value my sanity way more than that.
- codingminds 8y agoI'd find a HN post about this project and it's results very interesting
- alwaystyred 8y agoThere was an interesting post a few weeks back on Autotrader moving all of their online properties over to LetsEncrypt https://news.ycombinator.com/item?id=17949741 https://news.ycombinator.com/item?id=17949741
- mtkd 8y agoSame risk with domains and DNS - when people question why CSCGlobal or UltraDNS charge so much compared to cheaper alts - it's because they have your back when you miss something
- Redsquare 8y agoThe only way Ultra have your back is ripping the shirt off you in overage charges
- davidwparker 8y agoWhat is O2? It's not mentioned in the article but it's in the title.
- jacobwg 8y agoO2 is one of the major telecommunications service providers in the UK: https://en.wikipedia.org/wiki/O2_(UK) https://en.wikipedia.org/wiki/O2_(UK)
- deleted 8y ago[deleted]
- nothrabannosir 8y agoIt took some warming to, but I have come round to appreciate letsencrypt's short certificate lifetimes. Monthly renewal should be maximum for any system, but ideally you'd want to go weekly. Assuming your renewal is automated, I don't see any downside, only benefits. It properly internalises cert renewal as part of standard system operations, bringing it into your daily ops instead of having it as some scary gray undocumented box for the next guy to tick. Much like writing tests, it initially hurt my ego a bit, but I've come to like and proselytise it.
- tqkxzugoaupvwqr 8y agoYour proposal may lead to an unintended DDoS of the CA.
- jjeaff 8y agoLet's Encrypt recommends a daily cron job for cert renewal. The certbot only actually pings the servers for a renewed cert if the current cert is within 30 days of expiration.
- geofft 8y agoMost CAs already operate an OCSP responder which supports querying daily (if not more frequently) to make sure a certificate still hasn't expired; it checks the database and responds with a signed and timestamped acknowledgement. (Those that don't need to support frequent queries of a static certificate revocation list file.) So this is already a level of requests that CAs have to be built for. And even if it didn't, scaling certificates from once a year to once a week requires merely 50x capacity - a finite number, and something that's very easy to plan out. If CAs in the '90s and '00s were able to handle a certain once-a-year request load, surely CAs today can handle 50x that.
- entity345 8y agoThere is no need for enforced expiry of certificates, especially here where there are used internally on a controlled network. If there is a requirement or good practice to periodically replace key-pairs then it should be a network management operation. Equipment should never stop working because it decided that a key-pair had expired. X509 certificates are not technically optimal in many situations and, let's face it, the main driver for expiry dates is that certificate vendors want to sell more certificates...
- anilakar 8y agoHardware manufacturers: How do you manage certificates that the customer will not touch or even see? Do you deliver new CAs alongside firmware updates? Do your software lifecycle requirements take certificates and cryptography in general into account?
- dikei 8y agoMultiple providers all over the world are affected as well. For example, all Vietnamese providers has their data service disrupted yesterday.
- OnTheHoof 8y agoQualys is used by O2 which has the ability to manage certificates - see their website. Most large IT systems use centralised monitoring using feeds from several sources. Did someone forget to monitor this data or was the feed from this data either ignored or out of date?
- deleted 8y ago[deleted]
- Maven911 8y agoIf an ssl cert is expired on a website, you don't have access to the secured website. In this case though, how does it affect Ericssons telcom equipment (core routers and switchs for mobileand voice) ?