15 ms·
Microsoft Teams outage due to expired certificate
- gjsman-1000 7y agoI'm actually curious: Is there a market for a SaaS which simply keeps track of certificates and when they expire? (Perhaps even with an auto-Deploy new certificate mechanism?)
- hashhar 7y agoDatadog has agents to monitor that.
- candiddevmike 7y agoI don't think so. The people who would need a service like this don't think they do, and everyone else uses let's encrypt.
- kevin_thibedeau 7y agoIt could be a freebie offered by cloud providers. Maybe MS should do this and eat its own dog food.
- mankyd 7y agohttps://sslmate.com/certspotter/ https://sslmate.com/certspotter/
- mytailorisrich 7y agoIt's s trivial to monitor certificates so I'm thinking that those who know they need it have it.
- zaphoyd 7y agoUptime Robot is a simple SaaS that monitors TLS cert expiration dates (in addition to other things).
- deleted 7y ago[deleted]
- jmstfv 7y agoI run a SaaS where certificate expiration monitoring is one of the features. But that's more of a nice-to-have feature rather than a primary thing that brings in customers.
- loopdoend 7y agoLove your site and ethos. Very cool.
- gerdesj 7y agoPerhaps but I call it doing my job. I run up a SSL cert check on icinga for each system as needed. It is quite trivial to roll your own script or find one that can be run from cron. It would probably need more work maintaining an account with a saas.
- mattmanser 7y agoWhy is it needed or your job? You work in an industry whose entire purpose is to automate such things.
- randyrand 7y agoYes, but will your company remember to fill your job when you leave?
- inkeddeveloper 7y agojob security?
- bcrosby95 7y agoJust use whatever you use to monitor your production systems in the first place. E.g. we use nagios.
- buryat 7y agoDatadog does that https://www.datadoghq.com/blog/monitor-tls-with-datadog/ https://www.datadoghq.com/blog/monitor-tls-with-datadog/
- pedroborges 7y agohttps://ohdear.app https://ohdear.app does that.
- jonas21 7y agoI think most monitoring services will let you know if your certificate is about to expire. For example, I use https://checklyhq.com https://checklyhq.com and it lets you configure how far in advance it will alert you.
- theklub 7y agoYes
- briffle 7y agoits a very, very simple update alert to add to Prometheus to monitor that, and alert if the cert is within so many days of expiring. You need the 'blackbox exporter' and a simple rule such as: alert: TlsCertExpiringSoon expr: (probe_ssl_earliest_cert_expiry - time()) < (86400 * 14) for: 10m labels: product: Name_of_Product severity: page annotations: description: the tls cert for the URL {{ $labels.instance }} expires in less than 14 days! summary: TLS cert for {{$labels.instance}} expiring
- zymhan 7y agostatuscake.com does it, but I just solved it with Datadog
- vivalibre 7y agoKeyChest.net does this and posts about it regularly to HN: https://news.ycombinator.com/from?site=keychest.net https://news.ycombinator.com/from?site=keychest.net
- krzat 7y agoTeams is great improvement over Skype for Business, which was great improvement over Lync, but it's still garbage. Interesting how the same company also made awesome VSCode.
- xeromal 7y agoVS Code isn't tailed to work in a corporate work environment with all the BS that comes with it.
- gambiting 7y agoIs it? I feel like every iteration is worse than the other. For Christ sake, Teams can't even scroll a few messages up without having to load more, and then it immediately forgets the latest messages. Like, it's an IM program that has trouble remembering more than 20 messages at once?? Also if you send messages quickly they appear in reverse order, again, it's an IM program that can't even do the "messaging" part of IM right.
- bouke 7y agoIndeed, a small list of the problems I have with Teams: https://www.reddit.com/r/Office365/comments/axzdct/my_issues_with_teams/ https://www.reddit.com/r/Office365/comments/axzdct/my_issues....
- lostgame 7y agoVSCode is an Electron app...I can't really compliment it. It's slow and cumbersome on my 2017 MBP.
- akerro 7y agoThe same company also did Windows Vista
- jordache 7y agoskype is terrible.. especially, the skype client for macOS.. It's barely usable.
- matt_morgan 7y agoThis has happened to me enough times to be embarrassing. It seems to happen to other people who you'd think have some sensible way to avoid it. Is there a reminder service out there that specializes in your long-term expiring things? I'm not sure what would be different about it than a regular calendar, but it seems like many of us need something that makes this easier.
- STRML 7y agoYeah, Cert Spotter does this (https://sslmate.com/certspotter/ https://sslmate.com/certspotter/). It's a huge help both to be able to monitor cert issuance for a domain and all of its subs, and to be reminded when a cert is expiring.
- snarfy 7y agoIt wouldn't be different than a regular calendar. It would be nice if they prompted you to 'add to calendar' when you are creating certs.
- inetknght 7y ago> It would be nice if they prompted you to 'add to calendar' when you are creating certs. That would actually be a really _awesome_ integration of ACME client software with CalDAV [0] and other ticketing software APIs like Jira: set up calendar events and/or tickets with increasing priority to verify that the certificate is updated. [0]: https://en.wikipedia.org/wiki/CalDAV https://en.wikipedia.org/wiki/CalDAV
- Someone1234 7y agoThere are multiple reminder services. At larger companies a lot of the issue isn't literally generating reminders, it is making sure they're sent to the correct people/departments and are actioned by anyone. For example you sometimes have reminders sent to ex-employees, or sent to a mailing list and everyone assuming everyone else is going to action it. Or the reminder gets ping-ponged between multiple managers via email with nobody either able or willing to deal with it. None of these are tech' issues, and they don't have technology solutions as a consequence. So whenever I see an embarrassing expired cert, I don't assume technical malfunction, I assume political malfunction.
- stygiansonic 7y agoA similar problem with Azure happened way back in 2013: https://www.computerworld.com/article/2495453/microsoft-s-azure-service-hit-by-expired-ssl-certificate.html https://www.computerworld.com/article/2495453/microsoft-s-az... More recently, it happened with Ericsson: https://www.theverge.com/2018/12/7/18130323/ericsson-software-certificate-o2-softbank-uk-japan-smartphone-4g-network-outage https://www.theverge.com/2018/12/7/18130323/ericsson-softwar... This article has some information about how Let's Encrypt enabled an "automated process that handles renewals": https://duo.com/decipher/proposal-to-make-https-certificate-expire-yearly-back-on-the-table https://duo.com/decipher/proposal-to-make-https-certificate-... I wonder if such a process should be made an industry standard? Does anyone know if there are any proposals for it?
- riffic 7y agoIt's called ACME protocol and it's RFC 8555. https://en.wikipedia.org/wiki/Automated_Certificate_Management_Environment https://en.wikipedia.org/wiki/Automated_Certificate_Manageme...
- cbhl 7y agoLet's Encrypt literally is an implementation of the industry standard; the standard is called Automatic Certificate Management Environment. https://tools.ietf.org/html/rfc8555 https://tools.ietf.org/html/rfc8555
- rationalfaith 7y agoNo biggie. they should look into automating it, or at least part of it.
- ce4 7y agoDoes anyone know what endpoint's certificate had expired? Would be interesting what CA they used for it and if it's a SAN certificate. Edit: here's the certificate log of the teams subdomain but I couldn't find the one that expired today in it https://crt.sh/?q=teams.microsoft.com https://crt.sh/?q=teams.microsoft.com
- chrishas35 7y agoThere were no end-user visible certificate errors during the outage, so I suspect it was an internal/backend cert.
- notimetorelax 7y agoMS has internal CA, it’s most probably some internal back-end talking to some other internal service.
- tialaramex 7y ago> if it's a SAN certificate All certificates in the Web PKI are obliged to use SAN (Subject Alternative Name). Although it is often mistaken for some sort of aliasing feature, SAN is the Internet's agreed alternative way to name things. X.509 is intended as part of the X.500 directory system, a global directory system which obviously was never actually built, and so its built-in name scheme doesn't resemble anything that actually exists. When Netscape invented SSL back in the 1990s they hijacked X.509's Common Name field to write DNS names in, but this field is just defined as arbitrary human readable text. "news.ycombinator.com" is text, but so is "News·YCombinator,coM " and only one of those is a DNS name. So, when the IETF standardised PKIX it designed a dedicated schema for the various types of Internet names to put into certificates, Subject Alternative Names SANs. Unlike the Common Name a SAN dnsName literally can't be anything but a DNS name using A-labels ("punycode"). So the opportunity for confusion is removed. Since it doesn't need to address universal human text it doesn't need to support weird encodings or anything else. When PKIX was standardised the existing abuse of Common Name was grandfathered in. PKIX says all certs should list one or more SANs but they may continue to have a DNS name in the Common Name too. The Baseline Requirements, years later, explicitly tell CAs to only use a Common Name which matches one of the SANs they've also baked into the certificate, if there are no SANs they're doing it wrong. Alas, as so often, commercial priorities beat security and so even last decade it was common to find people trying to get away without SANs. However the roll out of Certificate Transparency allowed us to have a clear view, without waiting for incident reports from affected users, of non-compliant certificates still being issued, so we could address it as it happened. A few years ago Firefox and Chrome (and Safari I think?) were able to remove their code for trying to process the human readable Common Name as if it might be important, and so if anybody were to issue such a certificate with no SANs today it wouldn't "work" in popular browsers.
- sproketboy 7y agoFAIL.
- akerro 7y agoIs anyone else redirected by this url to https://guce.advertising.com/collectIdentifiers?sessionId=3_cc-session_3982b38b-2de1-484a-98bd-4a0b8635308c https://guce.advertising.com/collectIdentifiers?sessionId=3_... ? https://gfycat.com/fortunateyawningcopperhead https://gfycat.com/fortunateyawningcopperhead My router is blocking this domain on DNS level in OpenWRT, techcrunch.com is redirecting me there, so I can't visit this page. Edit: I literally can not visit any techcrunch.com article, all of them redirect me to this doggy ads+tracking domain. It doesn't matter if I came from google, DDG, reddit or HN.
- kkirsche 7y agoI am not being redirected
- u42p37yYYRjvhD 7y agoyes, i'm being redirected too. engadget also does this, locking me out of both pages.
- soptikha2 7y agoI can confirm that I'm being redirected as well.
- abrodersen 7y agoThis was happening to me the other day!
- frio 7y agoIt's a Javascript based redirect; block all JS on the page and it'll stop happening. Super annoying.
- zelphirkalt 7y agoIf I block all JS, it still redirects and then uMatrix shows a page informing me that the ads page is blocked. The very first scripts I could allow are those ad scripts. Simply unusable page. "techcrunch" ...
- gwbas1c 7y agoAnother HN post mentioned that a lot of collaboration sites were down because many people are telecommuting due to the Wuhan virus. I honestly thought this was why Teams was down for me.
- cesarb 7y ago> Another HN post mentioned that a lot of collaboration sites were down because many people are telecommuting due to the Wuhan virus. For future reference, it was probably this one: https://news.ycombinator.com/item?id=22222121 https://news.ycombinator.com/item?id=22222121
- godelmachine 7y agoI am going to seize this opportunity and rant out my angst against Microsoft’s worst product till date. Has anyone even felt that Teams is a heavy app that consumes a lot of time to come alive? Even during calls, the quality is horrible that I don’t even want to describe the pain I go through. There’s strong distortion and voices will never be heard clearly.
- m4tthumphrey 7y agoYou are not alone but MS has market share and easy reach with 365. Slack (as an example) is a much better product but MS can afford to be lazy. Hopefully when Slack releases SIP support it will be forced to improve.
- cdolan 7y agoI dont find teams to be perfect, but I find slack to be overpriced at $15/user. Happy to use Teams as part of my $15/user license that comes with email, excel, PowerAutomate, etc etc
- godelmachine 7y agoI thought Microsoft charged $5/ user for Office 365.
- technion 7y agoOffice 365 means a lot of different things. $5 will get you a business mailbox but it won't get a copy of excel you can install and run offline.
- noelsusman 7y agoIf you're a decent sized organization, you can pay $12.50 per user per month to Slack for a messaging service, or you can pay $20 per user per month to Microsoft for Outlook, PowerPoint, Word, Excel, Access, Publisher, Exchange, Sharepoint, and a messaging service. If you just want messaging then you can pay $8 per user per month, and they'll throw in Exchange, Sharepoint, and a terabyte of cloud storage per user on top of it. I don't think it's possible for Slack to be better enough to justify that kind of a price difference.
- sergiotapia 7y agoIt happens - yesterday we went down for the same god damn reason. There has be a better solution.
- _bxg1 7y agoAmazing. A company like Microsoft could afford to hire an entire department to do nothing but make sure certificates don't expire, but this still happens. Jokes aside, I don't understand how this problem hasn't been solved in the general case.
- wnevets 7y ago>I don't understand how this problem hasn't yet been solved in the general case. isn't that what ACME is suppose to do? https://en.wikipedia.org/wiki/Automated_Certificate_Management_Environment https://en.wikipedia.org/wiki/Automated_Certificate_Manageme...
- _bxg1 7y agoAnd yet
- naikrovek 7y agoMicrosoft are not even close to the only company this happens to. The problem is a human one - as people change positions within a company, or leave a company, responsibility for this kind of thing can fall through the cracks. Happens all the time, even if ACME is employed, and it's unlikely to ever stop happening.
- _bxg1 7y agoAll I'm saying is, that downtime probably cost them a lot of money, and resulted from something that's extremely preventable. Even if it is a human problem, one would think they'd allocate the resources necessary to solve it.
- naikrovek 7y agoyes, in an ideal world it wouldn't happen. and if you really make it a priority you can certainly mitigate it.
- 7y ago
- w0m 7y agoouch
- jlgaddis 7y agoapt install ssl-cert-check
- certera 7y agoI'm going to shamelessly plug my project, Certera, here. It handles monitoring/tracking, cert issuance and renewals and helps larger organizations manage their certificate needs more consistently. https://docs.certera.io https://docs.certera.io
- reagan83 7y agoI just tried to go to www.certera.io to learn more and got a “connection not private” warning page in Safari on iOS. Very ironic :)
- cdolan 7y agoI did not have this issue on iPad OS
- woofcat 7y ago~# dig -t A +short www.certera.io certera-io.github.io. 185.199.108.153 185.199.110.153 185.199.111.153 185.199.109.153 ~# dig -t A +short certera.io 185.199.108.153 185.199.109.153 185.199.110.153 185.199.111.153 Looks like https://www.certera.io https://www.certera.io is going to github of which is only returning a cert for itself, and not his domain name.
- snowwrestler 7y agohttp://www.certera.io http://www.certera.io redirects properly to https://certera.io https://certera.io. https://www.certera.io https://www.certera.io fails the certificate check. It's a good example of the difficulty of getting TLS perfectly right. In theory this set up is fine; the default behavior of all the browsers when typing "www.certera.io" is to interpret it as a request for http://www.certera.io http://www.certera.io. But if the client has anything in place that automatically upgrades http to https before submitting the request, you're going to need a valid cert for the www subdomain in place or you'll throw a cert error before reaching the redirect. Even if your site omits the www subdomain in production (as certera does), a lot of users will just type it in anyway. So, you better be ready to handle that request via https.
- novok 7y agoThere should be a page out there that lists Microsoft outages due to missed certificate or domain expiration. Like that hotmail one long time ago. I would think at this point they would be their own major certificate authority and maybe domain registrar.
- klodolph 7y ago> I would think at this point they would be their own major certificate authority and maybe domain registrar. From experience this probably wouldn't fix things. What often happens is that somebody creates a system that uses a certificate, doesn’t automate renewal, and then the person responsible for renewing it changes teams or leaves the company. Email reminders only go so far—they not only need to go into the right inbox, but the person watching that inbox has to care.
- jeffrallen 7y agoMy last domain expiration outage happened like that. If it's in production, just buy a 10 year cert. This virtually guarantees an outage after 10 years but virtually guarantees it won't be your fault when it happens...
- klodolph 7y agoMy approach is the opposite. For production certs, I buy them with the minimum length (usually 1 year). This exposes problems in our automation sooner, and keeps the process more fresh in our minds. I really like that Let’s Encrypt certificates last only 90 days. That said, the CYA aspect of “10 years, somebody else’s problem” is really appealing. If only I believed that it wouldn’t be my ass on the line 10 years from now!
- tialaramex 7y ago> just buy a 10 year cert New certificates in the Web PKI ("SSL certificates") have a maximum lifespan of 825 days. This is enforced (if a CA were to issue a certificate with a longer lifespan Chrome for example would just treat this certificate as invalid). The commercial CAs mostly offer one year or two years, with renewals using the 825 day limit to offer renewals in the overlap, so e.g. you buy two years in June 2018, in April 2020 you can pay for two year renewal and the new certificate expires in June 2022 not April 2022. If you're using certificates in your own PKI (as it's likely Microsoft actually was in this particular incident) then there's no need to buy them and it's up to you what your appetite for risk is on when they expire.
- chasd00 7y agowhen getting a site/API on its feet, enabling https and the cert is usually the last thing to get done and an afterthought. Certs are easy to forget about but when they expire they shut.down.everything.
- killjoywashere 7y agoWhich is what makes Let's Encrypt sooooo attractive. Free certs are nice but the client is dreamy.
- ChuckMcM 7y agoIt is an interesting side effect of the tenancy of software developers these days that any process that requires action on a > 2 year interval is likely to fail, if the cycle is 5 years or more it will always fail. The turnover insures that nobody in the department was there when the process was started/last interacted with, and so it is off the collective organizational radar so to speak.
- ChrisSD 7y agoThis is why Let's Encrypt has a short cycle.
- tialaramex 7y agoNot really. The short (90 day) Let's Encrypt expiry is intended to promote automation because it's annoying to do so many renewals by hand, and is also a reflection of the relatively short lifetimes of most Internet names. Historically it was common to issue 3 year certs, and five year certs weren't rare (until 2015). But whilst it's reasonable to expect microsoft.com or bbc.co.uk belonging to the same outfit in five years, it's hard to be as sure about say jsnes.org (currently a Javascript NES emulator) or catandgirl.com (a web comic by Dorothy Gambrell) which might well entertain offers from somebody else who wanted those names. The underlying domain name is typically on an annual renewal cycle with perhaps just 14 days grace if you stop paying, and individual FQDNs might have even shorter turnaround. With a five year certificate this means you could buy a certificate the day before your renewal payment is due, and then still have an apparently good, working certificate for that name five years later when it's owned by somebody else entirely who has no idea you once owned that name. Not great. Let's Encrypt's renewal cycle closes this gap considerably. The BRs were also amended, the limit is now 825 days instead of 39 months or (originally) 60 months.
- jamiesonbecker 7y agoThey tweeted that it was an authentication certificate. I.e., probably not a regular TLS domain certificate or similar (still could be TLS client cert though), but probably more like a certificate/key that one service used to log into another. A lot of microservice/container/kubernetes setups use them for all kinds of stuff, which is really a big step forward over password logins. Not like it matters, but it kinda does, because those tend to be private and internally generated, and not necessarily signed by an external certificate authority.
- insomniacity 7y agoI have this problem, and it concerns me, because there's no external polling check that's going to spot that. You have to rely on either the code itself checking each time it uses the certificate, and alerting. Or (taken from elsewhere in this thread) you test it during your build, and hope that someone is still building the code by the time the cert comes up for renewal. I'll probably be doing both.
- cutler 7y agoOur HTTPS overlords have much to answer for. How many static sites, for example, really need HTTPS and the non-trivial maintenance involved in the average Apache/Letsencrypt/certbot setup? Talk about sledge-hammer to crack a nut. And renewals every 3 months?! Don't get me started. Sure, the likes of Microsoft should be able to do better but isn't there a message here? Beyond secure sites such as finance, government, logins and ecommerce the whole HTTPS certificate nonesense is a giant burden/cost with no benefit.
- tatersolid 7y ago> How many static sites, for example, really need HTTPS All of them. Script jacking, ad insertion, redirection, tracking insertion, etc. are all done at scale by everything from national ISPs to coffe shop routers. HTTPS provides authenticity of all transmitted data; this is more important than confidentiality because without authenticity you can’t tell that you are talking in secret with an atttacker.
- cutler 7y agoJust listening to the number of complex shenanigans experienced sysadmins have to employ to keep up with the demands of managing HTTPS makes me wonder how on earth your average non-technical DIY static site developer has a chance in hell of keeping his site from failing modern browsers' requirements. Universal HTTPS is a bad joke.
- jmspring 7y agoDownvoted. If one is registering domains and dealing with DNS entries, etc, it’s not a stretch to find someone to setup LetsEncrypt for you. Heck - googling - https://tecadmin.net/auto-renew-lets-encrypt-certificates/ https://tecadmin.net/auto-renew-lets-encrypt-certificates/
- cutler 7y agoAs a sysadmin myself the number of non-trivial modes in which certbot can fail never ceases to amaze me. Running Apache? Watch your certbot renew fail to bind to port 80 because your server is running. Now your renewal cron task needs to take into account stopping and restarting the server which the standard cron task does not include. What made the web great was that it didn't keep out the DIY developer. Now it does exactly that .... via universal compulsory HTTPS on trivial, static sites.
- baobabKoodaa 7y agoStatic sites can be easily published on Netlify or GitHub Pages at no cost - certificates provided by the host.
- dannyw 7y agoTo be honest, even before Let's Encrypt I could buy a PositivesSSL from namecheap for about $10 a year, and set it up with Apache or Nginx in about 20-30 mins. Had to renew every X years which was annoying, but cmon. There were step by step tutorials. Now, with Let's Encrypt, there's no excuse to not SSL.