9 ms·
Slack is experiencing a service disruption
- iamjohnsears 5y agoSwitching to Cloudflare DNS fixed this for me
- michael_michael 5y agoThank you. Switching to 1.1.1.1 got me back online.
- eigthbits 5y agoGood tip, back up, appreciate it
- daxuak 5y agoThank you, I switched to cloudflare/google DNS and flushed router's DNS cache, then it worked.
- phlhar 5y agoWow no DNS A Record for slack.com. Somebody fucked up
- keville 5y agohttps://lists.dns-oarc.net/pipermail/dns-operations/2021-September/021340.html https://lists.dns-oarc.net/pipermail/dns-operations/2021-Sep...
- deleted 5y ago[deleted]
- terom 5y agohttps://dnsviz.net/d/slack.com/YVXX_g/dnssec/ https://dnsviz.net/d/slack.com/YVXX_g/dnssec/ the dnsviz analysis showing the slack.com zone DNSKEY existing at 12:55, followed by the the .com zone DS record at 15:30. However, the next analysis at 17:24 shows both the .com zone DS and slack.com DNSKEY records have disappeared! Given that the slack.com DNSKEY shows up with a 1h TTL and the .com zone DS has a 24h TTL, they are screwed in the presence of cached slack.com DS records from the .com zone. Do not throw away your DNSKEY until your delegation's TTL has absolutely positively surely expired from any resolver caches! The slack.com domain is an AWS Route 53 zone, I'd be really interested to see a post-mortem explaining what happened here. Are they unable to recover the KSK/ZSK and restore the DNSKEY/etc records?
- dnsmichi 5y agoGreat analysis, thanks! Slack support says that users should tell their ISPs to invalidate the DNS cache for slack.com https://status.slack.com/2021-09/06c1e17de93e7dc2 https://status.slack.com/2021-09/06c1e17de93e7dc2 (access with 8.8.8.8 as resolver - fallback https://slack-status.azureedge.net/ https://slack-status.azureedge.net/) Since the faulty DS record was in .com, everyone has a max wait-for-ttl-to-expire time of 24h. Google/Cloudflare etc. seem to also invalidate .com caching very quickly, 8.8.8.8 quickly was the first workaround. Meanwhile, 14 hours later, DTAG in Germany still does not resolve. The default resolvers have dnssec enabled. dig slack.com +cd tells the resolver to skip dnssec validation tests, and then it works again. Screenshots with the command output in https://twitter.com/dnsmichi/status/1443840645513293853?s=2 https://twitter.com/dnsmichi/status/1443840645513293853?s=2 Very interested in the post-mortem analysis. I think there were similar mistakes as with nasa.gov incident and the comcast analysis in 2012: https://www.internetsociety.org/blog/2012/01/comcast-releases-detailed-analysis-of-nasa-gov-dnssec-validation-failure/ https://www.internetsociety.org/blog/2012/01/comcast-release... Learnings for me: - dnstracer (https://gitlab.com/dnsmichi/dotfiles/-/blob/main/Brewfile#L57 https://gitlab.com/dnsmichi/dotfiles/-/blob/main/Brewfile#L5...) helps with detecting missing glue records, but not dnssec - dnstrace (https://github.com/rs/dnstrace https://github.com/rs/dnstrace) is a better alternative with dnssec
- keville 5y agoIf you can't reach slack.com, here's their status page: https://slack-status.azureedge.net/2021-09/06c1e17de93e7dc2 https://slack-status.azureedge.net/2021-09/06c1e17de93e7dc2
- omreaderhn 5y agoBased on that status page and the list-serve email link you posted it seems like they still don't understand what's going on (if that email is correct).
- ricardobeat 5y agoIt seems to be standard practice to be as fuzzy as possible during an outage, and only share details later. Probably avoids anyone looking stupid if the initial hunch turns out to be wrong.
- pjlegato 5y agoMany outages are caused by malicious actors attacking your infrastructure. This cause may not be apparent at all until much later. It's better to avoid leaking any information in general during the incident, as it's often not immediately possible to know whether a hostile adversary exists, nor what advantages the adversary might derive from detailed updates during the incident.
- bastardoperator 5y agoThese are my favorite outages
- patrickbolle 5y agoShopify has been down quite a bit today as well.
- jftuga 5y agoYes it has, https://downdetector.com/status/shopify/ https://downdetector.com/status/shopify/
- giacaglia 5y agoCould it be related to: https://techcrunch.com/2021/09/21/lets-encrypt-root-expiry/ https://techcrunch.com/2021/09/21/lets-encrypt-root-expiry/?
- tialaramex 5y agoNo But, however in that regard it is interesting to see the shoe on the other foot for Firefox this time around. Historically Firefox used to rely on remembering recent intermediates it had seen to "fill in the blanks" when trying to access a site that doesn't bother providing any reason to trust the certificate it offers†. But, Mozilla spent years collecting knowledge of all unconstrained intermediates for trusted root CAs. Then they shipped the entire set as part of Firefox. So, a modern Firefox visiting a site that only has a Let's Encrypt certificate and no other reason to be trusted, sees that and goes oh, that's issued by Let's Encrypt's R3, which I have a copy of, and so it's trustworthy. Done. Whereas for several other popular browsers they're looking at potentially stale local information, and may conclude that they can't trust this site because some stale data has expired or they rely on an already expired certificate sent to them in some cases rather than disregarding it. † You're supposed to provide a "chain" (it doesn't actually need to be a chain, but that's most compatible) of other certificates to show why the leaf certificate you have is trustworthy, e.g. my certificate is from Z9, Z9 has a certificate from Big Trusted Corp Q46, Big Trusted Corp Q46 has a certificate from Very Famous Trusted CA, and the relying party (well, their client software) goes, "Oh, I see, and I trust Very Famous Trusted CA, so that means I trust you're you". But, lots of web sites (maybe 10-20% and more of the smallest with negligible IT budget) don't get this correct so web browsers try to work around it.
- Deathmax 5y agoThere's two Let's Encrypt issues at play at the moment. The issue that your comment about cached/discovered intermediates relates to is that some servers were still manually constructing the chain that goes leaf -> R3 -> DST Root X3, and the R3 intermediate signed by X3 expired on 29 September. This chain hasn't been returned by Let's Encrypt since May 2021. The other issue relates to the current default chain returned by Let's Encrypt that goes leaf -> R3 -> ISRG Root X1 -> DST Root X3 for Android compatibility. Most clients are able to successfully build a valid chain from the leaf to the still valid ISRG X1, however, old versions of OpenSSL (pre-1.1.0) and several other TLS libraries that don't explore the graph correctly barfs at a chain that terminates in the now expired X3 root. [1]: https://community.letsencrypt.org/t/production-chain-changes/150739 https://community.letsencrypt.org/t/production-chain-changes...
- thomascgalvin 5y agoProductivity, on the other hand, is through the roof.
- milkers 5y agoContrarily for me I can not add myself to company VPC, because the command works on Slack. What a bummer.
- snvzz 5y agoThat's some terrible OPs and ridiculous single point of failure.
- kaustubhvp 5y agotime to go home...
- dylan604 5y agocloses laptop. phew, short commute today.
- siva7 5y agothe covid years sufficiently described
- dylan604 5y agoif i were a poet, i might have tried to put it in a meter. since i'm a coder, i made it a one-liner.
- JonathanMerklin 5y agoI'll do my next reboot at nine, I'll do my next commute with wine.
- sieabah 5y ago
- nathanyz 5y agoCould be related to the currently ongoing Let's Encrypt certificate expiration event(1) that is causing fun for engineers today(2). (1) https://community.letsencrypt.org/t/help-thread-for-dst-root-ca-x3-expiration-september-2021/149190 https://community.letsencrypt.org/t/help-thread-for-dst-root... (2) https://twitter.com/search?q=letsencrypt&src=typed_query&f=live https://twitter.com/search?q=letsencrypt&src=typed_query&f=l...
- betaby 5y agoNo. DNSSEC has nothing to do HTTPS.
- nathanyz 5y agoInterdependencies among services can cause all sorts of unexpected issues.
- eropple 5y agoNo, it's not. This is a fault that appears to be from removing DNSKEY and DNS records at the exact same time. Let's Encrypt doesn't have anything to do with that even if somebody uses it. Slack uses DigiCert.
- kelnos 5y agoIt can, but it did not, in this case. Slack uses DigiCert, not LetsEncrypt, and if you poke entries into your /etc/hosts file for various slack hostnames, things work just fine (which they wouldn't if it was a TLS cert problem).
- madars 5y agoYou can test it at home using "delv www.slack.com @4.2.2.1 +rtrace" (4.2.2.[1-6] are Level 3's servers): ;; fetch: www.slack.com/A ;; fetch: com/DS ;; fetch: ./DNSKEY ;; fetch: slack.com/DS ;; fetch: com/DNSKEY ;; fetch: www.slack.com/DS ;; validating slack.com/SOA: got insecure response; parent indicates it should be secure ;; no valid RRSIG resolving 'www.slack.com/DS/IN': 4.2.2.1#53 ;; broken trust chain resolving 'www.slack.com/A/IN': 4.2.2.1#53 ;; resolution failed: broken trust chain
- exikyut 5y agoFWIW, after getting exactly the same output from `delv` locally, I noted that $ dig +short slack.com @4.2.2.1 15.206.34.128 and I take this to mean that, despite failing DNSSEC, Level 3 is yielding an A record - and that the major players have basically monkey-patched this into working again, and it just needs to propagate through all the caches now?
- ArchOversight 5y agoNope, Level 3's DNS servers don't do DNSSEC validation, as can be tested with dig: % dig www.brokendnssec.net +dnssec @4.2.2.1 ; <<>> DiG 9.10.6 <<>> www.brokendnssec.net +dnssec @4.2.2.1 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58098 ;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags: do; udp: 8192 ;; QUESTION SECTION: ;www.brokendnssec.net. IN A ;; ANSWER SECTION: www.brokendnssec.net. 300 IN A 104.22.35.212 www.brokendnssec.net. 300 IN A 172.67.36.129 www.brokendnssec.net. 300 IN A 104.22.34.212 ;; Query time: 84 msec ;; SERVER: 4.2.2.1#53(4.2.2.1) ;; WHEN: Thu Sep 30 17:59:35 MDT 2021 ;; MSG SIZE rcvd: 97 % dig www.brokendnssec.net +dnssec @8.8.8.8 ; <<>> DiG 9.10.6 <<>> www.brokendnssec.net +dnssec @8.8.8.8 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 22514 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags: do; udp: 512 ;; QUESTION SECTION: ;www.brokendnssec.net. IN A ;; Query time: 96 msec ;; SERVER: 8.8.8.8#53(8.8.8.8) ;; WHEN: Thu Sep 30 17:59:44 MDT 2021 ;; MSG SIZE rcvd: 49
- jftuga 5y agohttps://downdetector.com/status/slack/ https://downdetector.com/status/slack/
- telesilla 5y agoWe had this issue with a desktop client, this seems to be working for us as a fix: https://twitter.com/Zap42/status/1443647882045927427 https://twitter.com/Zap42/status/1443647882045927427 Also, just waiting, or maybe rebooting. Edit: ah wrong thread sorry! This is re: letsencrypt.
- judge2020 5y agoSounds like that's a different issue then, because that solves the LetsEncrypt issue - the Slack issue is thanks to DNSSec (and Slack uses a Digicert HTTPS certificate[0]) 0: https://crt.sh/?Identity=slack.com&exclude=expired&match== https://crt.sh/?Identity=slack.com&exclude=expired&match==
- kelnos 5y agoSlack doesn't appear to use LetsEncrypt (checking their cert on slack.com and app.slack.com shows a DigiCert cert), so this will not do anything to help.
- black_13 5y agoSlack is a disruption slack is an answer to question that no one is asking. “Did you ask the slack channel” is new speak for go “i dont know”.
- testplzignore 5y agoI like how all 5 status updates (as of now) basically say the same thing, though they rushed the third one and forgot to apologize :)
- zucked 5y ago>We are aware of connectivity issues (…) In order to resolve this faster, your ISP (Internet Service Provider) will need to flush their DNS record for slack.com. Please reach out to your networking team to provide them with this information. Yeah, sure, lemme just pick up the phone and call my ISP and let 'em know to flush their DNS record. I'm sure the T1 rep will absolutely know how to handle that.
- betaby 5y agoThat's a laughably bad and dishonest response from Slack. They are redirecting the blame to ISPs now.
- azundo 5y agoAre they blaming them though? Is this not the actual fastest path to resolving the issue?
- walrus01 5y agounless you are a customer of a very tiny ISP the odds of getting the person on the phone who has root on their recursive caching nameservers provided to DHCP clients are minuscule.
- znpy 5y agoAnd chances are they wouldn't flush that entry for you anyway.
- kreeben 5y agoI wonder how much electricity will be spent flushing then repopulating the caches of each and every DNS server on the globe. Prolly not that much?
- lima 5y agoNo need to flush the full caches. Clearing caches for individual zones is not uncommon.
- 40acres 5y agoStarted a new job in July. It's my first time using slack so heavily for day to day business needs. After some adjustment I've gotten used to it and enjoy it a lot. Now it's down and I feel like I can't do anything. We have gchat available but no one else is using it and we can't recreate our slack channels in gchat. I've been rendered totally ineffective.
- shados 5y agoMy last couple of jobs have involved being on slack almost all the time. Since when its down we couldn't do anything, I just got into the habit of keeping a Google Chat room that everyone knows about. If Slack goes down, we just hop into the google chat room. Google Chat got a lot better in the last few years, it's actually fairly serviceable now.
- myth_drannon 5y agoI managed to find a work around by connecting using my company's VPN but still can't do directly on my home line
- sys_64738 5y agoThis is a good thing.
- wly_cdgr 5y agoFinally, we can get some work done, kekkek
- jp0d 5y agoIt's quite bad at the moment. I can't message anyone at work!
- annoyingnoob 5y agoIf only DNS wasn't so slow.
- dukeyukey 5y agoIt can't be great, working something where if you cause an outage, your entire industry knows in five minutes.
- tbarbugli 5y agoFrom their status page > Less than 1% of users may be experiencing trouble connecting to Slack This is a good example of how to not communicate uptime issues. It clearly shows that Slack is trying to downplay the problem and do damage control. This is not smart and gets customers upset.
- qw 5y agoAlso it seems to be based on assumptions and not facts. It is currently not working for many people at work who are sitting at home or at the office. Changing DNS to 8.8.8.8 "fixes" it, but it is clearly impacting more than 1% considering they are spread out over several countries and ISPs at home.
- fulafel 5y agoAnyone have theories if they could have (and if so, why didn't they) just put back the signatures?
- lewsid 5y agoSlack is continuing to have issues this morning. You know it's bad when even the status page is down.
- LegitGandalf 5y agoAs I've used slack over the last 5 years I've come to the conclusion that they have a quality problem of some sort. The software just behaves weird, especially in the voice and screen sharing areas. It seems like their management runs the "quality via escalation" anti-pattern.