7 ms·
Slack was down
- wgjordan 6y agoslack.com returning 503 Service Unavailable. See https://status.slack.com/2020-05/87ad1d12e36fcf0c https://status.slack.com/2020-05/87ad1d12e36fcf0c for incident details: > [May 12, 4:53 PM PDT] Users have reported general performance issues such message sending failures and timeouts. We’re working to get things back to normal as quickly as possible and will provide an update shortly.
- juancampa 6y agoReminder: have a secondary form of real-time communication for your team (e.g. Telegram). What is everyone using?
- lykr0n 6y agoEmail
- Jedd 6y agoMy IRC bouncer (using thelounge, hosted on my own systems) continues to survive through various O365, Slack, etc outages. Can highly recommend.
- sneak 6y agoIf you’re going to recommend something non-e2e for a business team, go with email that is already distributed to everyone’s contact lists, policy controlled, 2fa’d, et c - not some snake oil like Telegram.
- goatherders 6y agoAether. In fact looking at it to replace Slack entirely. Much better controls. Getaether.net/pro
- rolleiflex 6y agoHa, Great to see our thing here. Let me know if you (anyone) ha(s|ve) any questions.
- brassattax 6y agoAre you the head that pops up on your website?
- rolleiflex 6y agoYup.
- no_wizard 6y agoGoogle Hangouts Chat, because our whole company has the G Suite For Business
- hliyan 6y agoEveryone in the team is once again learning the ins and outs of Google Chat after 5-10 years as we speak...
- jaaron 6y agoDiscord (we're a game company). Email too. And some folks are starting to use Google's chat.
- Arathorn 6y agoMatrix :)
- anoa_ 6y agoWhich you can bridge to your slack rooms to keep the existing conversation going!
- apeescape 6y agoTandem
- olyjohn 6y agoDisagree. You will spread out the communication and cause problems. People will use their preferred client instead of the one that everybody should be using as the primary. You also have to manage accounts and permissions for both. Unless you like people sending information around to their personal accounts, which I'm sure a lot of places do. E-Mail is good enough to let people know that something like Slack is down. It's close enough to real-time to be used as a backup for something like this, and everybody already has an e-mail address. My company already has Slack, HipChat and Teams, and it's a communications nightmare. People are constantly confused and asking for which chat client one manager uses, while others don't know where to go for the support channels. Maybe this sounds like old-school IT, but the new-school "set everything up ad-hoc using your personal shit" just sets you up for disaster down the road.
- BryantD 6y agoYou couldn’t do this with Telegram, but if you were using Hipchat or Teams or Mattermost or Discord as your backup, you could just keep the instance locked down most of the time. Open up permissions when Slack is down.
- zenexer 6y agoYou can do that with Telegram if you have a bunch of centrally-managed groups; simply mute them until you need them. Admins can unmute them as necessary. That being said, I don’t think I’d want Telegram to be my failover. They have a tendency to experience regional issues with great frequency, and if you’re in a lot of active chats, things start to get buggy.
- u801e 6y ago> My company already has Slack, HipChat and Teams, and it's a communications nightmare. People are constantly confused and asking for which chat client one manager uses, while others don't know where to go for the support channels. I'm curious about how many people in the company are affected by these problems. Most developers and infrastructure people use the same communication tools, but perhaps support and sales may use their own tools. Given that, how often do developers and infrastructure need to communicate in real time with sales or support? I would suspect that it's not often enough that email wouldn't work as a way to handle issues or open support tickets. Now managers have to deal with communications between their own team and other teams and parts of the org, but, in my opinion, having to deal with different chat platforms is part of their job and they should be the ones to deal with it rather than forcinng everyone to use the same platform, even if it's suboptimal for certain users.
- manquer 6y agohangouts/teams (if you already use gsuite/0365 ). More important problem is our own devops relies heavily on slack based monitoring and chat ops, all integrations don't port without work and never designed for DR
- xref 6y agoWe have a Keybase group that gets used twice a year when Slack takes a dirt nap. We also use Zoom so I guess the buyout doesn't have security implications for us?
- iamryo 6y agoNot official, but our team just made a seamless transition to chatting in Keybase.
- decebalus1 6y agothat's unfortunate
- koverda 6y agohaving trouble sending messages, some aren't making it through
- Jedd 6y agoHeh. Slack isn't down per se -- on their status page they do say there are only problems with 2 / 10 components. Messaging and Connections.
- quickthrower2 6y agoThat means down. What the f' else do they do?
- dmitrygr 6y agocurrently status page claims 9/10 services are up (https://imgur.com/a/qQBax40 https://imgur.com/a/qQBax40): - Login/SSO No issues - Connections Something's not quite right View details - Messaging No issues - Link Previews No issues - Posts/Files No issues - Notifications No issues - Calls No issues - Search No issues - Apps/Integrations/APIs No issues - Workspace/Org Administration No issues
- quickthrower2 6y agoCalls no issues is BS. I couldn't make a call during that dark time.
- Jedd 6y agoQuite. Perhaps I should have been less subtle than opening with a 'Heh', and made it clear the claim that only the messaging & connection parts were failing was tongue-in-cheek. This page's title is "Slack is down" but Slack's status page at the time was modestly conceding that "Something's not quite right".
- interestica 6y agoIt's interesting because the sarcasm came through for me and only because you said "they do say" rather than "they say" - a subtle difference that is probably odd to convey to non-English speakers.
- dpix 6y agoI hope they change the status page from "Incident" to "Outage". Too often lately have I seen status pages (mainly Github) reporting incidents when a feature or site is down completely. Seems like their SLAs are influencing the reporting of their uptime...
- wgjordan 6y agoLooks like it's now updated!
- pthomas551 6y agoSLAs and maybe even company politics. Incidents are politicized at a lot of companies. Even if the official rhetoric says "when it comes to incidents we don't blame people and there are no politics" the reality is often the complete opposite.
- birdyrooster 6y agoPerhaps an "outage" label causes the incident to trigger a more widely attended post-mortem meeting and so they are careful not to over-use it.
- blaser-waffle 6y ago"Incident" could mean several things. Outages mean down, badly, and require RFOs. In my experience working at an ISP we only did any sort of "blame-game" if customers demanded an RFO. We still did post-mortems but usually in a more gentle, roundtable way.
- NullInvictus 6y agoIt's a classic example of Goodhart's Law. Setting a goal of "Zero or Near" down times doesn't necessarily mean less down times, but it usually does mean politics, shell games and 'clever' bookkeeping by middle and lower management, all of which is clearly visible to the rank and file. Before you had down times and customers wondering if they could trust you. After a goal of "Zero or Near", you still have "down times", and you still have customers wondering if they can trust you while they stare at a status page is clearly lying to them. But now you have employees watching their managers lying to their bosses, and wondering why they shouldn't do the same. You have a fall in morale, because those pretty numbers usually come at the expense of the things on the "Our Values" page. But to the Suits the graphs look better, and that's the important bit. This isn't how is always is or always goes, but I've seen it enough.
- MeetingsBrowser 6y agoNo one on my team can send messages at all. Anyone else having any success?
- dmitrygr 6y agoapi.slack.com is returning 503s 503 Service Unavailable No server is available to handle this request. And thanks to the electron-ness of the app, on the mac you cannot even drag the window - the titlebar is drawn by javascript and without it, it cannot be dragged
- jorblumesea 6y agoOh dear: https://imgur.com/a/5aQkRvv https://imgur.com/a/5aQkRvv
- benatkin 6y agoNow might be a good time to add new custom emoji, because that service doesn't appear to be down. https://cultofthepartyparrot.com/ https://cultofthepartyparrot.com/
- EdwardDiego 6y agoI love how one horny kakapo became a legend, never change Internet. (His lovelorn antics also raised a bunch of donations to kakapo conservation when it went viral, ka pai, Sirocco)
- slajax 6y agoDoes IRC still exist?
- slajax 6y agoFreenode, here I come.
- lordraj 6y agoYou should try matrix, on riot.im
- DonHopkins 6y agoMaybe they have an old Glitch or Game Neverending server still running they can use for chatting. https://en.wikipedia.org/wiki/Glitch_(video_game) https://en.wikipedia.org/wiki/Glitch_(video_game) https://techcrunch.com/2008/04/02/game-neverending-rises-from-the-dead/ https://techcrunch.com/2008/04/02/game-neverending-rises-fro... https://en.wikipedia.org/wiki/Ludicorp https://en.wikipedia.org/wiki/Ludicorp
- mepiethree 6y agohow do people at Slack communicate to fix an outage when Slack is down?
- leeoniya 6y agoDiscord?
- slajax 6y agoFreenode
- forgot_old_user 6y agopigeons? But seriously though, just cell phone I guess
- Threeve303 6y agoYeah, The procedure is outlined in RFC 2549
- flatiron 6y agoMy guess is a DR XMPP implementation.
- ggregoire 6y agostaging environment?
- vlan0 6y agoirc :)
- deleted 6y ago[deleted]
- manquer 6y agoIdeally your DR and emergency response systems should be not hosted on your infra and have circular dependencies . i.e.Amazon should not be hosting status page assets on S3 as they were. At the very least they should be hosted isolated of your production service components at all layers, so both don't go down at the same time. For slack my guess is their internal slack is not hosted on slack prod and hopefully also does not share components below L4 as well ( DNS, IP , domain, routing etc )
- quickthrower2 6y agoHad to email the team to use Zoom. Ugh! Will anyone turn up I wonder :-)
- fermienrico 6y agoIf you noticed how insanely Slack freaked out and handled the service outage. The application just got stuck, didn't know what to do, I see a faint 503 error message which looks like a webpage nginx error displayed right in the app. Clearing cache didn't help and the app just quit. Electron apps are a nuisance and should be eradicated from this planet. We've come a full circle -> Bloat the web -> Create javascript frameworks -> Build SPA/PWA apps -> Invade the desktop land -> Gain traction because "its crossplatform" and now try to reimplement GUI elements from some fancy_new framework that is developed by people who are entirely unqualified for it. Throw away all the work that's done by OS developers and designers that are getting paid as professionals to develop guidelines like this: https://developer.apple.com/design/human-interface-guidelines/macos/overview/themes/ https://developer.apple.com/design/human-interface-guideline... The modern tech stack needs to be refactored just like they refactor silicon architecture every 10 years: https://www.youtube.com/watch?v=Nb2tebYAaOA https://www.youtube.com/watch?v=Nb2tebYAaOA Edit: People talking about would native app handle this better? Sure. I argue that it wouldn't just get rid of all UI elements and drop the entire DOM on the page, the UI just crashed. And everytime I open slack it seems like its doing something else entirely. There is a massive difference in quality of native UI libraries and Bulma for example, and its not just the library, you gotta rely on React to redraw things and it depends on 4 billion npm dependencies. The web truly sucks. And it is now eating into desktop apps.
- ajkjk 6y agoThis is one of those stances that I see on HN all the time but never feel myself. Like, nothing about this outage affected my opinion of Electron apps at all. And in general I like them, as a developer and as a user. They're polishable with so much less effort.
- anyfoo 6y agoAs an OS engineer, I hate them. They cause problems and inefficiencies behind the scene that are pretty significant. I can see how you wouldn't say so from another position, it's a bit like propelling a passenger aircraft with controlled nuclear explosions: "It flies so effortlessly!"
- foobarbecue 6y agoWhen I first got a 503, the status site was all green. I checked back for at least 4 minutes and it was still green. Finally it started to change. Is that sort of lag normal?
- dmitrygr 6y agoIt is manually updated, by humans. Humans have high latency. It is not "agile" or "modern" or "hip" to have automated status testing/reporting. A human had to go and change that status page.
- taurath 6y agoWhen you're a public company the risk of your automated system saying something that invokes more fear than is necessary gets a lot higher.
- dmitrygr 6y agoYup, that is also why the status page now says things are "9/10 up" and for the first 20 minutes said "something is maybe slightly kinda wrong" nad not "outage". As if pretending nothing is wrong will make things right again.
- _jal 6y agoYep. I would be surprised to learn of non-infrastructure companies that wired their public outage notice directly to automated monitors with no human confirmation.
- foobarbecue 6y agoAha, I had figured it would be automatic. That does make sense, though.
- lykr0n 6y agoMaybe the status only changes when they've had 5 consecutive failures, and the probe only runs once a minute?
- 6y ago
- faCeti0us 6y agoIt's a cogent network issue...
- jpxw 6y agoHm, I don’t see anything on https://ecogent.cogentco.com/network-status https://ecogent.cogentco.com/network-status
- faCeti0us 6y agoOur cogent support rep said there's an outage. "We are currently experiencing an outage on our network. We are working to resolve this as quickly as possible. Please check http://status.cogentco.com http://status.cogentco.com for updates on this event."
- rantwasp 6y agoand they are using slack to communicate... so deadlock...
- zenexer 6y agoPeople were getting 503 errors, which indicates it’s something within Slack’s own infrastructure. Granted, external network issues can trigger internal problems, but I wouldn’t be pinning this on Cogent just yet.
- fabianhjr 6y agoSince no one can connect there are no messages to deliver and therefor no message errors therefor everything but connections is up and working. /s (The Slack Status Page)
- deleted 6y ago[deleted]
- manquer 6y agoThis is a big problem, for teams using Slack for their own DevOps- monitoring and ChatOps. DR plans have to start thinking how to get all the integrations working with multiple communication providers to handle this type of scenario
- deleted 6y ago[deleted]
- pmccarren 6y agoI'd appreciate a thorough postmortem on this one to satisfy my curiosity.
- _eht 6y agoFreenode is up. FYI.
- kabacha 6y agoso is matrix.org, wait a second — you could have your own matrix server or even multiple ones! Centralized connections are destined to fail.
- samcheng 6y agoLooks like it's back up, at least partially. They're making progress... It's crazy how easy it is to depend on Slack. Email collaboration feels downright archaic. What a great product!
- burgerzzz 6y agoSeriously, I just realized how much it's apart of my workflow as an engineer with the different integrations we've setup.
- jshevek 6y agoFrom what I have seen on https://downdetector.com https://downdetector.com today, I suspect this will prove to at least partly be on the AWS end.
- ipsum2 6y agoXBox Live and YouTube probably don't run on AWS, but also experienced a peak. This is a neat site, thanks for sharing it.
- dilandau 6y agoWhat a perfect example of the modern status page meme. App is completely fucking not working and it's only showing a slight problem with "connections to slack". Something's not quite right. Is this valley casualspeak or actually an attempt to minimize the fact that their shit is down? Ok bro, you're not fooling anyone though.
- tcbasche 6y agoProbably more diplomatic than 'we have no idea what's gone wrong'
- samrohn 6y agoIf anyone is curious about their deployment process https://slack.engineering/deploys-at-slack-cd0d28c61701?gi=efefc4866bb1 https://slack.engineering/deploys-at-slack-cd0d28c61701?gi=e... Seems hard for a new deployment to cause a global outage since they do a phased deploy. Could be some network glitch causing this issue. We never know until they come up with an update.
- dangwu 6y agoEverywhere I’ve worked has done phased deployments. Shit still goes wrong all the time. Also, I would wager that most outages are not due to code deployments, but live config changes.
- samrohn 6y agoMakes sense. I recollect some of the postmortem reports I read where live config changes triggered cascading outages.
- steelframe 6y agoUnderscoring the maxim in production environments: Configuration Is Code.
- deleted 6y ago[deleted]
- zenexer 6y agoAny chance we’ll get a post mortem on this? I’ve got a bet to win.
- franciser 6y agoJust a reminder, SLA doesn't apply to you if you are a free user. Folks, stop complaining about their downtime if you didn't even pay for the service. It's just like free food, don't complain about the taste if you get it for free.