11 ms·
Slack – Degraded service affecting multiple features
- tamalsaha001 7y agoIt is Friday evening somewhere!
- miyuru 7y agoit is Friday evening in South Asia
- jumbopapa 7y agoNo outage for me, but I'm seeing some messages appearing twice. Is that related?
- lm28469 7y ago> Is that related ? That's the symptom of the outage as far as I can tell.
- philshem 7y agoThe replies to this status tweet seem to indicate it’s related https://twitter.com/slackstatus/status/1144577107759996928?s=21 https://twitter.com/slackstatus/status/1144577107759996928?s...
- mjl- 7y agothe slack app was telling me messages weren't sent, asking for a retry. perhaps people retry and succeed twice. but i've seen messages as sent twice in my window, but they disappeared after reload of the app/page.
- koenigdavidmj 7y agoMy org is having a few people see that, or messages getting silently dropped, or failing to edit an existing message.
- jumbopapa 7y agoOkay, yeah. I'm seeing some of these issues now. It's odd because the message said it failed to edit, but my client still shows the message as edited.
- sgt 7y agoI'm getting a bunch of duplicate messages. Luckily these guys are building a chat platform and not financial systems.
- nielsole 7y agoBest thing is that some messages go through despite returning 500s. This leads to our integrations spamming our companies channels with redundant messages while retrying. :/
- cabaalis 7y agoOuch, that one hurts. I don't know slack API but the contract expectation in the call should absolutely be "500 means the message failed to send" Tells me there's some subsystem throwing the error, not the actual messaging (or, if intermittent, something wrong with the distribution.) What do you want to bet it's some kind of new analytics?
- bdonlan 7y agoIt's not possible to always ensure that you report whether an operation succeeded. Consider a timeout - you waited 5 seconds for the backend to respond and heard nothing - was it, or will it be successful? You don't know. Typically a 500 should be considered as a "may or may not have succeeded", and steps taken to deduplicate operations if necessary.
- toast0 7y agoHttp 500 is like when the server drops connection before returning a response. Something happened, but you have no idea what happened. You can't safely retry unless your request is idempotent enough.
- oxfordmale 7y agoHas the CTO and key technical staff retired after the Slack IPO ? :-) In the UK there is no real outage, although Slack struggles sending some messages and this can result in some messages appearing twice. This problem seems to be worse when communicating with offsite staff, onsite messaging seems to be fine.
- TelmoMenezes 7y agoIn other news: an unexplained productivity spike has been recorded today across the tech industry.
- darkwater 7y agoOr not. I'm remote and I cannot ask questions nor coordinate action to solve live production problems due to this outage. Slack is becoming a SPOF for many organizations, especially distributed.
- pmlnr 7y agoThis is part of the problem - we all need to ask too many questions. We should be working on pre-defined tasks most of the time, where it doesn't matter if it's waterfall pre-defined or scrum time allocated.
- ibero 7y agoThe commenter above said they had to "solve live production problems", how exactly would "pre-defined tasks" help them?
- pmlnr 7y agoI slipped over the "live" word, but I still stand by what I said. I most probably work in a similar environment, with tonnes of live production issues, and I believe it's a result of not being allowed to proactively plan for situations - most of the issues are a result of cost savings, systems driven to their absolute maximum capacity, etc. Live production issues are a byproduct of moving too fast, breaking too many things.
- wastedhours 7y agoI'd style my way out of that argument by saying out reliance on tools like Slack rather than other pre-planned work builds a reliance on that one tool for all tasks being done "in the moment", which then means it's relied on as the medium in live production issues. If there was a lower reliance on IM for all tasks, other solutions for crisis management would be utilised (potentially).
- yyyk 7y agoIs it just me, or do those outages seem far more common lately? Or maybe this is happening just as often as it used to, but now we're growing more dependent on cloud-based software so this is reported more prominently?
- lallysingh 7y agoMore dependant.
- rnicholus 7y agoboth - have a look at Slack's status history. They seem to have an outage of some sort every month
- mbesto 7y agoMeta - can we not post outages to HN? I understand that post mortem's are interesting learning lessons, but if you're curious whether S3 or whatever is down or not right at this moment, then use the subsequent status pages of those services.
- Kiro 7y agoNo, this is major and should be on top of HN IMO.
- majewsky 7y agoHN is a more reliable status page than most status pages.
- invalidusernam3 7y agoPersonally, I enjoy the speculation and discussion during an outage like this. Also, occasionally someone from the mentioned company pops up in these threads
- sergiomattei 7y agoAgreed. It's fascinating to see speculation and it's almost a game when the real reason is found - we see who got it right! It's also important - many here use Slack, and an outage may be confusing - most of us don't immediately look at status pages, so we have somewhere to go and discuss the happening.
- deleted 7y ago
- kevcampb 7y agoIf it's any consolation, their status board has actually been reporting errors (hi cloudflare)
- jgrahamc 7y agoYou're implying we don't update the status board when there are errors?
- kevcampb 7y agoThe recent outage was very very delayed. I stand by my comment. Also don't get me wrong, you do a great service. It's just a pet peeve that it seems invariably status pages are a lie.
- deleted 7y ago[deleted]
- kevcampb 7y agoLet me quote from the previous HN discussion ”"”sauldcosta 4 days ago [-] We use downdetector.com because status pages tend to take up to an hour or so to update, if they ever do. reply jgrahamc 4 days ago [-] 1042 UTC First alert of global traffic problem 1057 UTC Internal group chat room up and running 1102 UTC Status page updated So, first alert to status page was 20 minutes.””” In those 20m we had repeatedly checked your status page, realised it was our issue and started pulling engineers to deal with it as per procedure. People are on call, it's highly disruptive. Surely you knew within those 20m that something was up? In the end we realised it wasn't our issue because we checked Twitter. Edit - added quotes
- jgrahamc 7y agoYep. I tend to agree that we could have gone faster. The difficulty is that getting clear information out fast when you are dealing with a difficult situation is hard. But, I guess we could have put some status up quicker.
- pmlnr 7y agoThere are reasons why email is async and supposed to be decentralized, with priority based fallback options. (The priority in mx records). Most email servers try to deliver to fallbacks, and even if that fails, will try to deliver for _days_. For all people who think slack can replace email, think about these safeguards.
- thejohnconway 7y agoIsn't it better in nearly every situation to know something has failed immediately, rather than get a delivery failure notice a couple of days later?
- deleted 7y ago[deleted]
- ajross 7y agoIn general mail servers will reply back with a failure notice within an hour or so. This isn't a standardized behavior and timeouts and retry semantics differ.
- fogetti 7y agoThe whole point of email is fire and forget
- pmlnr 7y agoIf it's a temporary outage, eg. network loss of a datacenter, email is not lost. It will be delivered with a delay, tried over and over again, until the timeout limit is reached. Due to the async nature the delay is not a problem.
- yifanl 7y agoI don't think any serious IM platform really intended to replace email, they serve different (but related) purposes.
- pmlnr 7y ago
- sidcool 7y agoIf slack outage brings it on top of Hacker News, I think it's a compliment!
- dillonmckay 7y agoYou must not be on the $12/user/month plan.
- lokimedes 7y agoA resurgence of IRC is nigh!
- psnosignaluk 7y agoYou mean I have to open OpsGenie and Grafana now instead of waiting for a ping in the ops channel??? Also, how many IT service desk ticket systems just lit up with people asking them to fix Slack?
- valerij 7y agoreminds me of that one time a genius at IBM created an `ibm-global-announcements` channel and force-invited 200'000 people in it, and then some guy `@channel`d and all ibm's slack workspaces were down for 30 minutes https://status.slack.com/ibm/2018-03/f01d4c22cd953dd7 https://status.slack.com/ibm/2018-03/f01d4c22cd953dd7 https://i.imgur.com/Rk6Kdgp.png https://i.imgur.com/Rk6Kdgp.png EDIT: also that channel made using slack impossible for mac book air users, they had around 80% cpu usage for slack. so basically entire marketing and PM part was unable to work that day. developes machines were wasting around 20% on slack. after people started complaining in that channel, posting in it was limited to admins only, but they didn't lock commenting. so, for approx 6 hours all of IBM was posting memes in ＴＨＥＴＨＲＥＡＤ as we dubbed it, and @mentioning the genius who created that channel. next day the channel was nuked, not even an archive preserved. some guy calculated that the entire affair, considering electricity prices, a 20% decrease in developers productivity, and so on resulted in IBM loosing several millions with that stunt fun times p.s. shout out to Martinj for that spicy jeff-coffee-mug meme
- freehunter 7y agoThere should be a way to turn off @here and @channel. It should be a user-level setting, a channel-level setting, and an organization-level setting. Even if my org or channel doesn't opt out, I should be able to.
- deleted 7y ago[deleted]
- foobarian 7y agoI hate that it can't be controlled per channel. There are many channels where it's useful and desirable, so our org doesn't disable it globally. This creates situations where new people @here and @channel not knowing any better, which leads to hurt feelings all around.
- ilikehurdles 7y agoSome of this you can already do. Users can opt out of @here and @channel for any given channel. I know that orgs can restrict usage of @everyone, I assume they can do the same for @here/@channel but I'm not sure.
- rooam-dev 7y agoDid they enable "replace email" feature?
- nashahg 7y agoHaven't noticed anything out of the ordinary. https://finance.uonbi.ac.ke/node/114148 https://finance.uonbi.ac.ke/node/114148
- ilikehurdles 7y agoRun a virus scan after clicking on nashahg's link.
- lqet 7y agoCan someone explain to me why any company should outsource something as critical as internal communication to a company that is 5 years old? The video messaging left aside, is it really that better than time-tested solutions like email or just running an IRC server?
- trickstra 7y agoOur company switched from self-hosted RocketChat to Slack just this week. Personally I didn't agree with it, but it's not my call. The reason was mostly that Op guys were too busy with other things and didn't want to care about yet another critical company-wide service. They though outsourcing it would solve the downtimes and bugs. This outage couldn't come at a better time.
- xiconfjs 7y agono, it's not. It's just a company based whatsapp group chat you can use while looking busy
- zymhan 7y agoIt has emojis and is the Next Best Thing (tm)
- falcolas 7y agoBecause they don’t trust their IT or operations groups to manage IRC or any other open source communication system; even as they trust those same people to run the software that makes them money. /shrug
- lqet 7y agoIRC is just a protocol, neither the server nor the client do have to be open source.
- e1ven 7y agoIt's a reasonably nice client, and available on every platform. It has easy integrations with basically everything, and non-developers can set them up on their own. It has bouncer functionality built in - You don't miss messages. It can also archive them, and handle them in a corporate-compliant way without needing to speed a bunch of time setting that up manually. Because it's a central company and not IRC push-notifications are easy. Emoji are shared corporate wide, anyone can add them. It's a really nice out of the box experience. You can absolutely replicate it with enough setup and tweaking, but the free version of Slack will go a long way.
- jzig 7y agoBy accident I noticed that you can add as many characters as you want to the end of the url and it will still load that page. Neat.
- QuickToBan 7y agoWhy don't people use free software instead, e.g. a Matrix homeserver run via https://github.com/matrix-org/synapse/ https://github.com/matrix-org/synapse/ ? IRC without a BNC doesn't really work for a corporation due to disconnects.
- whatshisface 7y agoHow well does Matrix handle VoIP?
- pmlnr 7y agoBecause it's way harder to get it running right.
- exabrial 7y agoOne problem with Slack we have in the finance world is that _literally everything_ employees say on company assets needs to be recorded and readily accessible for audit (No this is not an invasion of privacy, you're on company time on company systems, we don't peer into your personal phone or anything). Unfortunately, Slack does not do this very well. I'd really like to know if there's another service that _does_ do this well.
- fancy_pantser 7y agoZulip and a very big disk.
- exabrial 7y agoLooked at this, appears to make it difficult by default: https://zulip.readthedocs.io/en/latest/production/security-model.html https://zulip.readthedocs.io/en/latest/production/security-m...
- rishig 7y agoMy guess is most of the major chat players handle this pretty easily -- it's a non-starter for many potential users if you don't. Zulip stores all data indefinitely (including message edits, etc) in its default configuration.
- nostoc 7y agoMattermost, but you'll have to manage it yourself.
- daneyh 7y agoSymphony chat, what goldmans is trying to aim as the bloomberg IB killer. It's quite nice and designed for banks and API/Bot friendly. url:https://symphony.com/en-US https://symphony.com/en-US
- 2a96eb7d685a49c 7y agoMattermost? https://docs.mattermost.com/administration/compliance.html https://docs.mattermost.com/administration/compliance.html
- gonzo41 7y agoThey should call this sort of event a 'Slackening'
- biztos 7y agoJust yesterday I was musing that if I were King of the (World|Company) I'd want an open-source Slack-alike that I could just drop into the Cloud of my choice and operate entirely within my private network, subject to my own access control just like other internal services, and with full access to all message histories in whatever database-like thing it uses in its Cloud. Sure, I'd still have a SPOF but it's game over anyway if my Cloud goes dark. Is there such a project, and if so does it have any traction in the real world? (Oh and ditto for every damn thing Atlassian owns. Yikes.)
- thg 7y agohttps://mattermost.com/ https://mattermost.com/
- asah 7y agoWe use this at my company - perfectly reasonable UI, don't know about the APIs/integrations, which I assume are way behind Slack...
- lwhalen 7y agoThe Mattermost API is a superset of Slack's. "Most" integrations "just work", the rest need a minimum of tweaking to DTRT.
- zild3d 7y agoMattermost API being a superset of Slack's seems to be mostly true, but looking at the features our Slack app uses I don't see block kit (https://api.slack.com/block-kit https://api.slack.com/block-kit) or message scheduling. If we were to integrate with Mattermost it doesn't look like it'd be a ton of work though Have to say Mattermost's API docs are so much easier to read than Slack's.
- icelancer 7y agoSlack's API documentation and handling of accounts/bots is so obscure. I've never seen anything that bad from a major company.
- notacoward 7y agoCame here for the "Slack shouldn't exist anyway" comments and was not disappointed. Or rather I was, but in a different way. Best wishes to all of the folks at Slack and those who depend on it.
- mstg 7y agoHonestly, the same. It's tiring that people just has to suggest self-hosting IRC or any other software all the time when Slack is mentioned. It's a good way to communicate and it's easy. Hope they resolve it, overall a bad situation.
- buboard 7y agoWell i m not affected. But then again i dont use it
- jbverschoor 7y agoWooo productivity!!
- jbverschoor 7y agoSo now everybody can get back to $WORK