6 ms·
Slack is offline
- legohead 9y agodon't push code at 4pm! and on Halloween, oof..
- deleted 9y ago[deleted]
- akulbe 9y agoSlack DOWN! Productivity UP! #jokingpeopleCOMEON
- peterlk 9y agoThis page seems to not be throwing 500s: https://status.slack.com/2017-10/8b0d4d44ea53726f https://status.slack.com/2017-10/8b0d4d44ea53726f
- deleted 9y ago[deleted]
- cadr 9y agoWhen the status page is returning a 500 error... not a good sign.
- Gibbon1 9y agoIt's a sign that we need to appease the beer gods.
- munk-a 9y agoIt's an even worse sign when the page half-loads with some stylesheets missing.
- Florin_Andrei 9y agoMaybe they should not serve the static content with "Cache-Control: max-age=1". That's rarely a good idea.
- cadr 9y agoYeah, it alternates loading telling me everything is fine, and just giving me a nginx 500 error. Seems that the status page should be hosted differently so it can be up even when other things are having issues.
- deleted 9y ago[deleted]
- traviscj 9y agoOn the other hand, makes it sound more likely to be a routing/reverse proxy issue instead of (say) a database issue. Those sound easier to deal with via a rollback vs something like "oops we dropped a critical index on the `messages` table".
- nyrikki 9y agoWith the length of the outage it seems more like a DB issue. And if this is still correct there look to be some fragile dependencies. https://aws.amazon.com/solutions/case-studies/slack/ https://aws.amazon.com/solutions/case-studies/slack/ The API isn't even returning valid error codes. logging error: {"subtype":"api_call_error","message":"{\"ok\":false,\"error\":\"_http_error\",\"status\":0,\"retry_after\":null}","stack":"Error\n As you can hit their api servers I am betting some replication error in mysql but that is just a guess based on that case study. I am betting they are saying 'connectivity' because that is the error the client logs.
- nyrikki 9y agoIt is back up for me, but now I am annoyed they can't follow specs at all. For l They ignore: https://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html https://standards.freedesktop.org/basedir-spec/basedir-spec-... And kitchen sink everything under the xdg config dir... ~$ ls .config/Slack/ Cache/ Cookies-journal dictionaries/ installation Local Storage/ Preferences QuotaManager-journal Cookies databases/ GPUCache/ local-settings.json logs/ QuotaManager storage/
- deleted 9y ago[deleted]
- seanp2k2 9y agoReturned for me when I was looking but took ~30 seconds to do so. They should host their status page on a separate domain and use something like CloudFlare in front of it to help with sudden spikes in traffic. Another alternative is to use Twitter / Facebook as the status page and let them deal with the traffic spikes, or just serve static HTML.
- msarchet 9y agonot anymore? - EDIT: the status page was fine and then it wasn't.
- yellowapple 9y agoSeems like every other request is throwing a 500 error. Maybe one server in a load-balanced cluster is erroring out?
- Narzerus 9y agoO.o god help us
- coleca 9y agoHow can I tell my team that Slack is down? :-)
- 5ilv3r 9y agopagerdoodie?
- nategri 9y agoYo, people who like to complain that Slack just re-implements IRC... This Is Your Moment
- ovrdrv3 9y agohttps://xkcd.com/927/ https://xkcd.com/927/
- ilovecomputers 9y agohttps://xkcd.com/1782/ https://xkcd.com/1782/
- Flammy 9y ago.
- auscompgeek 9y ago> Anyone who doesn't know, Slack actually runs off the IRC protocol underneath the hood. [citation needed]
- d4l3k 9y agoPretty sure they provide an IRC interface, but almost certainly don't use IRC internally. There's almost no way they could support any of their fancier features using IRC. Reactions etc would be horrible to implement.
- helper 9y agoYeah and the IRC interface has been getting worse recently. When they added the shared channels across teams they completely broke being able to '@' a user from the IRC gateway. Support said something along the lines of 'yup and we're not planning on fixing it'. I'm expecting them to completely turn off the IRC gateway in the next year or two.
- kiloreux 9y agoI just realized that being a remote team. I can't join any of my teammates. Neither see live changes to the infrastructure and the repositories. The things we take for granted.
- urda 9y agoIf slack being down means you lose all insight into your build process and code management, you seriously need to introduce a secondary option immediately.
- typomatic 9y agoOP didn't say "all insight". There's a difference between being unable to see a stream of change events and not being able to see the current state of the system. The latter is completely unacceptable, whereas the former is just annoying.
- Scriptor 9y agoIf you're a fully remote team it'd probably be wise to figure out some fallbacks.
- Johnny555 9y agoI'm sure they have fallbacks, but when their ecosystem (apparently) evolved around Slack, the fallbacks are less effective. Polling Jenkins to see when your job is done is more time consuming than receiving a Slack message.
- ytjohn 9y agoI'd like to message members of my team about this issue, but slack is down...
- vacri 9y agoEmail lives on.
- a12l 9y agoTeam: if you're reading this, get out the radios.
- BrentOzar 9y agoAnd of course today's the first day we're using Slack for audience Q&A at a conference. 360 folks in a room now have to...raise their hands! So barbaric.
- oplav 9y agoHere's the incident page: https://status.slack.com/2017-10/8b0d4d44ea53726f https://status.slack.com/2017-10/8b0d4d44ea53726f
- eridius 9y agoHasn't Slack learned yet that you're supposed to host your status page on a different infrastructure?
- Johnny555 9y agoI think they did. The slack.com IP's are owned by AWS, while status.slack.com resolves to some Digital Ocean IP's.
- eridius 9y agoThen why did the Slack Status page have so many problems at the same time? Half the time loading it would give a 500 Internal Server Error, 45% of the time you'd get broken resources (images and/or CSS), and only 5% of loads would give you the full working page.
- Johnny555 9y agoMaybe because it's under a lot more load during an outage and they haven't upsized the status page infrastructure to handle their ever increasing user base.
- philwelch 9y agoDNS issues?
- detaro 9y agoMaybe they underestimated how much resources their status page needs during an actual outage.
- philfreo 9y agoReminder why your status page should be hosted in a very different way than your regular infrastructure... so you're way less likely to end up with issues on both at the same time. I like how statuspage.io even has metastatuspage.com in case their primary domain/DNS/TLD has issues.
- theboywho 9y agoReminder that you should check things first before commenting. Slack's status page is in a different infrastructure, somewhere using digital ocean while slack.com is using AWS.
- deleted 9y ago[deleted]
- cheerioty 9y agoThat's why there's StatusPage.io :)
- tooltalk 9y agoI think you meant to say statuspage.io, not statuspage.com
- internalfx 9y agoMy RocketChat server is running just fine ;P
- 5ilv3r 9y agoSo is my jabber server!
- internalfx 9y agoIs this getting flagged off of HN?
- gtdawg 9y agoLooks like it. When I first loaded HN 2 minutes after this outage started, the story was #2. Then I refreshed after 3 minutes and it wasn't on the front page at all. Used the search tool to find it and then upvoted.
- internalfx 9y agoLooks like somebody wants to keep this quiet. 65 points in 12 minutes is good enough to be #1.
- Arathorn 9y agoThis is probably preaching to the choir, but hosting your own FOSS chat is nowadays a very viable way to avoid being dependent on a centralised service like Slack. Your options include: * Riot.im / Matrix.org (decentralised global network; e2e encryption; open protocol) * Rocket.Chat (Meteor-based; focus on UX and feature) * MatterMost.com (clone of Slack UI; open core license) * Zulip.org (all about threads!) * ...or indeed IRC or XMPP. (disclaimer; I work on Matrix).
- artur_makly 9y agolove MatterMost as it is ITAR compliant
- Veratyr 9y ago"International Traffic in Arms Regulations"? How is that relevant to a chat app?
- arturmakly 9y agoif u work on military or DOD projects,u cant use most cloud platforms like slack. MM solves that compliance requirement
- hbvvvbgvv 9y agodoes that mean no Indians are allowed on it?
- acchow 9y agoIs there reason to believe self hosting will have better uptime?
- flukus 9y ago1. Self hosting doesn't have to operate at the scale of slack, so there's a whole slew of issues avoided. Pushing text messages around really isn't that difficult when you aren't serving millions of customers. 2. You can perform maintenance outside of office hours, with SaaS you don't get to decide when an upgrade (and potential outage) happens. I don't care about 99% uptime, I care about having 99% uptime while I'm working. 3. You can have backup services.
- chris_wot 9y agoCausing chaos at my workplace. We have Slack integrated into our incident management solution... very, very unfortunate.
- jnaulty 9y agoWe are aware of connectivity issues and are actively lnvestigating. 3:58 PM PDT・See in your timezone They spelled investigating with a lower-case 'l' :\ Does that bug anyone other than me?
- Waterluvian 9y agoIt doesn't bug me. But I really want to understand how that happens.
- bramen 9y agoJust a simple typo. The keys are pretty close together if your finger slips, and I imagine they have enough problems distracting them from proper spellchecking at the moment. :-)
- fiatpandas 9y agoWhen your hands are shaking from adrenaline because your pre-IPO company is suffering from a global outage, you might hit an l instead of an i.
- theossuary 9y agoNo worse feeling than typing out a status update without any idea of what's going on.
- alexasmyths 9y agoI worked for BlackBerry when outages started to become a thing. When your business motto is 'always on' - it's really, really bad to be 'off' - it's a deep transgression of the brand promise. BB was structured poorly for this - they didn't grasp the concept of multiple nodes of redundancy very well. (Easy in hindsight). But - we should all be impressed at how highly available Google, FB and some other brands are. That's impressive.
- orthoganol 9y agoHow is there nothing on the front page of HN, slack being out for almost an hour now? Given timeframe and upvotes, how is this not the top of HN?
- 5ilv3r 9y agoIt is now
- carlchenet 9y agoYou could use this time to read The Slack threat https://carlchenet.com/the-slack-threat/ https://carlchenet.com/the-slack-threat/
- SolaceQuantum 9y agoThis is certainly scary for the Slack devs... Happy Halloween!
- nouveau0 9y agoSpoopy day at Slack
- quadrant6 9y agoWow, talk about realising how much we rely on Slack.. suddenly I feel so disconnected and alone. https://www.customd.com/articles/65/slack-is-down https://www.customd.com/articles/65/slack-is-down
- NightMKoder 9y agoI'm hoping they publish a public post-mortem. Learning from this kind of outage is the best kind experience for engineering - though it's far better when only staging goes down and not prod.
- deleted 9y ago[deleted]
- losingthefight 9y agoLooks like they won't: https://twitter.com/SlackHQ/status/925586114152411137 https://twitter.com/SlackHQ/status/925586114152411137 We've no solid plans right now as we're focused on tidying things up internally, but will consider it. Thanks again for holding tight
- nikolay 9y agoNice! I wondered why my productivity suddenly doubled up!
- artur_makly 9y agothe loss of revenue.. is unquantifiable.
- partycoder 9y agoSlack has a nice market share, but also many competitors, many of them 100% ripoffs with the same features (to name a few, Attlassian HipChat and MS Teams... not to mention open source products). Slack has been experiencing service degradation often lately, so I would not be surprised if people start switching. In our team we already started looking for an alternative.
- bojo 9y agoHipChat came out long before Slack.
- partycoder 9y agoDid not know that, thanks for point it out.
- dbenhur 9y agoYeah and they've had service degradation and availability issues long before Slack also. ;-P
- c0smic 9y agoJust a spooky coincidence it's on Halloween?
- kamilszybalski 9y agoSlack is currently down and I've realized, for better or worse, what Slack has really done.. It's created an expectation for immediacy. I thought about sending my question to someone via email but then just thought, "I'll wait for Slack to be back up, it'll be faster anyhow".
- matchagaucho 9y agoMy first thought: "Slack is down? Better post to Slack and let the team know."
- jonkiddy 9y agoYou're not alone. I had the same thought.
- 4lch3m1st 9y agoI'm not really a Slack user, but isn't Keybase always an option?
- JshWright 9y agoIf by "always" you mean "as of a couple weeks ago, and with a small subset of the features".
- westoncb 9y agoMaybe Slack decided to give us all a break for Halloween :) Also curious: why is this so low on the front page? 250 points, posted an hour ago...
- jaredsohn 9y agoFYI, slack seems to be up again now (at least for me.)
- pkrumins 9y agoI've switched to using TwistApp (https://www.twistapp.com https://www.twistapp.com) with my team. Unlike Slack where you have channels where everyone talks about everything, TwistApp bases conversations around threads. Every problem that's being worked on has its own thread. Once it's completed, I close and archive the threads. Very effective for getting things done as every task is isolated in a separate thread and discussions don't overlap. Also read this post by Amir, the founder of TwistApp, "Why we're betting against real time messaging" - https://blog.doist.com/why-were-betting-against-real-time-team-messaging-521804a3da09 https://blog.doist.com/why-were-betting-against-real-time-te...
- _lce0 9y agoWhy CPU goes like crazy every time slack loses connectivity? You can easily test this by disconnecting from wifi. As soon as you're offline the fan starts spinning until you get your connection back.
- scott_karana 9y agoI'm pretty sure that "refresh" tries to bootstrap the whole world. Eg, it'll reload all JS assets in addition to just restarting WebSockets