10 ms·

Slack was experiencing an outage

94 points by tjwds 5y ago

deleted 5y ago
[deleted]
VWWHFSfQ 5y ago
I don't mind Slack outages at all.
- benburleson 5y ago
  I'd rather have Zoom outages, actually.
  - deleted 5y ago
    [deleted]
- gharman 5y ago
  Yeah my first thought was “if only...”. Sadly, it’s still working just fine for me.
dijit 5y ago
"Hey guys, remember when _all of IRC_ went down?" Obviously this sucks and I feel bad for all of those affected, especially the people fixing this or who depended on Slack for their workflow. But it's hard to overstate the dangers of over-centralisation like this, and I say it as a person who uses Slack professionally. Maybe running your own Zulip instance isn't as sexy or has the same integrations, but at least you can have a person responsible for fixing it and get status updates, and ultimately: as much as Ops is a dirty word; being able to plan your downtimes can help a _lot_.
- pavel_lishin 5y ago
  > "Hey guys, remember when _all of IRC_ went down?" I certainly remember downtimes, and netsplits bad enough that it would have made team collaboration impossible depending on who was connected to what. I think the lesson here isn't "run and maintain your own version of X", but "make sure you have other ways to communicate besides X".
  - dijit 5y ago
    I think that's a fair conclusion honestly, though the cynic in me does want to point out that people do integrate their work against slack and that is difficult to maintain in parallel. Netsplits are mostly a solved problem these days, both with increased reliability of the internet and with network failure proxies like RobustIRC (for the linking protocol); and I remember fondly the times where we had a single IRC instance for all of our company (which had people working across: Thailand, USA and N.Europe) without issue. But, yeah, I think what you said is completely fair.
    - pavel_lishin 5y ago
      > though the cynic in me does want to point out that people do integrate their work against slack and that is difficult to maintain in parallel. Tell me about it! Two minutes after I posted that comment, I realized that our deploy system relies heavily on Slack.
      - Hamuko 5y ago
        >I realized that our deploy system relies heavily on Slack. How? The only deployment integration we have for Slack is that we get a message when a deployment start and when a deployment ends.
        Spivak 5y ago
        @deploybot do_release v2.3.45 I mean I've seen weirder. For Slack specifically I had a fight with a dev team because their deploy tool failed if it couldn't push a message to Slack.
  - u801e 5y ago
    I can't think of a time where the internal IRC server where I work went down. Netsplits shouldn't be an issue for that particular use case.
  - outworlder 5y ago
    > I certainly remember downtimes, and netsplits bad enough that it would have made team collaboration impossible depending on who was connected to what. But it certainly wasn't all the entire IRC network(s). Which is the parent's point. Slack is centralized. It was a common practice back then to have one "server". You would be hard-pressed to find mentions of "high availability" as servers were a very scarce resource. Small networks sometimes repurposed existing machines that were serving other workloads and added the IRC daemon.
- blacktriangle 5y ago
  Same goes for GitHub. We took the most trivially distributed and self-hosted technology ever written and converted it into a single point of failure. We software developers as a profession seem to have cultivated an almost active distain for anything resembling long-term thinking.
  - tinomaxgalvin 5y ago
    Think this is over the top Worst case put git repo on LAN.
    - wumpus 5y ago
      That gets access to the repo, but not issues, actions, code reviews, etc etc.
      - JadeNB 5y ago
        Once you take all this into account: > That gets access to the repo, but not issues, actions, code reviews, etc etc. (to the extent that it's external to git, e.g., housed in GitHub rather than the repository), I think we're no longer talking, as your grandparent was, about: > the most trivially distributed and self-hosted technology ever written ….
        wumpus 5y ago
        I rarely run into projects using just git these days, and I don't seem to have confused anyone by slightly shifting the topic.
  - taude 5y ago
    Trying to figure out where a github outage actually really impacted us in any meaningful long-term manner in the past five yeears. maybe we couldn't ship code for a couple hours, but all those PRs still did get out in an eventual timely fashion? I suppose there might be some people who had a service outage at the same time and their restrictive pipelines could only take tweaks via github, which would suck, but I'm guessing a majority of people, it's not worth hosting and maintaining all this infra yourself?
- asurne 5y ago
  My previous company had IRC as a backup in case their primary form of communication goes down. You get the benefits of advanced communication systems with the ability to continue to work in an outage, and while it would be nice for everyone to have IRC, only ops folks really need it to sustain operation.
- caturopath 5y ago
  I agree that centralization is a real risk, but I don't buy for a minute that your self-hosted Zulip instance is apt to have lower unplanned downtime than Slack.
  - dijit 5y ago
    Why? My central IRC node at a previous job had a better uptime than slack, and my previous company Gitlab server has lower unplanned downtime than Github... The key is that when I do my maintenances (on gitlab, for example) I can take the thing offline to do my work, which makes it much easier than trying to canary a change. It means I can plan a rollback or keep the backup in place. It's not that I'd have a higher uptime than slack really (though, I have experienced this) it's that the downtime works for the company and if it ever did fail we have the knowledge to fix it on our own schedule. I know that's scary to some, but it's a comfort to others.
  - iso1631 5y ago
    4 months ago https://news.ycombinator.com/item?id=25632346 https://news.ycombinator.com/item?id=25632346 7 months ago https://news.ycombinator.com/item?id=24800558 https://news.ycombinator.com/item?id=24800558 8 months ago https://news.ycombinator.com/item?id=24269767 https://news.ycombinator.com/item?id=24269767 12 months ago https://news.ycombinator.com/item?id=23161623 https://news.ycombinator.com/item?id=23161623 I don't specifically know about zulip, but I've got many internal servers and services with uptimes far in excess of that Here's one, not exactly a lightly loaded server 19:10:57 up 1511 days, 7:18, 1 user, load average: 45.70, 41.32, 38.48
    - jrockway 5y ago
      What practices do you use to ensure higher availability than Slack? Hope is not a strategy. (Answering my own question, I'd guess that the strategy is "never change anything". To some extent, I don't need 80% of the features that Slack offers, and code that you don't have is code that can't break. Not restarting stuff to release new code is also good for uptime. Would be interesting to see a SaaS that has a super-stable-never-change-anything release track, hosted on infra with similar goals, and see how that uptime compares to people's homegrown version of the same strategy.)
- MichaelApproved 5y ago
  > at least you can have a person responsible for fixing it At what cost? Even with the occasional outages, Slack is still more cost effective than having your own person (or team for larger organizations) handle your communication software.
  - dijit 5y ago
    I guess that depends? The company I mentioned in another thread had a single IRC node that basically never died. For 8 years (the life of the company) that machine sat by and did it's job without problems. Was it patched? No, and that's annoying, but that cost literally _nothing_ to the company (except a sip of power). Zulip isn't bad, it definitely doesn't need a whole person to manage it, probably 1hr/mo is good enough. How much you pay for an hour (or even 10!) a month of engineering time is almost certainly less than you'd be paying for Slack if you're more than 100 people; especially at their single-sign-on provider pricing.
    - MichaelApproved 5y ago
      You: “Was it patched? No” Also you: “but that cost literally _nothing_ to the company.” Risk is an expense.
  - rootusrootus 5y ago
    Not only that, but there's no guarantee that a local ops team would actually be any more successful, on average. I'm begging our IT folks to give up on hosting Jira ourselves because they can't make it work well. Random outages, misbehavior, and at the best of times it is sloooooooow. Just use the hosted version and let someone else scale the infrastructure to support it. (or better yet, don't use Atlassian, but that argument's a non-starter at my company).
    - krisoft 5y ago
      Give your IT folks some slack. (hah) We use the hosted jira and it is slow and has random misbehaviours too. Altough can’t remember any complete outages.
- markwillis82 5y ago
  We’ve ran our own mattermost server for almost 5 years. Sits alongside our hosting environment and not had any outages, except maintenance. Some staff moan it’s a “poor mans slack” but it works well, keeps everything in our platform and virtually free. So I ignore their groans
  - it33 5y ago
    Mattermost CEO here, thanks for the share out! There's some organizations that also use us for security, e.g. https://www.youtube.com/watch?v=cYl1SXFV_nE&feature=youtu.be https://www.youtube.com/watch?v=cYl1SXFV_nE&feature=youtu.be, though we're happy to have y'all using us for any reason :)
- vlunkr 5y ago
  It might be hard, but I think you've overstated the dangers of overcentralizing here. If slack is down for two hours, what's the worst that could happen? Everyone works on something on their own and sends an email or uses one of 50 video chat apps if they need to talk. I think occasional downtime is expected and acceptable.
- renewiltord 5y ago
  Actually, the synchronized downtime means that everyone is down at the same time so that's way more preferable to having your own independent downtime. There's no coordination loss because everyone is explicitly off-grid for this period. The common signal is advantageous. And if you're a vendor, then your customers are sympathetic if they're also suffering at the same time.
- tshaddox 5y ago
  Will your self-hosted solution have better or worse downtime than Slack? Surely that’s the better question than whether something like this is centralized or not.
- phailhaus 5y ago
  > but at least you can have a person responsible for fixing it and get status updates Er, what? With Slack, there is an _entire organization_ responsible for fixing it and giving out status updates.
- mbesto 5y ago
  > But it's hard to overstate the dangers of over-centralisation like this, and I say it as a person who uses Slack professionally. I think you're overstating the importance of a chat based application. It's not mission critical to operate a software business or any business for that matter so why is this so "dangerous"? > Maybe running your own Zulip instance isn't as sexy or has the same integrations, but at least you can have a person responsible for fixing it and get status updates, and ultimately: as much as Ops is a dirty word; being able to plan your downtimes can help a _lot_. Let's play the devil's advocate and say it IS mission critical. Now you need to pay full time staff (read -> not cheap) and computing resources and integration development (depending on solution) for it to be useful AND guarantee uptime. I don't understand what the big fuss is about Slack vs IRC. Pretending as if every business in the world has access to competent sys admin folks is not only foolish, but it's dangerous to assume.
  - chipgap98 5y ago
    > I think you're overstating the importance of a chat based application. It's not mission critical to operate a software business or any business for that matter so why is this so "dangerous"? I feel like the truth is somewhere in the middle here. In an increasingly remote world and with the many ways slack integrates into businesses I think it is incredibly important to some businesses and maybe even mission critical. But I totally agree with your second point that just because chat is important doesn't make running your own IRC instance a good decision or resource investment.
- rhuber 5y ago
  "Hey guys, remember when freenode was taken over by one guy?" (It was, like, yesterday.) It's hard to overstate the dangers of over-centralisation like this, and I say it as a person who used freenode professionally. https://www.kline.sh/ https://www.kline.sh/
  - nexuist 5y ago
    ...and the moderators who resigned immediately set up their own network under their own terms (libera.chat), so I'm not really sure what your point is? The difference is that you can jump to another server with an IRC client, you can't jump to another Slack with the Slack client.
- polote 5y ago
  > Maybe running your own Zulip instance isn't as sexy or has the same integrations, but at least you can have a person responsible for fixing it and get status updates, and ultimately Let's replace something that is almost never down don't require any maintenance (Slack) with something that requires maintenance and will probably go down more frequently than Slack. I hardly see any benefit in term of availability going on Zulip
  - iso1631 5y ago
    > Let's replace something that is almost never down don't require any maintenance (Slack) Slack is constantly down -- every few months it reaches HN as a "slack is down" story.
  - pritambaral 5y ago
    > that requires maintenance and will probably go down more frequently than Slack Been running Zulip for two years at two companies. Never gone down, not even during maintenance (zero-downtime reloads FTW), which was twice, for version upgrades. > don't require any maintenance (Slack) Neither does zulip.com, which is a managed service by the makers and maintainers of Zulip and costs the same as Slack.
- deleted 5y ago
  [deleted]
- throwaway894345 5y ago
  I’m a lowly SRE, and I’m not technical enough to use IRC effectively despite on and off attempts spanning more than a decade. Believe it or not, I’m not the least technical person in our org! This, I think, is the biggest reason my company, and presumably many others, don’t use IRC, never mind that running your own IRC or Zulip or whatever doesn’t inherently gain you any extra uptime and in all likelihood costs you more uptime (who is going to be better able to support your application, your Ops team with dozens of other critical applications to worry about and no inherent expertise with the application, or the dedicated team with deep expertise and a reporting structure all the way up to the CEO who are singularly aligned with the goal of running this application? Note: this argument tends to apply to any managed-service vs self-host debate. Note that we wouldn’t have a debate at all if every Zulip/IRC/etc outage hit the front page of HN. :)
- randomsearch 5y ago
  I also recommend building your own power station, ISP, and power station so you’re not reliant on centralised services.
  - iso1631 5y ago
    Our data centres have two power feeds, UPS on one of the legs, and backup diesel generators. Network is resilient via fibres in different directions. My laptop has a backup UPS built in. If wifi breaks, I can tether off my phone. Power, ISP, UPS and Generators have all failed, but none are as ropey as slack (At least in the west - we had power issues in our Kabul officer earlier today, although even then slack has more outages than that)
  - dijit 5y ago
    If we’re going to extremes: why not outsource having a CEO? I mean. A company that gives you a managing director or chief executive can do so with far more availability than just one guy! But really. Your statement is absurdist, critical infrastructure doesn’t have to be outsourced. It’s a choice and there are trade offs.
- mdoms 5y ago
  Why would you care if "all of IRC" went down when you and your colleagues depend on one particular server?
dogecoinbase 5y ago
I recognize the ways in which this is a difficult problem, but it remains frustrating to me that the application isn't capable of providing this information directly to end users. Instead you end up with confusion, missed communication, and occasionally people thinking they've been fired! Surely we can do better.
- tekstar 5y ago
  Long time ago, the day I finished my probation period at work was also a multi-hour gmail outage. I got up, couldn't load my email, and had a panic attack for a few minutes before finding out what was going on
- harikb 5y ago
  I am sad to see that our profession hasn't figured out how to make a decent desktop app with local-storage (that continues working at least in minimal fashion when the server is out of commission). Slack, Teams, they are all similar. Many times I open the Teams window back from a coffee break and I get a blank page. The software can't even detect that it is a blank page being displayed and some progress ball / beach ball whatever. I say this in the middle of attempting a Tauri desktop app. I am sure my app will have the same problems, but my other options are even worse.
misnome 5y ago
> Reloading Slack (Command + R / Ctrl + R) may help Slack to load as expected. Well, it went from readable to "Server Error something went wrong". I haven't seen the entire slack app 500 before!
- boiler_up800 5y ago
  Same thing happened to me. Times New Roman font no less.
  - luke2m 5y ago
    It is electron...
    - deleted 5y ago
      [deleted]
- wicked_gazelle 5y ago
  Slack was working for me this morning (desktop and mobile), albeit a bit slow, but once I hit Command + R, it gave me "Server Error something went wrong" too. It logged me out of my work's workspace from both desktop and mobile device so I can't communicate with my team. Don't press Command + R/Ctrl + R, if Slack is working for you!
deleted 5y ago
[deleted]
deleted 5y ago
[deleted]
readonthegoapp 5y ago
I wonder if any companies have experimented with intentional occasional slack 'outages'? Maybe 1-3pm daily? I've seen some of the Gitlab Gospels that describe how to not use slack or whatever messaging tool they use in excruciating detail, and it made me think... ...if it requires this much instruction on how _not_ to use it, then maybe there is a tool that is a bit more intuitive.
- novok 5y ago
  We teach children how to use their voices all the time (ex: "indoor voice"), rules of social interaction and so on. Beyond a certain point, you need social rules for social creatures.
- btown 5y ago
  Shutting down messaging as a cultural break would be fine... if Slack wasn't also the searchable knowledge base of record for nuanced specifications, customizations, and relationships with business partners. When it becomes that, it becomes critical infrastructure!
sergiotapia 5y ago
We have a Discord server as backup just in case. Just added 15 engineers quite seamlessly, I recommend it for teams
- ev1 5y ago
  In a corporate environment I'd get absolutely brutalised by lawyers for doing this. The amount of invasive data collection, fingerprinting, phone number requirements, permanent plaintext logging of messages, file scanning, process scanning and upload...
  - sergiotapia 5y ago
    We don't share any PII or PHI whatsoever on Discord so it's been ok'd for emergency use.
  - synicalx 5y ago
    You reckon Slack and Teams don't do any of that?
- Diederich 5y ago
  We use keybase chat as a backup; works pretty well.
LegitGandalf 5y ago
I started using slack about two years ago, and in my opinion they have a quality problem. There are just too many issues that crop up when they roll out change. It reeks of the "just get it done" anti-pattern where done is change that hasn't had the chaos tracked down and killed. I bet internally they are waiting for bugs to get reported instead of pro-actively running the changed software under representative load to hunt and kill chaos.
slackerIII 5y ago
Interesting! I co-host a podcast about notable internet outages, and our very episode was about Slack's Jan 1 outage: https://downtimeproject.com/podcast/episode-1-slack-vs-tgws/ https://downtimeproject.com/podcast/episode-1-slack-vs-tgws/ If they publish a good postmortem on this, we'll definitely do an episode about it.
- jamwt 5y ago
  Heh, and the last time the podcast published an episode about an Auth0 outage, Auth0 had another outage the next day. Now Slack follows shortly after an episode. How eerie!
- jcims 5y ago
  Hey I just want to toss out kudos on your podcast. I found it through someone linking to it on a recent HN story and have really enjoyed catching up in the catalog. Super cool idea and really beneficial. One website suggestion, add your transcripts to your search index.
  - slackerIII 5y ago
    Oh dang, I totally missed that. Great idea, thank you!
bkircher 5y ago
Productivity goes up 130%
sys_64738 5y ago
Slack seems superfluous now that MS Teams is consuming everything before it.
- fractalf 5y ago
  > Slack seems superfluous now that MS Teams is consuming everything before it. MS Teams suck so bad from a developer perspecrive compared to slack it's never going to be a proper alternative. I'm not dissing, this is just my experience using both
  - sys_64738 5y ago
    They’re both electron based so both have issues. Must say I prefer Teams.

94 points by tjwds 5y ago

deleted 5y ago

[deleted]

VWWHFSfQ 5y ago

I don't mind Slack outages at all.

benburleson 5y ago
I'd rather have Zoom outages, actually.
- deleted 5y ago
  [deleted]
gharman 5y ago
Yeah my first thought was “if only...”. Sadly, it’s still working just fine for me.

dijit 5y ago

"Hey guys, remember when _all of IRC_ went down?" Obviously this sucks and I feel bad for all of those affected, especially the people fixing this or who depended on Slack for their workflow. But it's hard to overstate the dangers of over-centralisation like this, and I say it as a person who uses Slack professionally. Maybe running your own Zulip instance isn't as sexy or has the same integrations, but at least you can have a person responsible for fixing it and get status updates, and ultimately: as much as Ops is a dirty word; being able to plan your downtimes can help a _lot_.

pavel_lishin 5y ago
> "Hey guys, remember when _all of IRC_ went down?" I certainly remember downtimes, and netsplits bad enough that it would have made team collaboration impossible depending on who was connected to what. I think the lesson here isn't "run and maintain your own version of X", but "make sure you have other ways to communicate besides X".
- dijit 5y ago
  I think that's a fair conclusion honestly, though the cynic in me does want to point out that people do integrate their work against slack and that is difficult to maintain in parallel. Netsplits are mostly a solved problem these days, both with increased reliability of the internet and with network failure proxies like RobustIRC (for the linking protocol); and I remember fondly the times where we had a single IRC instance for all of our company (which had people working across: Thailand, USA and N.Europe) without issue. But, yeah, I think what you said is completely fair.
  - pavel_lishin 5y ago
    > though the cynic in me does want to point out that people do integrate their work against slack and that is difficult to maintain in parallel. Tell me about it! Two minutes after I posted that comment, I realized that our deploy system relies heavily on Slack.
    - Hamuko 5y ago
      >I realized that our deploy system relies heavily on Slack. How? The only deployment integration we have for Slack is that we get a message when a deployment start and when a deployment ends.
      - Spivak 5y ago
        @deploybot do_release v2.3.45 I mean I've seen weirder. For Slack specifically I had a fight with a dev team because their deploy tool failed if it couldn't push a message to Slack.
- u801e 5y ago
  I can't think of a time where the internal IRC server where I work went down. Netsplits shouldn't be an issue for that particular use case.
- outworlder 5y ago
  > I certainly remember downtimes, and netsplits bad enough that it would have made team collaboration impossible depending on who was connected to what. But it certainly wasn't all the entire IRC network(s). Which is the parent's point. Slack is centralized. It was a common practice back then to have one "server". You would be hard-pressed to find mentions of "high availability" as servers were a very scarce resource. Small networks sometimes repurposed existing machines that were serving other workloads and added the IRC daemon.
blacktriangle 5y ago
Same goes for GitHub. We took the most trivially distributed and self-hosted technology ever written and converted it into a single point of failure. We software developers as a profession seem to have cultivated an almost active distain for anything resembling long-term thinking.
- tinomaxgalvin 5y ago
  Think this is over the top Worst case put git repo on LAN.
  - wumpus 5y ago
    That gets access to the repo, but not issues, actions, code reviews, etc etc.
    - JadeNB 5y ago
      Once you take all this into account: > That gets access to the repo, but not issues, actions, code reviews, etc etc. (to the extent that it's external to git, e.g., housed in GitHub rather than the repository), I think we're no longer talking, as your grandparent was, about: > the most trivially distributed and self-hosted technology ever written ….
      - wumpus 5y ago
        I rarely run into projects using just git these days, and I don't seem to have confused anyone by slightly shifting the topic.
- taude 5y ago
  Trying to figure out where a github outage actually really impacted us in any meaningful long-term manner in the past five yeears. maybe we couldn't ship code for a couple hours, but all those PRs still did get out in an eventual timely fashion? I suppose there might be some people who had a service outage at the same time and their restrictive pipelines could only take tweaks via github, which would suck, but I'm guessing a majority of people, it's not worth hosting and maintaining all this infra yourself?
asurne 5y ago
My previous company had IRC as a backup in case their primary form of communication goes down. You get the benefits of advanced communication systems with the ability to continue to work in an outage, and while it would be nice for everyone to have IRC, only ops folks really need it to sustain operation.
caturopath 5y ago
I agree that centralization is a real risk, but I don't buy for a minute that your self-hosted Zulip instance is apt to have lower unplanned downtime than Slack.
- dijit 5y ago
  Why? My central IRC node at a previous job had a better uptime than slack, and my previous company Gitlab server has lower unplanned downtime than Github... The key is that when I do my maintenances (on gitlab, for example) I can take the thing offline to do my work, which makes it much easier than trying to canary a change. It means I can plan a rollback or keep the backup in place. It's not that I'd have a higher uptime than slack really (though, I have experienced this) it's that the downtime works for the company and if it ever did fail we have the knowledge to fix it on our own schedule. I know that's scary to some, but it's a comfort to others.
- iso1631 5y ago
  4 months ago https://news.ycombinator.com/item?id=25632346 https://news.ycombinator.com/item?id=25632346 7 months ago https://news.ycombinator.com/item?id=24800558 https://news.ycombinator.com/item?id=24800558 8 months ago https://news.ycombinator.com/item?id=24269767 https://news.ycombinator.com/item?id=24269767 12 months ago https://news.ycombinator.com/item?id=23161623 https://news.ycombinator.com/item?id=23161623 I don't specifically know about zulip, but I've got many internal servers and services with uptimes far in excess of that Here's one, not exactly a lightly loaded server 19:10:57 up 1511 days, 7:18, 1 user, load average: 45.70, 41.32, 38.48
  - jrockway 5y ago
    What practices do you use to ensure higher availability than Slack? Hope is not a strategy. (Answering my own question, I'd guess that the strategy is "never change anything". To some extent, I don't need 80% of the features that Slack offers, and code that you don't have is code that can't break. Not restarting stuff to release new code is also good for uptime. Would be interesting to see a SaaS that has a super-stable-never-change-anything release track, hosted on infra with similar goals, and see how that uptime compares to people's homegrown version of the same strategy.)
MichaelApproved 5y ago
> at least you can have a person responsible for fixing it At what cost? Even with the occasional outages, Slack is still more cost effective than having your own person (or team for larger organizations) handle your communication software.
- dijit 5y ago
  I guess that depends? The company I mentioned in another thread had a single IRC node that basically never died. For 8 years (the life of the company) that machine sat by and did it's job without problems. Was it patched? No, and that's annoying, but that cost literally _nothing_ to the company (except a sip of power). Zulip isn't bad, it definitely doesn't need a whole person to manage it, probably 1hr/mo is good enough. How much you pay for an hour (or even 10!) a month of engineering time is almost certainly less than you'd be paying for Slack if you're more than 100 people; especially at their single-sign-on provider pricing.
  - MichaelApproved 5y ago
    You: “Was it patched? No” Also you: “but that cost literally _nothing_ to the company.” Risk is an expense.
- rootusrootus 5y ago
  Not only that, but there's no guarantee that a local ops team would actually be any more successful, on average. I'm begging our IT folks to give up on hosting Jira ourselves because they can't make it work well. Random outages, misbehavior, and at the best of times it is sloooooooow. Just use the hosted version and let someone else scale the infrastructure to support it. (or better yet, don't use Atlassian, but that argument's a non-starter at my company).
  - krisoft 5y ago
    Give your IT folks some slack. (hah) We use the hosted jira and it is slow and has random misbehaviours too. Altough can’t remember any complete outages.
markwillis82 5y ago
We’ve ran our own mattermost server for almost 5 years. Sits alongside our hosting environment and not had any outages, except maintenance. Some staff moan it’s a “poor mans slack” but it works well, keeps everything in our platform and virtually free. So I ignore their groans
- it33 5y ago
  Mattermost CEO here, thanks for the share out! There's some organizations that also use us for security, e.g. https://www.youtube.com/watch?v=cYl1SXFV_nE&feature=youtu.be https://www.youtube.com/watch?v=cYl1SXFV_nE&feature=youtu.be, though we're happy to have y'all using us for any reason :)
vlunkr 5y ago
It might be hard, but I think you've overstated the dangers of overcentralizing here. If slack is down for two hours, what's the worst that could happen? Everyone works on something on their own and sends an email or uses one of 50 video chat apps if they need to talk. I think occasional downtime is expected and acceptable.
renewiltord 5y ago
Actually, the synchronized downtime means that everyone is down at the same time so that's way more preferable to having your own independent downtime. There's no coordination loss because everyone is explicitly off-grid for this period. The common signal is advantageous. And if you're a vendor, then your customers are sympathetic if they're also suffering at the same time.
tshaddox 5y ago
Will your self-hosted solution have better or worse downtime than Slack? Surely that’s the better question than whether something like this is centralized or not.
phailhaus 5y ago
> but at least you can have a person responsible for fixing it and get status updates Er, what? With Slack, there is an _entire organization_ responsible for fixing it and giving out status updates.
mbesto 5y ago
> But it's hard to overstate the dangers of over-centralisation like this, and I say it as a person who uses Slack professionally. I think you're overstating the importance of a chat based application. It's not mission critical to operate a software business or any business for that matter so why is this so "dangerous"? > Maybe running your own Zulip instance isn't as sexy or has the same integrations, but at least you can have a person responsible for fixing it and get status updates, and ultimately: as much as Ops is a dirty word; being able to plan your downtimes can help a _lot_. Let's play the devil's advocate and say it IS mission critical. Now you need to pay full time staff (read -> not cheap) and computing resources and integration development (depending on solution) for it to be useful AND guarantee uptime. I don't understand what the big fuss is about Slack vs IRC. Pretending as if every business in the world has access to competent sys admin folks is not only foolish, but it's dangerous to assume.
- chipgap98 5y ago
  > I think you're overstating the importance of a chat based application. It's not mission critical to operate a software business or any business for that matter so why is this so "dangerous"? I feel like the truth is somewhere in the middle here. In an increasingly remote world and with the many ways slack integrates into businesses I think it is incredibly important to some businesses and maybe even mission critical. But I totally agree with your second point that just because chat is important doesn't make running your own IRC instance a good decision or resource investment.
rhuber 5y ago
"Hey guys, remember when freenode was taken over by one guy?" (It was, like, yesterday.) It's hard to overstate the dangers of over-centralisation like this, and I say it as a person who used freenode professionally. https://www.kline.sh/ https://www.kline.sh/
- nexuist 5y ago
  ...and the moderators who resigned immediately set up their own network under their own terms (libera.chat), so I'm not really sure what your point is? The difference is that you can jump to another server with an IRC client, you can't jump to another Slack with the Slack client.
polote 5y ago
> Maybe running your own Zulip instance isn't as sexy or has the same integrations, but at least you can have a person responsible for fixing it and get status updates, and ultimately Let's replace something that is almost never down don't require any maintenance (Slack) with something that requires maintenance and will probably go down more frequently than Slack. I hardly see any benefit in term of availability going on Zulip
- iso1631 5y ago
  > Let's replace something that is almost never down don't require any maintenance (Slack) Slack is constantly down -- every few months it reaches HN as a "slack is down" story.
- pritambaral 5y ago
  > that requires maintenance and will probably go down more frequently than Slack Been running Zulip for two years at two companies. Never gone down, not even during maintenance (zero-downtime reloads FTW), which was twice, for version upgrades. > don't require any maintenance (Slack) Neither does zulip.com, which is a managed service by the makers and maintainers of Zulip and costs the same as Slack.
deleted 5y ago
[deleted]
throwaway894345 5y ago
I’m a lowly SRE, and I’m not technical enough to use IRC effectively despite on and off attempts spanning more than a decade. Believe it or not, I’m not the least technical person in our org! This, I think, is the biggest reason my company, and presumably many others, don’t use IRC, never mind that running your own IRC or Zulip or whatever doesn’t inherently gain you any extra uptime and in all likelihood costs you more uptime (who is going to be better able to support your application, your Ops team with dozens of other critical applications to worry about and no inherent expertise with the application, or the dedicated team with deep expertise and a reporting structure all the way up to the CEO who are singularly aligned with the goal of running this application? Note: this argument tends to apply to any managed-service vs self-host debate. Note that we wouldn’t have a debate at all if every Zulip/IRC/etc outage hit the front page of HN. :)
randomsearch 5y ago
I also recommend building your own power station, ISP, and power station so you’re not reliant on centralised services.
- iso1631 5y ago
  Our data centres have two power feeds, UPS on one of the legs, and backup diesel generators. Network is resilient via fibres in different directions. My laptop has a backup UPS built in. If wifi breaks, I can tether off my phone. Power, ISP, UPS and Generators have all failed, but none are as ropey as slack (At least in the west - we had power issues in our Kabul officer earlier today, although even then slack has more outages than that)
- dijit 5y ago
  If we’re going to extremes: why not outsource having a CEO? I mean. A company that gives you a managing director or chief executive can do so with far more availability than just one guy! But really. Your statement is absurdist, critical infrastructure doesn’t have to be outsourced. It’s a choice and there are trade offs.
mdoms 5y ago
Why would you care if "all of IRC" went down when you and your colleagues depend on one particular server?

dogecoinbase 5y ago

I recognize the ways in which this is a difficult problem, but it remains frustrating to me that the application isn't capable of providing this information directly to end users. Instead you end up with confusion, missed communication, and occasionally people thinking they've been fired! Surely we can do better.

tekstar 5y ago
Long time ago, the day I finished my probation period at work was also a multi-hour gmail outage. I got up, couldn't load my email, and had a panic attack for a few minutes before finding out what was going on
harikb 5y ago
I am sad to see that our profession hasn't figured out how to make a decent desktop app with local-storage (that continues working at least in minimal fashion when the server is out of commission). Slack, Teams, they are all similar. Many times I open the Teams window back from a coffee break and I get a blank page. The software can't even detect that it is a blank page being displayed and some progress ball / beach ball whatever. I say this in the middle of attempting a Tauri desktop app. I am sure my app will have the same problems, but my other options are even worse.

misnome 5y ago

> Reloading Slack (Command + R / Ctrl + R) may help Slack to load as expected. Well, it went from readable to "Server Error something went wrong". I haven't seen the entire slack app 500 before!

boiler_up800 5y ago
Same thing happened to me. Times New Roman font no less.
- luke2m 5y ago
  It is electron...
  - deleted 5y ago
    [deleted]
wicked_gazelle 5y ago
Slack was working for me this morning (desktop and mobile), albeit a bit slow, but once I hit Command + R, it gave me "Server Error something went wrong" too. It logged me out of my work's workspace from both desktop and mobile device so I can't communicate with my team. Don't press Command + R/Ctrl + R, if Slack is working for you!

deleted 5y ago

[deleted]

deleted 5y ago

[deleted]

readonthegoapp 5y ago

I wonder if any companies have experimented with intentional occasional slack 'outages'? Maybe 1-3pm daily? I've seen some of the Gitlab Gospels that describe how to not use slack or whatever messaging tool they use in excruciating detail, and it made me think... ...if it requires this much instruction on how _not_ to use it, then maybe there is a tool that is a bit more intuitive.

novok 5y ago
We teach children how to use their voices all the time (ex: "indoor voice"), rules of social interaction and so on. Beyond a certain point, you need social rules for social creatures.
btown 5y ago
Shutting down messaging as a cultural break would be fine... if Slack wasn't also the searchable knowledge base of record for nuanced specifications, customizations, and relationships with business partners. When it becomes that, it becomes critical infrastructure!

sergiotapia 5y ago

We have a Discord server as backup just in case. Just added 15 engineers quite seamlessly, I recommend it for teams

ev1 5y ago
In a corporate environment I'd get absolutely brutalised by lawyers for doing this. The amount of invasive data collection, fingerprinting, phone number requirements, permanent plaintext logging of messages, file scanning, process scanning and upload...
- sergiotapia 5y ago
  We don't share any PII or PHI whatsoever on Discord so it's been ok'd for emergency use.
- synicalx 5y ago
  You reckon Slack and Teams don't do any of that?
Diederich 5y ago
We use keybase chat as a backup; works pretty well.

LegitGandalf 5y ago

I started using slack about two years ago, and in my opinion they have a quality problem. There are just too many issues that crop up when they roll out change. It reeks of the "just get it done" anti-pattern where done is change that hasn't had the chaos tracked down and killed. I bet internally they are waiting for bugs to get reported instead of pro-actively running the changed software under representative load to hunt and kill chaos.

slackerIII 5y ago

Interesting! I co-host a podcast about notable internet outages, and our very episode was about Slack's Jan 1 outage: https://downtimeproject.com/podcast/episode-1-slack-vs-tgws/ https://downtimeproject.com/podcast/episode-1-slack-vs-tgws/ If they publish a good postmortem on this, we'll definitely do an episode about it.

jamwt 5y ago
Heh, and the last time the podcast published an episode about an Auth0 outage, Auth0 had another outage the next day. Now Slack follows shortly after an episode. How eerie!
jcims 5y ago
Hey I just want to toss out kudos on your podcast. I found it through someone linking to it on a recent HN story and have really enjoyed catching up in the catalog. Super cool idea and really beneficial. One website suggestion, add your transcripts to your search index.
- slackerIII 5y ago
  Oh dang, I totally missed that. Great idea, thank you!

bkircher 5y ago

Productivity goes up 130%

sys_64738 5y ago

Slack seems superfluous now that MS Teams is consuming everything before it.

fractalf 5y ago
> Slack seems superfluous now that MS Teams is consuming everything before it. MS Teams suck so bad from a developer perspecrive compared to slack it's never going to be a proper alternative. I'm not dissing, this is just my experience using both
- sys_64738 5y ago
  They’re both electron based so both have issues. Must say I prefer Teams.