6 ms·
In other news: an unexplained productivity spike has been recorded today across the tech industry.
by TelmoMenezes 7y ago
In other news: an unexplained productivity spike has been recorded today across the tech industry.
- darkwater 7y agoOr not. I'm remote and I cannot ask questions nor coordinate action to solve live production problems due to this outage. Slack is becoming a SPOF for many organizations, especially distributed.
- pmlnr 7y agoThis is part of the problem - we all need to ask too many questions. We should be working on pre-defined tasks most of the time, where it doesn't matter if it's waterfall pre-defined or scrum time allocated.
- ibero 7y agoThe commenter above said they had to "solve live production problems", how exactly would "pre-defined tasks" help them?
- pmlnr 7y agoI slipped over the "live" word, but I still stand by what I said. I most probably work in a similar environment, with tonnes of live production issues, and I believe it's a result of not being allowed to proactively plan for situations - most of the issues are a result of cost savings, systems driven to their absolute maximum capacity, etc. Live production issues are a byproduct of moving too fast, breaking too many things.
- wastedhours 7y agoI'd style my way out of that argument by saying out reliance on tools like Slack rather than other pre-planned work builds a reliance on that one tool for all tasks being done "in the moment", which then means it's relied on as the medium in live production issues. If there was a lower reliance on IM for all tasks, other solutions for crisis management would be utilised (potentially).
- dasil003 7y agoWhat an incredibly arrogant generalization to make. You know there are also production issues created by moving too slow and failing to address pending scalability issues, right?
- pmlnr 7y agoheh. > failing to address pending scalability issues How is that not related to not getting enough funding and resources? And yes, having scalability issues means moving too fast - the _business_ moving too fast.
- kaikai 7y agoThis is starting to sound like the business version of starting physics problems with “imagine a frictionless surface...”
- kkirsche 7y agoFor anyone else curious, SPOF means single point of failure in this context.
- winslett 7y agoI was thinking "single point of focus."
- LinuxBender 7y agoAt my workplace we were told SPOF was too negative, so one of the engineers changed it to SPOS. Single Point Of Success.
- majewsky 7y agoSame here. I'm on support duty and have my own outage to attend to, and that's really really hard when you cannot communicate with your experts.
- jt2190 7y agoForgive me, but just to clarify... There is no “fallback” communication channel for your team to use?
- wil421 7y agoThere’s email, phone, whats app, group SMS, some form of WebEx or Zoom, or even a shared google doc spreadsheet with tasks for each outage. If Slack is your only forum of communication with your team it’s time to rethink your support structure and DR plans. Edit: It’s very unhelpful if I point out issues and don’t provide solutions. PagerDuty has a great article about Incident Response[1]. [1]https://response.pagerduty.com/ https://response.pagerduty.com/
- wastedhours 7y agoI don't want to sound snarky by questioning what's probably an org mandated tool usage, but I can't quite believe companies where there's only one accepted way to get in contact. Ideal would be a live conference call (phone, Skype, Hangouts etc...), but you've got SMS, email, heck, I once solved a pressing issue with my team in a Sharepoint Word document where all our comments were just new paragraphs being edited live (which actually had the handy benefit of being savable without needing to pay extra for the privilege...).
- matwood 7y agoI know it's 'old school', but we would all just get on a conference call before the fancy chat tools.
- pennaMan 7y agoHaving a single communication medium is a SPOF no matter what name it has
- ratsimihah 7y agoIsn't internet a single communication medium anyway?
- dpcx 7y agoIt is, technically. Good thing there's also phones and SMS.
- pfortuny 7y agoBut it is routed. Also your laptop usually has 2 Network cards on different physical layers... Not the same.
- bluejekyll 7y agoThe internet is specifically designed to route around downed links such that there is no SPOF, that’s the theory anyway. However, when major backbones go down it takes it a while to recover.
- kiallmacinnes 7y agoWell, kinda! Just look at this recent CloudFlare blog post[1]. It's true, theres no Single Point of Failure - there are MANY points of failure ;) [1]: https://blog.cloudflare.com/how-verizon-and-a-bgp-optimizer-knocked-large-parts-of-the-internet-offline-today/ https://blog.cloudflare.com/how-verizon-and-a-bgp-optimizer-...
- StreamBright 7y agoThat is rather unfortunate if there is only a single channel that an organization can use in a production outage situation.
- darkwater 7y agoWell, there are other means of communication but they either: * do not fit well for a quick message interchange like you can have during an outage (email) * don't have the whole company (or at least the whole tech team) on it (whatsapp, telegram, you name it) * make compliated to share big chunks of text/logs (phone call) It probably makes sense creating another secondary - usually dormant - communication hub like a WhatsApp group for emergencies but people tend to misuse it (like using it to ping for some minor incident when all the other more suited communication tools are working perfectly). Anyway it wasn't a big outage on our systems, just a minor hiccup, but it's a lesson learnt nonetheless.
- swebs 7y agoWhy can't you use email?
- dx7tnt 7y agoBecause Slack is so much better than email!
- icedchai 7y agoGood thing email still exists, right?
- matchbok 7y agoDo phones exist?
- SquishyPanda23 7y agoIt makes sense to have a fallback method set ahead of time, as others have mentioned. You won't have access to the things slack integrates with but at least you can talk. I know some people use IRC as a low-dependency fallback channel.
- jdenoy 7y agowe use a mumble server as backup and parallel mean to discuss in the operations department... Sometimes talking all together using voice as if we were in the same office is much nicer than writing!
- LinuxBender 7y agoMumble / murmur [1] is great. I set one up for a family member in minutes. Super easy to deploy. People should use their RC releases. They are solid, sound quality is great and it scales really well. A small to medium VM can handle thousands of people. [1] - https://wiki.mumble.info/wiki/Main_Page https://wiki.mumble.info/wiki/Main_Page
- Twirrim 7y agoAt one stage when there was an outage about a year ago, folks at work were scrambling around trying to get HipChat resurrected (which we'd been using prior to moving to Slack). I had an ircd of some description installed, configured and ready for use in under 3 minutes, and started to get my team using it as a stop gap measure. Had that rolling long before either the outage was resolved, or the HipChat server was resurrected. People forget just how stupidly easy it is to set up IRC.
- cloverich 7y agoDo you have a recommended server / client? The last time I tried to use IRC (joining an existing chat) I actually had quite a bit of trouble, and ended up just giving up.
- 7y ago
- duxup 7y agoIt depends. I had a similar outage at another job for a day. Pre slack days, but instant messaging and email was down. I needed to do some things so I just stopped checking with people. So did other people. The result was I found was that we were double checking, coordinating and doing a lot of verification that honestly wasn't needed. The sky didn't fall, everyone still saw the changes and were ok with it. After that folks stopped doing a lot of the traditional coordination that proved to be superfluous and maybe never accomplished anything. The handful of mistakes that happened, were also easily caught / fixed. Granted, this requires people to make good choices.
- orwin 7y agoYeah, i understand. hosting your own mattermost (or other slack clone) is not enough for you? You can also have 2 self-hosted solution on different servers (one on our own, one on any CSP), in case your server fails.
- phtrivier 7y agoAccording to some prospective studies, some people would (and please stay with me on that) actually use slack to communicate about work-related matters with co-workers. More study are needed to clarify if this work-related communication might actually be required for work to get done. (Cal Newport is definitely preparing a book on that any time soon.) Unfortunately, the researchers are too busy maintaining their IRC server to actual do it - but at least they're using IRC. Whereas, my productivity definitely goes up when HN is down ;)
- ryanlol 7y agoI’ve hosted multiple IRC servers over the years and can’t remember ever having to do “maintaining” after the initial setup.
- phtrivier 7y agoHonestly curious, how long did you last without having to: * update the version of the IRC server * update the version of the os of the machine running the IRC server * repair a broken disk / fan / overflowed disk on the machine * add / remove / reset password / change weird settings of users I'm not saying this is "unbearable", and plenty of organisations have people whose job description would probably correspond to doing those tasks. But I can definitely understand why you would want to skip them entirely and have someone host your chat - which is basically the job slack is paid for. "Mission critical, but not paid by your customer" is always tricky to staff for, isn't it ?
- ryanlol 7y ago>* update the version of the IRC server Odds are that you’ll have updates worth installing once every couple of years. >* update the version of the os of the machine running the IRC server Almost never unless there’s a remotely exploitable code execution vulnerability. Local bugs wont matter unless you run multiple services on the same box. >* repair a broken disk / fan / overflowed disk on the machine Depends on your hosting setup. With a cloud setup perhaps never. I’m certainly not trying to suggest that anyone should use IRC over slack in any situation, just that it’s not a horrible time sink requiring significant maintenance.
- swebs 7y agoI wonder if this will also link to a dip in RAM sales.
- Waterluvian 7y agoI'm part of a mostly remote team. Slack going down sucks. And Pre-slack suuuuuuuuucked.