15 ms·
Users may be unable to connect or experiencing degraded performance
- dswalter 6y agoAnecdotally, some messages seem to be failing to send.
- cordite 6y agoespecially if there are attachments.
- aardvarkk 6y agoIt's taking ages for me to switch channels in the Linux desktop app.
- danpalmer 6y agoWhich means there's a network request involved in switching channels, which is not a great design.
- ankit_it09 6y agoIts showing blank screen in desktop app.
- Mvandenbergh 6y agoGreat day to get some work done.
- dvtrn 6y agoI’ve recently taken an interest in data analytics and have been using daily volumes of interruptions I receive via slack (and other sources) as a data set to learn myself on and try to understand how I can be more productive individually. Today’s disruption should provide an interesting data point all of its own.
- Consultant32452 6y agoDo you work for yourself or someone else? If the latter, why do you spend any effort maximizing your efficiency? This is a pattern that I see often on HN. People take an employer's problem (they are wasting the time they've purchased with distractions) and making it their personal problem.
- ffdjjjffjj 6y agoSome people try to approach an employment relationship as one of good faith, rather than an adversarial one where you try to do the minimum amount of work without getting fired. Plus, investing in your craft has long term benefits for your career and skill set.
- Consultant32452 6y agoI don't think there's anything adversarial about simply doing what you're asked to the best of your ability. I might occasionally drop a tidbit of advice about how I could be used more effectively, but ultimately that's not my decision. Just because I perceive it to be wasteful doesn't mean it is to the person signing the checks. Regarding long term investment. For most people at most companies long term (more than 2-3 years) investment means moving to a better job somewhere else. Investing in your future is figuring out how to get the next job, not maximizing one aspect of your current job that your employer doesn't even seem concerned about.
- dvtrn 6y agoBecause I just want to. This is really just nothing but me giving myself a project to work on because I wanted to learn something.
- edoceo 6y agoYou tracking time on HN too? Not snark. I noticed that HN fills a void waiting for a build but then I'm still on here 10min after the build is done.
- 6y ago
- saxonww 6y agoI'm seeing slow performance switching/joining channels, and it's not clear whether DMs or @mentions are working consistently. Messages I send are frequently coming back as 'could not send message'. I'm not delighted by their use of might/may language and lack of detail in the status messages. Slack is pretty clearly not working well for a large number of users, and based on the above IDK what can be attributed to degradation vs. people being unresponsive.
- jrockway 6y ago> I'm not delighted by their use of might/may language and lack of detail in the status messages. Also check their status page in a month or so, and you'll see "October 5, 2020" with some message like "no issues".
- saagarjha 6y agoDoesn’t work for me either. Of course, the calendar says “100% uptime” even as there’s an outage right below it: https://status.slack.com/calendar https://status.slack.com/calendar
- fgkramer 6y agoWell, in their defence, you can't really know how much the uptime is going to be affected by until the incident is resolved. If it was a live counter then you'd see the uptime go up as the month progresses!
- deathanatos 6y agoJust compute live uptime for the portion of the month that has elapsed thus far…
- filleokus 6y agoI don't have a problem with it if they update the uptime stats afterwards, but why couldn't it be implemented "live"? Basically time "down" / total time elapsed during $period. At the beginning of the month/quarter/period you'd be at 100% (if they are up at that moment). Or even simpler, just have it be a measurement of a rolling window X days in the past.
- tspike 6y agoBecause then the metrics can't be reviewed/vetoed by executives before being published. Optics > accuracy for most organizations, sadly.
- boston_clone 6y agoUnfortunately true. About a year ago, I had reported a pod-specific issue to Zendesk (I'm fuzzy on the details). They confirmed the issue but also acknowledged that it would not reflect on status.zendesk.com. I've questioned status pages since.
- seibelj 6y agoSlack is unusable, but it did get me to restart Docker which had ballooned to 16 GB memory and a bazillion processes. Modern software development is something else
- etimberg 6y agoGood reminder that I needed to do this. Docker for me had gotten up to 13GB even though I have it supposedly limited to 8.
- aeyes 6y agoIf you are on a Mac you are probably looking at virtual memory because that is what Activity Monitor displays by default.
- etimberg 6y agoI was checking with top, but it's the same value. A double restart of the docker engine + changing the memory settings seems to have it holding. It's like an upgrade just ignored the previous setting.
- jrib 6y agowaiting the 10 seconds it takes to build a javascript frontend these days always leaves me reminiscing about my highschool days of editing my javascript in notepad and refreshing my browser to instantly see my mistakes. I understand there are benefits and reasons for the state of affairs...
- EarthLaunch 6y agoI just started using Snowpack and it is a godsend. It’s an anti-bundler that puts separate modules into a directory. Between that and using vscode remote SSH onto my server environment, it’s almost like the old days of save-refresh, but with auto refresh!
- taf2 6y ago
- luisnuxx 6y agoit's not slow performance, it's completely unusable to me.
- jacquesc 6y agoThis backend outage reveals how horrible the Slack UI is implemented. You post a message.... you think it got sent. Then 1min later, it tells you it was not.
- notyourwork 6y agoJust experienced this and decided to context switch to hackernews for a few minutes. Ironic that this post is the top on homepage.
- arbitrage 6y agoThat's the opposite of irony, as one would absolutely expect weirdness on Slack to be talked about on HN.
- notyourwork 6y agoBroadly speaking maybe, but situationally not really. The likelihood that the issue I'm experiencing is a service wide issue impacting many others is usually low probability.
- deleted 6y ago[deleted]
- quietbritishjim 6y agoThe message is in a light grey colour until it's confirmed. I'm not saying it's amazingly obvious but at least the information's there. In fact I'm mostly posting this comment so others know what to look for. (I know this because I just spent 60 seconnds staring at my message trying to figure out why the formatting was off before it told me it wasn't sent. It sent immediately when I retried though.)
- sixstringtheory 6y agoI’ve had messages that were black for at least one minute turn gray and only then notify me that they failed to send
- TheUndead96 6y agoI watched this a while ago, and it did not fill me with confidence for the Slack backend at the time: https://youtu.be/WE9c9AZe-DY https://youtu.be/WE9c9AZe-DY
- jsmith12673 6y agoCould someone comment on why exactly this a poor design for their backend? Genuinely curious, I don't have any real world context on systems like this.
- tmsh 6y agoIMHO, at scale SQL will breakdown. Even with sharding like Slack is able to per organization. It's why we have great things like Cassandra and DynamoDB. They're designed to solve replication in an easier way than replicating RDBMS iff you know your data access patterns in advance and they're not ad-hoc (which SQL is great at). This is the case for Slack. The typical way to solve RDBMS bottlenecks is to put a queue and messaging system in front of them. This breaks down when your services have bugs (my guess at what's happening). https://www.youtube.com/results?search_query=aws+rick https://www.youtube.com/results?search_query=aws+rick is pretty good on why some NoSQL approaches are a step forward (perhaps not MongoDB at scale if consistency is necessary https://jepsen.io/analyses https://jepsen.io/analyses). In particular though: https://youtu.be/hwnNbLXN4vA?t=992 https://youtu.be/hwnNbLXN4vA?t=992 There could be other issues about why Slack is slow. But at Slack scale, you need to be extremely heightened in your database strategy or you should follow the industry and use Cassandra/DynamoDB's built in partition tolerance. Key value stores scale horizontally much easier. B-trees don't scale as easily horizontally past a certain point. Essentially, good NoSQL DBs have abstracted scale for you (so you don't have to think about it as much). But you have to know the access patterns in advance (the types of queries and updates you'll be running for most use cases), since you need to design your table around these access patterns. RDBMS leaks scaling from the abstraction (you need to use message queues, etc.).
- paxys 6y agoIs the whole "NoSQL is better than SQL!" thing still going on?
- dannyw 6y agoDear Slack: Your service is down, it's not "slow performance". Be honest to your customers, or I will make sure our team switches to Microsoft Teams.
- sdflhasjd 6y agoTheir status page is annoying me more and more with these recent outages. Recalling the incidents in September, it's a bit cheeky they call that "99.93%" uptime (30 mins of downtime). Having remembered that there were at _least_ 2 seperate 1 hour+ incidents where I was struggling to send messages in Sept, I went through to look at the history, and they seem to have purposefully hidden these incidents on the calendar. There's no way of listing incidents in a month. The information and accessibility of the status page is obviously politically motivated by marketing.
- supernova87a 6y agoIf companies were honest, they would multiply downtime by the number of users at the time. Because of course, great uptime doesn't matter if it's only uptime when no one is using it in the middle of the night. It's like when trains report 95% on time, except that the times when the service is delayed is when the trains are full. All the delays that actual people experience (person-hours) happen in that 5%.
- Bedon292 6y agoIts interesting, I looked at that page earlier (~1145 EST) and it was actually tagged as an "Outage" rather than the "Incident" it is right now. For me it has only been slow performance all morning. Its not good, but not what I would call "down" either. At least from my perspective. I could have a bunch of other folks who are completely unable to send messages and I just can't tell.
- gauravphoenix 6y agothe update says: >Some users may be unable to connect to Slack, while others are still experiencing general performance issues.
- 6y ago
- jmuguy 6y agoNot a good time to be testing a slack bot :(
- Aaronstotle 6y agoPour one out for the SREs.
- HalcyonicStorm 6y agoAt least its during business hours for them.
- viraptor 6y agoDoesn't Slack have SRE-equivalents around the world now? (enough to do follow-the-sun support)
- the_duke 6y agoI have intermittently faced each of these issues over the last hour across several European and a US Slack: * Messages are shown as sent (not grey), then, after several minutes, a "Try to send again" message appears. Or the message turns grey. Or both. * The webapp randomly reloads. Sometimes already typed messages are preserved, sometimes they disappear. ( I assume a reload happens when a new version is deployed with some "force reload" flag ) * Threads and channels just show a loading spinner forever. * After switching channels, all fonts disappear and just the menu background remains. No remedy except reload. Outages happen. They get resolved. That's live, and it's fine. What is not fine is how ill-equipped the frontend is to deal with such failures.
- mrmuagi 6y agoI used enterprise slack ~4 years ago for a job and I remember having these issues every so often. ~2 years ago I used free gitter for a university project and I had the same issues a lot, and we had to switch -- funnily enough, we switched to discord and it was better than all of them.
- lsaferite 6y agoMy only real complaint with Discord is how it treats audio (and maybe video?) links when you close and open the application. Don't make the mistake of shutting off your computer while connected to a voice chat as it will automatically resume the connection when it starts back up. Mute status being the same as it was when shut down if I recall.
- war1025 6y ago> The webapp randomly reloads. Sometimes already typed messages are preserved, sometimes they disappear. ( I assume a reload happens when a new version is deployed with some "force reload" flag ) I have been seeing some version of this on my laptop for months. Don't seem to have the same issue on my work computer. Maybe it will finally get resolved.
- deleted 6y ago[deleted]
- swiley 6y agoWhy do companies spring for slack and webex when xmpp exists and works better?
- anaganisk 6y agoAny of the below or all of the below (Not my own opinions, its a collection) 1. Another maintenance burden 2. You have some other company to scold or blame who is responsible for that. 3. Running it at scale. 4. Have to devote time for building features in market vs existing solutions. 5. Their UI/UX is not polished enough 6. sales bribed the manager 7. XMPP sucks for their use case.
- koenigdavidmj 6y agoIt is a legitimate improvement to see everything that happened when you were offline. At that point you're not supporting XMPP; you're supporting "XMPP with some pixie dust to save that and let it be viewed, which is in a totally different place than your live chats". Slack is a not-great implementation of a good idea, including a use case that IRC and XMPP don't meet. Tulip is a better implementation of that good idea, plus yet another good idea.
- manigandham 6y agoConvenience and UX. Entire billion-dollar companies are built on delivering a better experience, even if technically inferior.
- sergiotapia 6y agoCost. It costs money to maintain it. Example: We pay for looker. One day I thought to myself I'll spin up Metabase and try it out for myself against our prod database. It worked very well, fast and easy to query my data unlike Looker. I haven't used it in about a month and today I try to access it but the app just doesn't load. Now I need to spend time to look into why this app isn't loading. Nevermind that I have zero Java experience or devops experience. I just ran a Docker file per their instructions. It's often cheaper to pay for a service.
- sails 6y ago
- rexreed 6y agoIs Discord a viable alternative to Slack? If not, why not?
- bostonbot 6y agoFor sure. We love discord. Less distractions and great integrations for developers.
- juvoni 6y agoYes it is, but a key feature discord lacks is threads. Still it is a huge upgrade from Slack in terms of performance and ease of use.
- rexreed 6y agoI am a heavy Slack user but I don't use threads. Threads seem like a confusing mess. Am I missing something not using Slack threads?
- strifey 6y agoIf everyone is bought-in on using them, they can be a really nice way to keep single topics in one place rather than having interleaved conversations in a single channel. If people don't use it 100% of the time, though, it's honestly worse because now you're not always sure where to read/continue a conversation.
- ryanSrich 6y agoWow really? I'm genuinely surprised. I couldn't imagine using Slack without threads, how do you keep discussions on track for certain topics? At any given time I have 30-40 threads going, which means I can easily track conversations. Without threads I'm just endlessly scrolling, or trying to find something, or god forbid relying on Slack's utterly broken search.
- rexreed 6y agoThe comment below is on point. It's all-or-nothing on threads. If people don't commit to using threads it gets hopeless really fast.
- james_pm 6y ago2.5 hours in and still "investigating". No new updates for the moment as we're continuing to investigate. Customers may notice that other functionality, such as Search, is also affected. Oct 5, 12:40 PM EDT
- mmaunder 6y agoAlmost 3 hours now and their status and our user experience still shows issues. This is a bad one for them. Monday morning too.
- sdfsdfsdfv 6y agoAll of Amazon is transitioning from Chime to Slack; this doesn't look great.
- deleted 6y ago[deleted]
- sergiotapia 6y agoWe have Discord as a backup for our engineering team. And Slack's UI needs to be fixed where if a message appears in the stream, it means the server persisted it.
- yters 6y agoSlow down is due to inefficiency in the Slack log secrets extraction pipeline.
- splatcollision 6y agoA great opportunity to update the part of my API code that deals with Slack to make it handle errors more gracefully!
- dasb 6y agoIt has had slow performance since I started using it 3 years ago.
- fpj 6y agoHas worked fine for me in Spain all day.
- sandGorgon 6y agoThe android mobile apps have been horrible ever since the update a few months back. messages dont get refreshed even if the status shows updated
- nt2h9uh238h 6y agoSlack is generally an immensely slow piece of Software.
- rexreed 6y agoAnyone use Rocket.chat? Is that a good alternative?
- mulmen 6y agoSlack is degraded. My team has fallen back to email. Is this irony?
- strgcmc 6y agoJust imagine what it's like for Amazon, that just rolled out Slack internally last 1-2 months. (hint: it's not great, but at least we can fallback to Chime)
- gtk40 6y agoInteresting. They keep trying to push Chime on us, so I'm surprised to hear they're using Slack internally.
- strgcmc 6y agoWas originally reported few months back: https://www.cnbc.com/2020/06/04/amazon-licenses-slack-for-workers-as-slack-adopts-aws-video-call-tech.html https://www.cnbc.com/2020/06/04/amazon-licenses-slack-for-wo... Actual rollout started couple months later. Currently living in a nebulous/purgatory in-between state, most people have both Chime and Slack installed on their laptops/phones. Can't fully commit to just one or the other, yet.
- cozzyd 6y agoslow performance is the baseline for Slack...
- dylan604 6y agoThe current thread title "Users may be unable to connect or experiencing degraded performance" seems like it could be used to just sum up 2020. Edit: Not just for slack, but 2020 in general
- fermienrico 6y agoAt the risk of sounding like Neil DeGrasse Tyson, I find it funny how pretty much all of us on this planet is blaming things on 2020 year where it is just an arbitrary slice of time we've created. People need something to fall back on and console themselves. Edit: Folks, I was kind of half joking and making fun of NDT. And we're discussing the definition of "Arbitary" and harvesting seasons. Lighten up people.
- progman32 6y agoI understand it as another way of saying "things have been bad lately", and not as blaming a particular slice of time. :shrug: Won't argue though - it is amusing.
- dylan604 6y agoThat's pretty much it. We've come to framing things in units we can understand. "Today is going to be a great day", "Yesterday sucked", "Here's to a new year", blah blah
- slingnow 6y agoI'm not sure I understand your argument. Would you rather people frame things in units they don't understand? What's your point?
- the_pwner224 6y agoIt's not blaming the issues on the year, but a joke that the current big arbitrary slice of time most humans use has had a bunch of bad stuff happen in it and keeps having more bad things happen.
- rch 6y agoThe Android Element client for Matrix has been working really well for me lately. There's a long way to go still, but I think there's reason for optimism.
- linsomniac 6y ago"Uptime for current quarter: 99.97%" AFAICT, this outage has been going on 6 hours and is still showing a ton of services as "Incident" status in the slack status page. Seems like uptime could be more like 99.7% and dropping. But, how do out count downtime of a distributed system? We probably need a new way of discussing availability. An outage to part of your system might only impact one or a group of users, but to those users you could have been 100% down, effectively, even as other users were not impacted at all. So maybe a series of user/%age counts? "This quarter 90% of our users/use cases had 100% availability", for example. With the gold standard being 100/100. Maybe numbers for 99, 90, and 50% of users/use cases? IOW: "1% of our users had 95% availability, 10% of users had 99.9, and 50% of users had 100%."
- joshuanapoli 6y agoFor simplicity, you could use the fraction of api calls that our service level objectives. Google/SRE calls it "aggregate availability". https://landing.google.com/sre/sre-book/chapters/embracing-risk/#risk-management_measuring-service-risk_aggregate-availability-equation https://landing.google.com/sre/sre-book/chapters/embracing-r...
- foxtr0t 6y agoYeah, this problem has a pretty obvious solution that is implemented by many teams: find a unit of work that we can measure success and failure over, report an aggregate metric over a specified time interval for availability tracking and plotting: # units successful / total # of units
- a1369209993 6y agoTo be fair, that has problems like counting "do this thing; do that thing" twice as much as "do these four things" rather than vice-versa. Still useful though.
- simonbarker87 6y agoSlack has been a nightmare since the start of August - my assumption is they just can’t handle the increased level of data going through their servers with the increase in video they will now be handling
- e40 6y agoA heavy user for the last 2 years, and I have not noticed any problems until today. EDIT: problems == sustained issues. Yes, there are occasional blips.
- simonbarker87 6y agoThe occasional blips have been driving me mad. Every day is one of: - video connects but no voice - voice connects but no video - calls unanswerable - screen sharing grinds my laptop to a standstill - messages don’t send but don’t tell me for like 5 minutes - messages don’t come in and then a barrage all at once The screen sharing one is so bad I have to send zoom links out if I think I’m going to screen share in a call
- anderspitman 6y agoYou'll get more downtime hosting your own chat server. But you can also fix it immediately when something breaks.
- paxys 6y ago> But you can also fix it immediately when something breaks. Assuming that a random company will pay to have engineering/ops staff on-call 24x7 who all have enough knowledge about their internal chat system.
- anderspitman 6y agoI don't think there are many companies that don't already have at least a couple things that must stay up 24/7. Who fixes those? Or pay a company to host your chat system, but use an open source server and just have them manage your instance. If something goes wrong, you email them and they can quickly fix the issue. Because it's a simple system and they know the problem exists on your node. Besides, I don't need it 24/7. There's always email for that. I need chat during working hours. Slack is currently not fulfilling that need, and even if I could get ahold of someone there, that person wouldn't be able to fix my problem. Their system is too complex.
- tyre 6y agoif this slack outage lasts 6 hours, it's far less time then someone would spend futzing around with their own chat server that includes deployments, updates, security audits, etc.
- anderspitman 6y agoHonestly I don't even care if I had to spend 10x more time fiddling with hosting my own. The advantage is choice. If it breaks on a Friday afternoon, maybe we don't care and just call it for the weekend. But if it breaks in the middle of coordinating a deployment (like today), it can become priority #1 on the team until it's resolved. With slack I have no control.
- amsully 6y agoRecent intermittent issue asked me if I would like to log the restart of the application. Selecting yes provides a host of interesting configurations and logs. Some interesting logs most people can understand: [REDACTED] info: [OPPORTUNISTIC_RELOADS] User has been blurred for 1200000 ms, evaluating whether or not we should reload opportunistically [REDACTED] info: [OPPORTUNISTIC_RELOADS] Client is still visible to the user even though it is not focused, will avoid updating
- Stierlitz 6y agoWhat does the back-end run on? Whatever happened to the distributed in distributed systems? Just how difficult is it for text typed up on my computer to appear on your computer.
- Svetlitski 6y agoConsidering how long this outage has been going on, I'm very interested in reading the post-mortem on this.
- ma5ter 6y ago"Unfortunately, your call is unable to go through right now. Please try again later." We are facing this error for the last few weeks, in case if a member drops off the call due to internet issues he/she can't join back until ~15 to 20 mins. We all are pissed off because of this horrible issue and planning to use something else for our stand-up call