14 ms·
Slack’s Incident on 2-22-22
- belter 5y agoA Date that is both a Palindrome and an Ambigram: https://www.jagranjosh.com/general-knowledge/22-02-2022-is-both-a-palindrome-and-an-ambigram-1645449743-1 https://www.jagranjosh.com/general-knowledge/22-02-2022-is-b...
- sva_ 5y agoWell, not the way they format it.
- ncmncm 5y agoWhich was, notably, Tuesday.
- scrollaway 5y agoA post-mortem with non-ISO dates? Even on THAT date?! :)
- smegsicle 5y agolooks unambiguous to me lol
- mulmen 5y agoOk, what is the format string? a) m-dd-yy b) m-yy-dd I can't tell. How do you disambiguate?
- rjh29 5y agoIt can't be either, 22 is not a valid month. I agree with you though, the point of a date like yyyy-mm-dd is to avoid working out stuff like this. You don't pick a date format based on whether the current date is ambiguous or not.
- mulmen 5y agoGood catch. I updated my post. The question remains, how can this format be disambiguated? Agreed, this is why ISO8601 exists.
- RandallBrown 5y agoIt's disambiguated by context, like most ambiguities in language. This is the title of a blog post after all.
- lkbm 5y agoIf you're trying to write a date parser for their outage report titles, the problem isn't the format of this date. It's that this is their only outage report with a date for the title. It's the title because it's a novel date, and formatted this way to emphasize the novelty. It's also a date for which your question is irrelevant: it's the same either way.
- oxfordmale 5y agoWhich major outage ? According to the Slack uptime, there was barely 1.5 hour of outage :-) P.S. Yes I know the uptime is decide by committee, and doesn't reflect reality. I am just being cynical.
- epmatsw 5y agoMcRib is a hilarious name for a service
- whoopdedo 5y agoI certainly wouldn't trust it's availability. (The original McDonald's McRib sandwich is well known for only being sold a limited time.) So "Mcrouter" comes from Memcache-Router, then the obvious McDonalds jokes are made and someone cleverly suggests "Mcrib" for the next service. But I can't think what the backronym would be for it. Memcache Ring Buffer maybe. Or Broker.
- jonah-archive 5y agoRIB is a common term in networking for "Routing Information Base" (being the set of all routes which could be chosen to be installed in the routing table (or FIB -- "Forwarding") by the control plane. I don't know that this is the actual etymology but it's not implausible.
- phan 5y agomemcache router interface broker
- bee_rider 5y agoApparently it generates configurations for Mcrouter. Could be MemCache-Router Instance Borker.
- true_religion 5y agoI think you meant Broker, but the misspelling is an act of genius since we are talking about downtime caused by an infrastructure failure.
- qubyte 5y agoAnd I misread it as McBorker and now I can't stop chuckling.
- mescaline 5y agoAn over communication platform should have scheduled outages like this regularly!
- AndrewUnmuted 5y ago
- jdlshore 5y agoThis sort of handwavy conspiracy thinking is distressingly common. What basis do you have for your suspicion? Is it just “big company bad”?
- mulmen 5y agoI choose to believe it is not common thinking but instead commonly verbalized among the minority with such thoughts.
- jdlshore 5y agoYou're probably right. I've noticed a trend of people fantasizing out loud in the past 3-5 years. It's nearly always cynical / conspiracy-theory fantasies, and they're stated as fact (or near-fact, as GP did), but without any backing information or logic... just pure fantasizing. But, thinking back, it's not that common. Hopefully just a fad. People expressing their frustration at inequality and fear of corporate dystopia.
- AndrewUnmuted 5y ago
- alphabettsy 5y ago> engulfs the worker inside a dead-eyed grunt culture, featuring an endless spree of work-life balance destroyers. It might be great for people who ask for things from others, but for the people who have to actually do the thing being asked of them, Slack is a nightmare world. I think it depends on the organization and how you use it. In a previous role I would’ve agreed with you. People expected you to reply at all hours, where I am now that isn’t the case. Tools do not create toxic culture or destroy work-life balance. Organizations do that.
- diarrhea 5y agoThat date format is actually the worst I have ever encountered. m-d-y, with year in 2 digits, numbers not zero-padded, US "order" yet using dashes. It's like a moderator of /r/ISO8601 came up with the worst possible format on purpose. Am I missing something?
- bugeats 5y agoThe incident was caused by a database query that was "2-22-22" when it was supposed to be "2-22-22".
- Invictus0 5y agoIt's a quirky nod to the fact that all the digits were 2 on that day in this format.
- adamomada 5y agoThis is what you sometimes see for best-before dates in Canada. Even better, because our dates are “supposed to” be like 22/2 but I don’t think anyone here does that, except Quebec perhaps. Sometimes you just have no clue
- missedthecue 5y agoI mean, there is literally no way to confuse it with another date, unless you go back 100 years, when Slack didn't exist. There is no 22nd month, so we know the 22s are the day and the year, leaving only the 2 to be the month. Is it really that difficult to parse?
- johannes1234321 5y ago> leaving only the 2 to be the day I think you meant "to be the month" there. qed■
- diarrhea 5y agoBeautiful.
- 5y ago
- pierrebai 5y agoSometimes new roll-out causes outage, sometimes, roll-out are delayed due to the overall system architecture. Reading the post-mortem, I could not help but be reminded of this issue as described here: https://www.youtube.com/watch?v=y8OnoxKotPQ https://www.youtube.com/watch?v=y8OnoxKotPQ
- anonu 5y agoOne of my favorite videos. And yes, slack is way more complex than I could ever imagine
- encryptluks2 5y ago> including the author — which certainly made my role as Incident Commander more challenging! As if no other way to communicate exists? I remember using Slack, feeling fed up with emails, until I realized that if I wanted to sync Slack messages offline and have a standard way to view these messages that I was SOL. I am so glad that I've returned to email and optimized my workflow to use email effectively and efficiently. The best part is no more vendor lock-in.
- orf 5y ago*22/2/22
- adamomada 5y agoYou know how the date looked strange to you? It’s the same for your correction, but for other people
- orf 5y agoFor a statistically insignificant portion of people, sure. It doesn’t make it any less correct.
- 4ggr0 5y agoFor the whole of Europe it would be 22.02.2022, how is all of Europe statistically insignificant?
- orf 5y agoThe official EU rules say 22.02.2022, but nobody in Europe would have trouble parsing 22/2/22 or any variation thereof. And the / (or -) separator is indeed used in parts of the EU. It’s the ordering that’s significant, not the separator.
- Symbiote 5y agoThe separator is often a good clue. Dots and dashes strongly imply d-m-y, slashes imply an English date, which might be m/d/y if it is from North America. A mixture is even more likely to be d-m-y, today is 27/4-2022 in Danish handwriting.
- 4ggr0 5y ago> 27/4-2022 in Danish handwriting I've never seen a date written like this, interesting.
- dormando 5y agoHi! I'd like to offer some hopefully useful information if any Slack folks end up reading this, or anyone else with a similar infrastructure. I'll start with some tech and make a separate philosophical comment. Also caveat: I have no deep view into Slack's infrastructure so anything I say here may not even be relevant. YMMV. First some self promotion: https://github.com/memcached/memcached/wiki/Proxy https://github.com/memcached/memcached/wiki/Proxy memcached itself is shipping router/proxy software. Mcrouter is difficult to manage and unsupported. This proxy is community developed, more flexible, likely faster, and will support more native features of memcached. We're currently in a stabilization round ensuring it won't eat pets but all of the basic features have been in for a while. Documentation and example libraries are still needed but community feedback help speed those up tremendously (or any kind of question/help request). It's not clear to me why memcached is being managed like this; mcrouter seems to only be used to abstract the configuration from the clients. It has a lot of features for redundant pools and so on. Especially with what sounds like globally immutable data and the threat of cascading failures during rolling upgrades it sounds like it would be very helpful here. If cost or pool sizes are the main reasons why the structure is flat, using Extstore (https://github.com/memcached/memcached/wiki/Extstore https://github.com/memcached/memcached/wiki/Extstore) can likely help. Even if object value sizes are in the realm of 500 bytes, using flash storage can still greatly reduce the amount of RAM necessary or reduce the pool size (granted the network can still keep up) with nearly identical performance. Extstore takes a lot of tradeoffs (ie; keeping keys in RAM) to ensure most operations don't actually write to flash or double-read. Extstore's in use in tons of places and everyone's immediately addicted. Finally, the Meta Protocol (https://github.com/memcached/memcached/wiki/MetaCommands https://github.com/memcached/memcached/wiki/MetaCommands) can help with stampeding herds to help keep DB load from exploding without adding excess network roundtrips under normal conditions. I've seen lots of workarounds people build but this protocol extension gives a lot of flexibility you can use to help survive degraded states: anti-stampeding herd, serve-stale, better counter semantics, and so on.
- dormando 5y agoNow a more philosoraptor style comment: I see Mcrib is a service built to quickly detect and replace memcached's. I treat memcached in infrastructure as a very stable service. Meaning it is infrequently necessary to upgrade it, and it will generally not fail on its own. If it does it will be highly infrequent compared to services with higher churn or more complexity/dependencies. This means if they're failing often enough that you need to rapidly detect and replace them you have a more fundamental problem. From a structural standpoint I think my technical comment can be useful. If things really are failing this much A) you should figure out why and slow that down. B) if you have a generally stable system and understand the typical rate of failure, you can add tripwires into Mcrib to avoid over-culling services and loudly raise alarms. Then C) you can improve technical reliability with redundancy/extstore/etc. I've also seen plenty of times where folks have a dependency of a service determine if that service is usable, which I disagree with quite strongly. Consul being down on a node should trigger something to consider if the service is dead. It's important both for reliability (don't kill perfectly working things because you end up having to design around it), and for maintainability as you've now made people afraid of upgrading Consul or other co-dependent services. Other similar failures are single-point-of-testing availability checking where instead you probably want two points of truth before shooting a service. Now you risk people being afraid of upgrading probably anything, which means they will work around it, abstract it, or needlessly replace it with something they feel safer managing. The latter is at best a waste of time, at worst a time bomb until you find out what conditions this new thing breaks under. This isn't advocating that you design without assuming anything can fail anywhere at any time; just pointing out that how often a service _should_ fail is extremely useful information when designing systems and designing fail safes, alerts, monitoring, etc.
- bognition 5y agoIt’s likely that the memcached install is so large that the underlying instances themselves are failing. When you have hundreds or thousands of instances, failures in the instances themselves become pretty regular.
- dormando 5y agoI can say with certainty this isn't strictly true. The failures should be relatively rare; when I say relatively I mean on the level of natural node failure. If natural node failure isn't survivable without special systems to quickly replace downed nodes you don't actually have an N+1 redundancy system. Thus, the pools aren't large enough :) Or, in this case, if they really are failing this much then having them always lose their cache is a major reliability hole. It's a subtle difference. I think many operators get used to node failures being extremely common when they don't necessarily have to be. I suspect the note on "if they come back on their own ensure they're flushed" meaning they have something unusual causing ephemeral failures. If that's just "cloud networking" there isn't much they can do but it's almost always fixable.
- notacoward 5y agoI love the diagrams of the cache<->DB cycle in normal vs. degenerate states. Those illustrate the problem very clearly and succinctly, and I hope they make it into a textbook some day. Kudos.
- godmode2019 5y agoTidbit: 2-22-22 was also when Russia invaded Ukraine And Joe Bidens statement about the invasion was on 2-22-22 2:22pm on the dot. I could not figure out the significance of this more than 11:11 was when WW1 ended, but it's probably something else.
- Kwpolska 5y agoNope, you’re off by two days: https://en.wikipedia.org/wiki/2022_Russian_invasion_of_Ukraine https://en.wikipedia.org/wiki/2022_Russian_invasion_of_Ukrai...
- godmode2019 5y agoIt was the announcement of invasion. 22-2-22 Then Biden did a speech against on 2:22 22-2-22 The actual war started in 2014
- godmode2019 5y agoTo refresh your memory https://edition.cnn.com/europe/live-news/ukraine-russia-news-02-22-22/index.html https://edition.cnn.com/europe/live-news/ukraine-russia-news...
- 58x14 5y agoJust my personal take, I think this is a really well-written incident postmortem. It's specific, extensive, candid, and dare I say, entertaining? Many incident reports are fully lacking in any meaningful detail, or wholly unapologetic. I actually enjoyed learning tidbits about the author, in particular their mention of https://how.complexsystems.fail/ https://how.complexsystems.fail/. Reading this boosted my confidence in Slack's teams, which should ultimately be the objective of a release like this. It's not pure PR nor a gruff legally-obligated disclosure. It helps that I wasn't really affected by this incident.
- vxNsr 5y agoThe fact that the current top comment thread is quibbling about the date format in the title seems to agree with this assessment, if there was anything real to complain about that’s what we’d be seeing, instead we get bikeshedding on the date format in the title of a post.
- dijonman2 5y agoWhich is aligned with US date formatting.
- philderbeast 5y agowhich only makes it confusing for the rest of the world
- JohnTHaller 5y agoIt's not that confusing if you know there are only 12 months. If it happened to occur on 2-4-22, then it would be pretty confusing.
- 8n4vidtmkvmk 5y agoits particularly bad because US dates are written with slashes. please leave dashes for ISO so we have some hope of understanding anything.
- olliej 5y agoI find reading about these incidents super interesting, and I generally find the work performed by the folk keeping these service running (and dealing with the inevitable falling over of any computer system). At the same time it seems like a horrifying job I would never ever want :D
- neerajk 5y ago“Mcrib is objectively a better system for generating memcached configurations — but its efficiency made the broader system behave in a less safe way.” Be good but not _that_ good :)
- _justinfunk 5y agoI'm looking forward to http://thisincidentreportdoesnotexist.com http://thisincidentreportdoesnotexist.com launching sometime later this year.
- manesioz 5y agoGreat post mortem, I love reading these. Its pretty neat that the tech industry is relatively transparent about these situations -- we all benefit from learning about them.
- di4na 5y agoAs someone that spend far too much time in incident report from all kind of industry, you made my day. I am really happy you think so. But no. We are really not transparent about them. At all.
- rossdavidh 5y agoGood writeup, although the reference to https://how.complexsystems.fail/ https://how.complexsystems.fail/ was even better.
- deleted 5y ago[deleted]
- throwdbaaway 5y ago> Because the GDM data is immutable and thus can tolerate staleness, the query was also updated to read from replicas as well as Vitess primaries. Any data that is fronted by the memcached caching tier can tolerate staleness right? There is not much difference between a short TTL of 1s versus an async replication delay of 1s. Abusing vitess primaries is the root cause of this incident, and a similar incident can happen even without any scatter query.
- perydell 5y agoThis is very transparent and a good write-up. I wonder if someone at Slack could explain how they calculate their downtime on their status page. This outage was for 3 hours and 14 minutes but they claim 99.79% uptime for the month of February.
- selcuka 5y agoIt is possible that approximately 50% of all users were affected, or the requests timed out 50% of the time.
- Robin_Message 5y agoI am confused by the query that had the problem. Specifically, I am confused by why the sharding is done by user id. Even the largest Slack instance probably has under 100,000 users and less than 1000 peak messages per second. That feels to me like it could be served by a single master DB. It feels like that would be a better way to shard. Major downside is the major difference in shard sizes so some management/migration might be needed but it seems doable to me. Certainly it feels naively that scaling should be easy due to the way slack instances are independent (unlike say Twitter).
- sophiebits 5y agoWorkspaces are not completely independent, such as for the Slack Connect feature. Some more details here: https://slack.engineering/scaling-datastores-at-slack-with-vitess/ https://slack.engineering/scaling-datastores-at-slack-with-v...
- Robin_Message 5y agoThanks! Indeed, the variability of size and usage of each instance was the big issue, and then doing features that crossed instances meant they'd be crossing shards whatever they did, so it made sense to fix the variability issue. (I'm also surprised/reminded how fast Slack grew and how quickly it became effectively ubiquitous — I think every company I've contracted for in the last five years has used Slack).
- iamcal 5y ago> Even the largest Slack instance probably has under 100,000 users and less than 1000 peak messages per second. This is not true, by an order of magnitude.
- ethical 5y agoDoesn't anyone learn anything. Youth is not an achievement. Experience is. Having to refer to a book, rather than good change management practices, highlights the madness of agile tosh, and ignorance in terms of capacity and performance management. It's also a data breach incident (denial of service, unavailability), I hope they reported this to the UK ICO, and each countries data protection regulator. Amateurish rubbish.
- sofixa 5y ago> Youth is not an achievement. Experience is. Old man yells at cloud vibes here. > It's also a data breach incident (denial of service, unavailability) How is unavailability a data breach?!
- ethical 5y ago1: Yup. 2: Its the legal definition under the GDPR ( a European law, that applies to the US, via Privacy Shield). If you can't get to your data - you get the idea? TLDR;
- macintux 5y ago> The notification obligations under the GDPR are only triggered when there is a breach of personal data which is likely to result in a risk to the rights and freedoms of individuals. https://www.lexology.com/library/detail.aspx?g=03e8a988-7c9e-4576-94ea-5ab13f2cb240 https://www.lexology.com/library/detail.aspx?g=03e8a988-7c9e...
- deleted 5y ago[deleted]