11 ms·

Stripe – Outage postmortem

• pcunite 11y ago
  >> the database operator didn’t have a way to check whether the index had recently been used for a query. Appreciate the disclosure. Stripe engineers have been so helpful to me and my business. Keep up the good work.
• wpietri 11y ago
  Nicely done. Good job avoiding retrospectively blaming people and instead focusing on future system improvements. (For those wondering why this is important, Sidney Dekker's "Field Guide to Understanding Human Error" is a mindblowing book.)
  • deleted 11y ago
    [deleted]
  • jskulski 11y ago
    Agreed. It looks like this was caused not because the application developer or DBA caused an error, but because the system didn't allow for ticket dependencies.
    • wpietri 11y ago
      Definitely. But it would have been very easy to yell at the developer ("you should have known not to do it that way") or the DBA ("why are you doing tickets out of order? you know we have to do deletes last!"). Especially after a dramatic event, those are very easy reactions to have, and they can sound very sensible.
• noir_lord 11y ago
  This is one of the best incident reports I think I've ever read, detailed, honest and without recrimination. I'm reminded of something I read in a book a while back about aviation disasters "It's not the first problem that kills you, it's the second". In this post the described system for change management is at least as good as any I've seen in production and yet a series of small problems got out of hand quickly.
  • squiggy22 11y ago
    Totally agree. Came here to say pretty much the same thing. On a related note, highly recommend the Checklist manifesto (book) for devops.
    • j_s 11y ago
      http://amzn.com/B0030V0PEW http://amzn.com/B0030V0PEW
• depsypher 11y ago
  They clearly just need to add Devops. That'll fix everything.
  • depsypher 11y ago
    Wow, message received HN. Don't make fun of Devops! It doesn't seem that long ago that it was hailed as the silver bullet that eliminates the "throw it over the wall" mentality that causes issues exactly like this one. This issue was caused by a failure in communication between team members. That communication is just as important as good engineering.
    • kzhahou 11y ago
      Your comment was obviously meant as commentary on the DevOps trend, but in fact devops is not so well-defined a trend to make a good target for sniping comments like this. I mean, people talk about "devops" today to mostly just say "we must increase investment in our infrastructure." No one's saying "devops will fix everything." So I don't think people downvoted you for cutting on devops per se, as much as for not presenting any clear POV. It's not like Node, where evangelists DO go around saying it's the best thing ever and blindly ignore the problems with it. Now... there's a good target for snippy remarks! :-P
    • geofft 11y ago
      I would interpret the message as "Don't make fun," not "Don't make fun of Devops." If you have a technical point to make, being sarcastic and curt is one of the worst possible ways to accurately convey that technical point. Furthermore, if Stripe already uses devops, then your statement adds nothing to the conversation (maybe this follow-up comment would have, but your original comment didn't), and if they don't, then commentary about a practice that wasn't being used is the definition of an off-topic comment. So in either case, your comment is worthy of downvotes, regardless of the merits of its topic. I don't think I've ever seen anyone downvoted on HN for being opposed to established wisdom with good reason. I've often seen people downvoted who were being opposed to either established or non-established wisdom in a way that doesn't contribute productively to the conversation. They weren't being downvoted for their beliefs, but for their lack of productive contribution to the conversation.
• methehack 11y ago
  I appreciate the post-mortem and, of course, we've all been there. I have to say, though, that the cause is a little surprising. That DDL needs to be executed sequentially is pretty basic and known by, I'm sure, everyone in the engineering and operations organizations at Stipe. It surprises me that an engineering group that is obviously so clever and competent would come up with a process that lost track of the required execution order of two pieces of DDL. If process (like architecture) reflects organization, what does this mean about the organization that came up with this process? It's not a sloppy exactly. Is it overly-specialized? It reminds me of that despair.com poster "Meetings: none of us is as dumb as all of us" in that a process was invented that makes the group function below the competence level of any given member of the group.
  • eldavido 11y ago
    I had this thought too. I'm actually surprised it was something this simple for an engineering org as competent and storied as Stripe, guess it proves Murphy gets all of us at one point or another. The issue IMO is splitting an otherwise atomic procedure (creation/drop of index) into two change tickets. I'd be interested to know how the DDL was communicated to ops that led to its getting split.
    • vkat 11y ago
      We do a ton of database updates regularly and we rely on a simple wiki put together that has all install instructions for a release which goes through sr engineers peer review. Seems like having a single tool/document for code and db updates could have mitigated this?
      • mercurial 11y ago
        How about automated migrations? At the scale of Stripe, you may want to have dedicated personnel keeping an eye on it, but had it been in the same SQL file, none of that would have happened...
  • toomuchtodo 11y ago
    It's pretty easy to monday morning quarterback other org's choices or actions here. Every person posting here will have something in their org break at some point that was "obvious" to the rest of us. EDIT: Someone, somewhere is going to have a bad day because they didn't know what you did. This is why sharing knowledge is so important. That's part of what HN exists for! Share what you know! Help improve open source tools! Help your fellow IT professional get a good night's sleep.
• vruiz 11y ago
  > At 00:08 UTC, our on-call engineer had been paged and had responded. At 00:10 UTC, we linked the API degradation to the removal of the index. I'm sure stripe has very good metrics of all their systems, nonetheless that's some rockstar level debugging skills.
  • kzhahou 11y ago
    Well the DB admin had just removed the index a few minutes before, and immediately the API times spiked. When you remove an index there is exactly one risk --slowdown-- and that's precisely what the admin noticed.
    • vruiz 11y ago
      That might be it, but by "on-call engineer" I understood it was someone else.
      • kzhahou 11y ago
        Sure. I picture it as: "Hey {on-call-engineer}, we're seeing a huge slowdown all of the sudden." "Was there any change in the last few minutes?" "We just deleted a database index." "Yup, that'll do it. Gotta restore it."
  • thezilch 11y ago
    I'd hope one of the first things for DB slow down is the running queries (eg. `show processlist` in flavors of MySQL) and seeing the number of rows examined. Maybe your DB doesn't have that, so you see hopefully see long running queries and they are mostly the same one, or you can see the same queries in your DB's engine status (eg. `show innodb status` in MySQL). If you don't have rows examined from earlier, you can run whatever equivalent you have of `explain` on these long running queries that are happening a lot, and now you should find the number of rows it is examining (and probably that it is running a scan query and not indexed). I mean, you only get these scars from actually working with your DB and hopefully in development / capacity planning stages, so maybe that makes you a rockstar. Not to take anything away from the Stripe engineer(s).
• siliconc0w 11y ago
  For schema/index changes I prefer migrations. These should be committed along side the code, tested, and promoted through environments. This largely prevents dependency issues because the migrations are ordered. Devops is like 80% dependency management. It's painful initially but you have to crack down on manual changes to production - all production changes should be defined in code, committed to git, tested, and flow through non-prod environments first. (just had an outage yesterday that was essentially because I failed to do this)
  • deleted 11y ago
    [deleted]
  • rawnlq 11y ago
    Do people even test for perf issues in non-prod environments?
    • toomuchtodo 11y ago
      I've found it to be extremely difficult to replicate production workloads in staging/development.
    • egwor 11y ago
      If it's a requirement to be within X performance then yes.
    • siliconc0w 11y ago
      Yes you can do scaled down testing in non-prod. An example test may try 100,000 or 1,000,000 operations when in production you might see 100,000,000. So you won't catch subtle performance problems but you'll catch order of magnitude problems (i.e something that is O(n) vs O(log(n)) Although in this case the test for this change may just EXPLAIN a known query and ensure the output matches what is expected.
    • Alan01252 11y ago
      I try to at least have the slow query log on and monitored in staging. It's pretty easy to forget an index and not notice in a dev environment.
• nartz 11y ago
  Good post mortem, just some thoughts, would love to hear what others think. Shouldn't the app developers be in charge of deploying their code and making sure its working? It seems odd that they pass off a migration like this to a DBA to then go deploy randomly, or that they weren't around when it was happening to monitor it. Also, it seems like there should have been a script that can be run that encapsulates the required dependencies, namely, don't drop an index before building the new one maybe? This should at least minimize the amount of context needed in a ticket. Relying on fully correct context in tickets seems like it could be super error prone.
• codegeek 11y ago
  ohh. Could this mean that a payment may have failed ? I did have a failure of a payment around this time but it said "Card declined". So I hope it really was the card and not related to this incident.
  • matthewarkin 11y ago
    Failures caused by this should have returned an api_error, a card_error (like card declined) would have been returned upstream from Stripe.
• matthewarkin 11y ago
  Woot, a postmortem from Stripe! I've been asking for these for over a year and hopefully we'll see more from them in the future.
• leeleelee 11y ago
  What is the point of having a human "database operator" who carries out simple tasks like deleting an index when it shows up in a work request log? Is this how most companies structure their dev teams? If you have a human there to perform tasks, then it would seem natural to allow them "human advantages" such as the ability to communicate with the person who created the request, or the ability to have some of their own checks and balances before performing the index deletion (ex. let's take a look and see if this seems safe based on the current schema and codebase). I am also surprised how easy it was for a single dev to make a request that subsequently results in the modification of a production database.
  • thrownaway2424 11y ago
    Indeed, what is the point? If they have some separation of concerns between dev and ops then it also makes sense to give ops some separate rights and responsibilities, such as vetting a change like this by analyzing the resulting query plans from a sample of production database queries. I'm sure something like that will be in their internal postmortem action items under "prevention".
• perlgeek 11y ago
  It's a vicious circle: an index is missing, so requests take longer, so more DB queries run at the same time, the load on the DB server shoots through the roof. And then it needs to rebuild an index on top of the already above-average load.
• Animats 11y ago
  What's so striking about this is that entire retail chains can be shut down by a problem in some cloud server. Starbucks had a server outage in April which caused stores to close.[1] There's a trend towards "hosted POS", where point of sale systems have to talk to the "cloud" to do anything, even handle cash. Until recently, most POS systems were running off a server in the manager's office, which communicated to servers and credit card systems elsewhere. A network outage didn't affect cash transactions. Often, the systems could even process a credit card transaction without external help, giving up real-time validation but still logging the transaction for later processing. There's a single point of failure being designed into retail here. [1] http://www.usatoday.com/story/tech/2015/04/24/starbucks-point-of-sale-cash-registers-down/26338505/ http://www.usatoday.com/story/tech/2015/04/24/starbucks-poin...
  • thrownaway2424 11y ago
    When Blue Bottle Coffee switches to Square there was a noticeable decline in the throughput at the cash register. It just takes the retail employee longer to do anything on an iPad. Pretty much everything can be done faster on a real cash register. There have also been the requisite outages, of course. Recently I was at Blue Bottle and the Square terminal wasn't opening the cash drawer. They were making change out of the tip jar :-/
    • Animats 11y ago
      There's something that a lot of retailers don't get - never put an obstacle in front of the customer giving you their money. Don't let lines form at checkouts. Don't clog up the counter with impulse-purchase stuff. Don't put displays in the path of customers headed for checkout. Don't make customers jump through hoops with loyalty cards and data entry. Don't do anything that slows the checkout process. There are expensive retail consultants who clean up stores and improve sales by doing this. Gap gets this right. Gap stores have big, clear counters, so you can bring up lots of merchandise and have a place to put it. This increases sales per customer. Gap is amazingly successful despite a rather blah product line.
      • 11y ago
• jorgeortiz85 11y ago
  Hi, I work in infrastructure at Stripe and I'm happy to provide more insight. Several threads here have commented on our tooling and processes around index changes. I can give a bit more detail about how that works. We have a library that allows us to describe expected schemas and expected indexes in application code. When application developers add or remove expected indexes in application code, an automated task turns these into alerts to database operators to run pre-defined tools that handle index operations. In this situation, an application developer didn't add a new index description or remove an index description, but rather modified an existing index description. Our automated tooling erroneously handled this particular change and interpreted it not as a single intention but instead encoded it as two separate operations (an addition and a removal). Developers describe indices directly in the relevant application/model code to ensure we always have the right indices available -- and in part to help avoid situations like this. In addition, the tooling for adding and removing indexes in production is restricted to a smaller set of people, both for security and to provide an additional layer of review (also to help prevent situations like this). Unfortunately, because of the bug above, the intent was not accurately communicated. The operator saw two operations, not obviously linked to each other, among several other alerts, and, well, the result followed. There are some pretty obvious areas for tooling and process improvements here. We've been investigating them over the last few days. For non-urgent remediations, we have a custom of waiting at least a week after an incident before conducting a full postmortem and determining remediations. This gives us time to cool down after an incident and think clearly about our remediations for the long-term. We'll be having these in-depth discussions, and making decisions about the future of our tooling and processes, over the next week.
  • eldavido 11y ago
    hi jorge I'd actually applied to work at stripe about two years ago, you guys turned me down ;) I was responsible for ops at a billion-device-scale mobile analytics company for about 1.5 years. Your tooling is far superior to anything we produced. I like the idea of a single source of truth describing the data model (code, tables, query patterns, etc.) a lot, and doubly-so that it's revision-controlled and available right alongside the code. I think it's far from decided though, how much to involve human operators in processes like this. Judging from this answer, you seem to be on the extreme end of "automate everything". How then, I'm curious, do you train/communicate to developers what can be done safely vs. something that would cause i/o bottlenecks, slowdown, or other potentially production-impacting effects? Can you even predict these things accurately in advance? (Some of our worst outages were caused by emergent phenomena that only manifested at production scale, such as hitting packet throughput and network bandwidth limits on memcached -- totally unforseeable in a code-only test environment). It sounds like you let developers request changes (a la "The Phoenix Project") but ops is responsible for final approval of the change? That actually sounds like a great system. Would love some elaboration on this. In any case, great writeup and from one guy who's been there when the pager goes off to another, sounds like the recovery went pretty smoothly.
• deleted 11y ago
  [deleted]
• scurvy 11y ago
  "Quick code fixes" almost always make the problem worse and just cause stress and anxiety. It's always better to simply tackle the root cause and fix that. Unless you're disabling a feature, don't push "quick code fixes". You'll pay for it later.
  • MichaelGG 11y ago
    Except this seems to be counter factual? And the quick fixes were just temporary until the index was finished building. Seems like a perfectly fine thing to do to restore service during an outage, as long as they're rolled back or reviewed later.
    • scurvy 11y ago
      > Seems like a perfectly fine thing to do to restore service during an outage, as long as they're rolled back or reviewed later. Rolling back later seems to forgotten in most cases (unless it's disabling a feature). Then you end up with this weird behavior and code path that few remember. "Oh yeah we did that when...." You usually only rediscover it after that quick hack is a problem. This is decades of experience here.
  • mst 11y ago
    Given it was temporary and they said it led to the API running "slightly degraded", it sounds like disabling a feature is pretty much exactly what they did.
• protomyth 11y ago
  So, and engineer can just submit a production change that an database operator will execute? What is the ceremony around this and does Stripe employ DBAs for production. What is the review process for a production change?
  • jewel 11y ago
    The change was to add a new index and then remove the old one. It's something that would have passed review. The cause was a defect in the tooling. The requests weren't tied together in the way they were displayed and so the DBA removed the old index first.
    • protomyth 11y ago
      That's not a tooling defect, that's a process and personal knowledge of the system defect. No DBA should remove an index without knowing what is replacing it or that code got deployed to make it obsolete. Checking if that index is currently being used would be a minimum. If a DBA cannot stop a deploy you have process problems.
      • mst 11y ago
        The post mortem specifically noted that it looked unused, presumably because the code in master now claimed to depend on the new index.
        • protomyth 11y ago
          I do not mean by looking at the source code. I mean by looking at the production database that you are about to remove an index from. This is a financial institution, and I am really struggling with this post mortem to understand how they do not have more ceremony around production changes. Checks and balances are important.
  • aidos 11y ago
    I suspect that it's more common than you'd guess. In one situation I was working for a supplier to a massive corporation (a household name). They had all sorts of red tape in the process. Every change required filling in forms and getting official sign off from several parties before getting the release code over. Anyway, we had a database user that was so restricted the we couldn't run the install process of a new product. I knew they ran the releases as SA so the start of my release script upgraded all the permissions of our user to give full access to everything.
• radicalbyte 11y ago
  Is it an option for you to maintain a set of tests which simulate behavior on the tables? I've have great success doing this with datawarehouses (i.e. star schema, large tables with few writes). You run the tests after each index change on acceptance. It caught a errors. For OLTP it's harder, you need to record some production workload and reply it. At your scale it's easier said than done, though.
  • brown9-2 11y ago
    If the change that was introduced required creating a new index, it's unlikely that a test would have though to remove the now-outdated index or testing performance if the create-new/remove-old operations were reversed.
• vostro_mf 11y ago
  Sigh, yet another example of hot shot teams using MongoDB just because it's new and sexy. Existing, established tools such as Oracle and Postgres would have offered lots of ways of avoiding such a problem.
  • pbreit 11y ago
    Was it mentioned anywhere this occurred in their mongo environment?
  • deleted 11y ago
    [deleted]
  • rjbrock 11y ago
    I am a huge fan of postgres, but removing an index on postgres like this (especially if it is the primary means of querying a large table) would have the same effects. This is just a failure of the tool that executes indexing operations, and not of the db itself
  • cookiecaper 11y ago
    Although it's probably not directly relevant to this problem, I agree with you. MongoDB is the new MySQL; early on the scene and sexy, but at a real cost. There are other solutions doing the same things much better, and your life will be easier if you do your research before jumping into the sexy solution. Companies should think carefully before introducing MongoDB (or any immature project) into critical production stacks.
  • dang 11y ago
    We detached this subthread from https://news.ycombinator.com/item?id=10366459 https://news.ycombinator.com/item?id=10366459 and marked it off-topic.
• pbiggar 11y ago
  Very interesting. In a continuous delivery environment like this, you generally want to test and validate any change you make. Do any DBs support the idea of turning off an index (but keep updating it). That way you could disable it, keep going for a while, and then delete it once you're sure it's unused?
  • tempestn 11y ago
    In MySQL/MariaDB (and presumably other major databases) you can instruct any given query to ignore an index, but it has to be done at the query level. Perhaps the db interaction layer of the application code could be written such that all select queries are built with a variable containing a list of indexes to disable. You could then set that globally (or using dependency injection) and any query using the index would stop using it, without needing to go around and search for uses manually.
• ryporter 11y ago
  People seem to be focusing on the fact that the two operations should have been linked, but I think they actually should have been further separated. The old index presumably could have been left around for days, at the cost of some performance, and only deleted once they were sure that the new index was successfully being used by all production code. While splitting these two changes may be silly in the case of a simple index change, I think that it's a good general policy to only deploy the minimal set of changes to a production database at once. On the production system I manage (admittedly, much simpler, and many orders of magnitude smaller), I always deploy changes in two stages, with the second stage generally deployed about a week later, once I'm sure that only new code will be running against the database. This comment is not meant to second guess the Stripe developers (who produced a great postmortem), but to suggest another possible remediation.
  • mst 11y ago
    I had a similar thought but if it's that critical an index it's entirely possible they couldn't afford the write overhead to keep both copies around.
• la6470 11y ago
  Why would someone delete a existing index without recreating it first? This is common sense for a DBA but not for our full stack engineers who have to know everything about everything g.
• rdancer 11y ago
  "Outrage postmortem"
• coolrhymes 11y ago
  Great postmortem report. I am very surprised to see that the DB ops simply deleted the index without even considering the consequences. More importantly, why isn't the 2 changes part of a single migration file? For e.g. in Django, south migrations can have both remove, create indexes in one file and executed together. Also, like some of the guys mentioned, why is the update performed to your entire global cluster. Shouldn't it be incremental like for e.g. one availability zone at a time?
• t0mk 11y ago
  Would be cool to start the report with a tldr, containing just the essence of the incident, sth like "dev needed to update index and at time .. submitted two tickets for new ibdex creation and old index removal. at time .. Op processed the removal ticket first which caused outage in service .. It was alerted at time .. and on-call op identified it at time .. He proceeded to ... Just for people who want to know what's happened but dont care for details.