6 ms·

Too many details missing here to make any sort of conclusions. Yeah, of course bugs aren't supposed to make it to production. But they do. Is it easier to roll

• klodolph 7y ago
  > Is it easier to roll back than it is to roll forward? Usually it's not. If downtime has consequences then you have every incentive to make rollbacks happen at the touch of a button. From personal experience I know that plenty of teams simply are not at that point for many reasons. If rollbacks are “hard” in any way for you, then you should know the known consequences of difficult rollbacks—they incentivize you to push forward even when the new release is kind of broken. For example, let’s say that your team doesn’t test rollbacks and doesn’t have procedures in place. You know, in the corner of your mind, that a version rollout could potentially take your service completely down for days while people hunt down the bug or figure out how to do rollbacks on the fly. This is a predictable scenario to end up in, so at some point I recommend addressing it, or justifying why you don’t. > Continuing the rollout could very well have been the best option. Or not. Who knows. Rollback should be the default response in the face of uncertainty when you might have a bad rollout. Ideally. It should be the safe, one-button-press reset back to the way things were before. Again, I know that’s often not the case. Schema changes are the biggest sticking point, usually. I often see schema changes in two phases—first you update the schema, and then later you roll out functionality which uses the new schema, in a separate rollout. This is tricky, but most updates are not schema changes. Teams generally learn these things as object lessons (as in “we had downtime, wrote a postmortem, and it won’t happen again”) but that’s not the way things need to be. As the team evolves and the product matures, the release tooling gets better. You read articles like this and you can make the tooling better without the object lesson. No one-size-fits-all solution, obviously, but… painless, one-button rollbacks are one-size-fits-most for sure! Someday, someone writes an integration test that sets up a new and old version and makes them talk to each other, checking that they both work correctly. Another day, somebody writes an integration test that does a rollout from the previous version, followed by a rollback. And someone else adds more monitoring signals so if they see service failures above some threshold it automatically rolls back. These are all evolutionary steps. Yes, they don’t apply to all services. But—most of the services I run are somehow making money for the company, or people depend on them for critical pieces of functionality related to the company’s bottom line.
  • ryanbrunner 7y ago
    > Teams generally learn these things as object lessons (as in “we had downtime, wrote a postmortem, and it won’t happen again”) but that’s not the way things need to be. As the team evolves and the product matures, the release tooling gets better. You read articles like this and you can make the tooling better without the object lesson. No one-size-fits-all solution, obviously, but… painless, one-button rollbacks are one-size-fits-most for sure! One advantage of 'learning as you go' is that oftentimes you can afford to be messy and cut some corners when you're just starting out without it impacting people too much. I don't think anyone would disagree that having a rollout process that can be easily rolled back, utilizes things like feature flags, incremental rollout, and care around schema changes is a better process, but it's undeniably slower than "just roll it out and deal with the crashes", particularly when there's not plug and play standards for a lot of those processes and you need to build them yourself. And if you have a tiny amount of users who are the early adopter type, messiness around rollouts and the occasional outage might not be the worst thing if you can move much faster.
    • klodolph 7y ago
      > …messiness around rollouts and the occasional outage might not be the worst thing if you can move much faster. Unfortunately, my personal experience with software engineers pushing to “roll forward” on failure is that they want to meet arbitrary deadlines. Metrics that are good for the SWE (e.g. if we roll forward, I complete my Q4 OKRs) but not good for the team—total velocity isn’t negatively impacted by a single rollback, at least not much, but reliability is positively impacted by rollback policy.
  • bloody-crow 7y ago
    > If downtime has consequences then you have every incentive to make rollbacks happen at the touch of a button. Not all rollbacks are the same. Even in a theoretical scenario where you can roll back the code to a previous version with a push of a button, it's not always possible due to many factors. Sometimes new code creates and persists state that the old code can't handle and rolling back would just make the old version blow up because of it. Sometimes clients can't talk to the old code after talking to new code and expecting new protocol. The point is, there's a lot of factors that could make one or another remediation strategy more or less viable, and sometimes rolling forward is the best way to proceed. The article tries to advocate that rolling back is ALWAYS the correct thing to do, which is demonstrably not true for certain circumstances.
• deleted 7y ago
  [deleted]
• deleted 7y ago
  [deleted]
• dredmorbius 7y ago
  Feature flags, mentioned in TFA, are how you roll back quickly. Toggle the flag. Feature is disabled.
  • xorcist 7y ago
    Testing rollbacks are how you roll back quickly.
    • dredmorbius 7y ago
      The long causal chain is long.
• artsyca 7y ago
  Add to that standing orders to 'move fast and break things' -- this is just the same corporate shit warmed over that keeps people comfortably numb and confused about whether to lead follow or eat a donut