4 ms·

How to make breaking changes and not break all the things

40 points by mceachen 12y ago

lsh123 12y ago
One important point is missing from the article: make sure to have unit/functional tests to run to make sure your changes are indeed are not breaking anything.
- mtillman 12y ago
  for the record @mrm, the above comment was not from me. :) solid article though as usual.
- mceachen 12y ago
  Absolutely. Thanks. I've added Step 0 to reference this, and added an attribution at the end of the post. One thing that surprised me when I started at Twitter was the requirement for every line of code to be reviewed by another engineer—and if you change or add implementation, you can't ship if there isn't corresponding test changes or additions. More tests isn't a panacea, of course, though. There's definitely a spectrum of good, explanatory tests and tests-that-just-cement-the-implementation-details.
  - lsh123 12y ago
    The automated tests are not panacea of course. Yet, you need to have enough tests to give you confidence that the change indeed is breaking nothing. It is one story to perform a deep refactoring on a C++ or Java code with 95% tests code coverage where either compiler or tests will pickup things for you. And it is a completely another story to do a refactoring on a PHP frontend code w/o any compiler/static analyzer and w/o any tests where your best tool is search-and-replace. Even the "depth" of safe refactoring will be very different in those two cases.
    - mtdewcmu 12y ago
      Something that works in PHP's favor is that you can write very concise and clear code. At some level you have to trust your eyes.
- rjbwork 12y ago
  I've found that integration/functional tests are far more important for these sorts of base structural changes to applications, especially when it involves fundamental changes to how data is physically stored.
jakozaur 12y ago
Very good approach. Also having some controls like feature flags per customer/user is very useful. Sometimes migration takes a while and it is useful to move users gradually through those phases with ability to rollback.
- mceachen 12y ago
  Absolutely! I didn't reference it originally, but saw that it was already out in the wild, so I added a short paragraph about Twitter's solution, "Decider."
- MaxGabriel 12y ago
  When we were switching a contained part of our codebase from Mongo to MySQL, we ran both in parallel and stored a key in Redis to let us turn off the MySQL backend (we re-used our feature flag utility for this). This turned out to be quite useful. If we were throwing exceptions in the new MySQL code, or even just suspected the new system of causing problems, we could switch it off. We also switched it off so that we could easily run migrations without taking down the site.
quest88 12y ago
Thanks for the article!
- mceachen 12y ago
  I actually wrote it for some coworkers at Twitter—they were struggling with this, and it's a surprisingly non-trivial problem to tackle.
polemic 12y ago
This is a good starting point. One thing I'd add is the need to consider the effects of table locks during changes. It's a nasty surprise when you've tested your multi-stage process in dev, only to find during production updates that a long running select is blocking, or that your change causes request to queue until you exceed connection limits.

40 points by mceachen 12y ago

One important point is missing from the article: make sure to have unit/functional tests to run to make sure your changes are indeed are not breaking anything.

mtillman 12y ago
for the record @mrm, the above comment was not from me. :) solid article though as usual.
mceachen 12y ago
Absolutely. Thanks. I've added Step 0 to reference this, and added an attribution at the end of the post. One thing that surprised me when I started at Twitter was the requirement for every line of code to be reviewed by another engineer—and if you change or add implementation, you can't ship if there isn't corresponding test changes or additions. More tests isn't a panacea, of course, though. There's definitely a spectrum of good, explanatory tests and tests-that-just-cement-the-implementation-details.
- lsh123 12y ago
  The automated tests are not panacea of course. Yet, you need to have enough tests to give you confidence that the change indeed is breaking nothing. It is one story to perform a deep refactoring on a C++ or Java code with 95% tests code coverage where either compiler or tests will pickup things for you. And it is a completely another story to do a refactoring on a PHP frontend code w/o any compiler/static analyzer and w/o any tests where your best tool is search-and-replace. Even the "depth" of safe refactoring will be very different in those two cases.
  - mtdewcmu 12y ago
    Something that works in PHP's favor is that you can write very concise and clear code. At some level you have to trust your eyes.
rjbwork 12y ago
I've found that integration/functional tests are far more important for these sorts of base structural changes to applications, especially when it involves fundamental changes to how data is physically stored.

jakozaur 12y ago

Very good approach. Also having some controls like feature flags per customer/user is very useful. Sometimes migration takes a while and it is useful to move users gradually through those phases with ability to rollback.

mceachen 12y ago
Absolutely! I didn't reference it originally, but saw that it was already out in the wild, so I added a short paragraph about Twitter's solution, "Decider."
MaxGabriel 12y ago
When we were switching a contained part of our codebase from Mongo to MySQL, we ran both in parallel and stored a key in Redis to let us turn off the MySQL backend (we re-used our feature flag utility for this). This turned out to be quite useful. If we were throwing exceptions in the new MySQL code, or even just suspected the new system of causing problems, we could switch it off. We also switched it off so that we could easily run migrations without taking down the site.

quest88 12y ago

Thanks for the article!

mceachen 12y ago
I actually wrote it for some coworkers at Twitter—they were struggling with this, and it's a surprisingly non-trivial problem to tackle.

polemic 12y ago

This is a good starting point. One thing I'd add is the need to consider the effects of table locks during changes. It's a nasty surprise when you've tested your multi-stage process in dev, only to find during production updates that a long running select is blocking, or that your change causes request to queue until you exceed connection limits.