11 ms·

Move Fast and Break Nothing

459 points by bpierre 12y ago

vdm 12y ago
Comparing the results of requests processed against both the new and old system is a pattern. http://www.alwaysagileconsulting.com/application-pattern-verify-branch-by-abstraction/ http://www.alwaysagileconsulting.com/application-pattern-ver... It was also done in mainframes at the processor instruction level.
danso 12y ago
Quite a bit OT, but I've never seen the transform-rotate property used as a styling in text...probably something that can/will be used to death, but it looks great here...gives the piece a very magazine feel (that, plus the handplaced-layout of floating embed images)
- gknoy 12y ago
  I was about to say the exact opposite. Every other slide had text that was Just a Little Bit Crooked, and it is driving me __batty__. If the angle were larger, and it weren't used on every other slide, I might think it was cooler (and it IS neat that you can do that with transformations!). As it is, it's really distracting from the message. edit 2: Now that I've read farther, it bothers me less. :-) If I were part of the live audience, I would likey not have noticed. Please don't let the text presentation distract you from the excellent message. :)
- untog 12y ago
  Unfortunately, it looks awful in Firefox.
  - couchand 12y ago
    What version? I'm on 32 on Ubuntu and it looks clean and crisp.
lifeisstillgood 12y ago
This post just touches on so many of my personal "yes dammit yes" buttons I am purring like the kitten in the gif (go have a look) My new boss just gave a piece of advice that marries up very well with this - clients want Stability, Performance and Features - in that order. Consumers want features Then Stability and performance if they like it. So yes - hurry up and put the talk online please Zach - Itching to listen.
- sanderjd 12y ago
  Woah, I love your boss' advice! It is necessary to know your audience before you can prioritize. Also, I don't think consumers want features in the sense of "more", but in the sense of "key". Maybe it is Key Features, Stability, Performance, More Features, in that order.
  - couchand 12y ago
    In my experience clients and consumers really just care about Feature Lists; power users maybe lean more towards Key Features. Performance and Stability are only a concern when they've already become a problem.
    - sanderjd 12y ago
      Really? I don't think consumers have shown at all that they care about (or are even aware of) feature lists. Even facebook is shifting toward a strategy of cobbling together single-feature apps instead of building one big feature-heavy platform.
Fuzzwah 12y ago
As always, I enjoyed reading Zach's thoughts. Spotted one tiny error: http://zachholman.com/images/talks/break-nothing/anpp.jpg http://zachholman.com/images/talks/break-nothing/anpp.jpg annp should be anpp in this image.
- holman 12y ago
  Actually caught this last night and fixed it literally everywhere but the important place: this image. :) Fixed! Thanks for pointing it out.
- deleted 12y ago
  [deleted]
Animats 12y ago
Does the article get any better after the cat video and the mention of Cheetos™? I bailed at that point.
- holman 12y ago
  It doesn't. The author goes on a long, ambiguous tirade about Skittles™ and how they aren't as aerodynamic as once initially thought. You did good by refusing to read the rest of the post and informing everyone of that fact on Hacker News. Cheers!
  - trose 12y ago
    Fantastic response
- therzathegza 12y ago
  Wow, no fun not ever. To author: thanks for the reference to Coda Hale's Metrics talk. https://www.youtube.com/watch?v=czes-oa0yik https://www.youtube.com/watch?v=czes-oa0yik
mehrdada 12y ago
The problem with the new motto is that it is uncontroversial. As Zuck himself once mentioned at Startup School, corporate mottos are valuable only if they are controversial, that is, "be honest" is not a valuable motto, because it is obvious and uncontroversial, whereas "Move Fast and Break Things" is not so: it says something about prioritization that not everyone agrees with, and thereby communicates where the company stands on this issue. Everyone wants to move fast as long as nothing gets broken. I agree that it is possible to leverage technology to move faster while keeping things less broken, but it does not change the fundamental dynamic that at some point, you will have to trade _some_ stability for speed, and the new motto does not take a stance on which one it prefers at all.
- deleted 12y ago
  [deleted]
- pm 12y ago
  Move Fast (and Break Things) is the unwritten motto of every mediocre dev shop out there (breaking things being an unintended side effect of moving fast). It's not controversial.
  - pshin45 12y ago
    > ...is the unwritten motto... Exactly, it's unwritten because no dev shop would explicitly market themselves as being willing to break things in the name of speed. It's an unfortunate side effect that they don't want to call attention to, whereas Facebook wears it as a badge of honor.
    - nahname 12y ago
      Facebook, where the users are the QA department
      - mapleoin 12y ago
        This is the most important point here. Facebook can afford to do this because their users don't care. No one really needs Facebook or depends on it to work at critical times. Most products are not like that. Imagine what happened if Gmail started doing move fast and break things. Oops some of your email vanished.
aresant 12y ago
I thoroughly enjoyed the visual design and layout of this site. That he took the deck and turned it into vibrant and unique feeling content made it feel more special than a normal blog post. Not sure where I'm going with this but I liked the balance of medium.com-like-focus + the design and style content of a NYT special feature. I'm assessing my reaction from a web analysts perspective, just interesting how much more engaged I was w/this from the first second than normal.
- jgmmo 12y ago
  It felt similar to _why's stuff Like this 'Nobody Knows Shoes' book https://cloud.github.com/downloads/shoes/shoes/nks.pdf https://cloud.github.com/downloads/shoes/shoes/nks.pdf
michaelochurch 12y ago
Sorry to be That Guy but this is one of the reasons I'm really getting into Haskell, the more I study it. Clojure's still stronger for exploratory work, but Haskell's the only language where I feel comfortable that I'm breaking nothing when I "move fast" (at least, on code I haven't touched in a while). Ok, I'm totally That Guy so I'm just going to shut up.
- scott_karana 12y ago
  I think you must have missed the point somewhere. Test coverage and metrics are important not just to catch "programming" errors but also logical and semantic errors. Haskell only partially helps with those other categories.
  - michaelochurch 12y ago
    Absolutely true. I think that if you know how to use it, Haskell's type system can reduce your bug count substantially. It makes a difference, spending 30% of your time debugging as opposed to 70%. (These numbers are approximate and vary depending on the problem you're working on.) You still need to test, though. That is true. QuickCheck does a lot for that.
- calibraxis 12y ago
  How do you evaluate Clojure's core.typed and Prismatic Schema? (If you've had a chance.)
  - michaelochurch 12y ago
    I haven't. I tried core.typed a year ago and it was still not very mature, but I'm glad it exists and I'm sure that it has advanced a long way. It's an ambitious project and I'm really glad it funded (I donated).
lifeisstillgood 12y ago
What interests me in companies similar to GitHub is how the organisational politics reflects the code (or coding process) I assume if I knew more about fluid dynamics one could make some decent correlations between blockages, eddies, and viscosity and middle management sign offs, manual testing and IM channels. I guess I am looking for justification that a flat collegiate structure produces higher quality long term code.And some idea how to persuade people to adopt such a format - open, free flowing discussion about code and a data driven "prove it not say it with authority" seems to conflict with hierarchies. Or maybe I am just fed up with organisational politics. :-) If a githuber is on could they comment on how design decisions and business decisions are taken ?
tsunamifury 12y ago
I'm surprised that few seem to understand that Zuckerberg chose a motto with a deeper meaning than 'breaking products and code'. He meant break the systems, calcified behaviors, and system-think that ends up killing a company over the long term. Move fast, and don't go in predictable directions based on existing corporate momentum. The only way to disrupt yourself is to 'break things' i.e. architectures, regular income, etc that may be holding you back from a better opportunity.
- hawkice 12y ago
  This is some deeply Talmudic reading of a fairly straight-forward description of engineering deploy cycles. But why stop after just putting those words in his mouth, when he could be retroactively understood to mean so much more! Move fast! Run to all meetings! Take typing lessons so you move faster while coding! Break things! Break your laptop to get a faster one from the company! Move your laptop fast at the hard surface to break it faster and while moving faster! Some may say, if you keep running around with your laptop, you might actually break it. I call that synergy.
- jzwinck 12y ago
  I agree. From the article: > I can take a pretty obvious guess as of the external manifestations of their new motto: it means they break fewer APIs on their platform. I found this absurd. The idea that "move fast and break things" referred to a reduced level of API compatibility seems much too narrow. Yes, it's understood as a license to break parts of the site in the interest of advancing the product. Facebook did much more than that: it broke/disrupted/redefined our social fabric, the way we understand each other. Along the way it broke promises, expectations, trust, fears, introversion, and boundaries. And it did this very, very quickly--ten years of Facebook and we are all different. Imagine if the Manhattan Project used the same slogan. It's good for some, bad for others, but developed quickly, taken to market, and damn the consequences we're going to figure out what this thing can do.
RickHull 12y ago
I agree that breaking a production user's experience is as verboten as it gets for agile software development. That's why we have massive test suites and CI infrastructure. However, you do want to avoid "CI handcuffs", where either because CI is too brittle or because you're simply integrating too many independent projects which, in order to push a change requiring coordinating changes elsewhere, requires massively outsized developer effort in order to push things through with zero CI breakage. This is more of a problem in CI systems with binary metrics like PASS/FAIL where every change that doesn't PASS is rejected. It's too easy to "wedge" a system like this, where project X's change can't go through without project Y being able to handle the change, and you end up introducing multiple code paths in both projects on a solely temporary basis just in order to keep CI green and happy. Rather than having a zero-tolerance CI failure policy, developers should be allowed to break CI temporarily, so long as they fix it in a timely manner (within an hour or two). Per-developer breakage metrics, to the extent they are needed, should not be in terms of breakage counts but instead breakage durations. That is, outside of production, it's fine to break stuff and quickly fix it, so long as you don't leave it broken. The big problems are where domain-siloed developers break a zero-tolerance policy because it was necessary to relax it "temporarily", and things stay broken because the policy stays relaxed and cannot be reinstated without sirens going off. Then restoring the CI policy is blocked for everyone by the one guy who knows the AIX quirks for debootstrap or whatever. Instead, breaking changes should be "allowed" where there is a window to fix the error and still move forward. Only when the window closes without a fix should the breaking change be rolled back (automatically). This line of thinking lends itself to formal, automated policy, but this depends first on judgment and cultural approval. As it happens, the best CI system I'm aware of for truly distributed, multiple project integration is OpenStack's Zuul: https://github.com/openstack-infra/zuul https://github.com/openstack-infra/zuul I'm not sure if it accommodates my prescription above, but if not, they probably have a better idea.
- skybrian 12y ago
  If both projects share a source control system, you can commit a change and update its downstream usages all at once, with no breakage even temporarily. However, this requires source control that scales company-wide. (Git doesn't qualify.)
S_A_P 12y ago
I think of "move fast and break things" in a different context that most people(maybe). It doesn't say move fast and SHIP broken things. To me it speaks to the fact that you shouldnt be afraid to completely change something if its to improve the product. Eliminate technical debt. Dont code around a workaround that doesnt always work.
malkia 12y ago
... Two code paths (old&new) - I've been doing this to some extent without trying to get any insight, or make it a practice (of my own first), but often the old code just stays before deploying the newer (day or two), or before submission, and then it's killed (or #ifdef-out). I liked the "empathy" bit though. Recently I've did the UCLA Extension TMP courses, and one of the lecturers - Jorge Cherbosque, PhD talked exactly about this and much more - here is the lecture (highly recommended) https://www.uclaextension.edu/tmp/Pages/79th/D1.aspx https://www.uclaextension.edu/tmp/Pages/79th/D1.aspx Is there a video behind the talk?
robgibbons 12y ago
You don't need to break things with good regression testing.
kcorbitt 12y ago
Tangential, but does anyone know what Github uses to match their css and html classes in the integration suite? I often find that I don't clean up my css as the free as I should because I'm scared of breaking another page by mistake, so something that tells me "you're really not using this anywhere anymore" would be great.
- palcu 12y ago
  I don't know what Github uses, but where I work, we do screenshot testing. We have PhantomCSS[0], which is a helper on top of CasperJs that visits pages, takes screenshots and compares them. There are a lot of other tools, Huxley and Selenium come to my mind, that do the same stuff. Even Facebook[1] tests this way. [0]: https://github.com/Huddle/PhantomCSS https://github.com/Huddle/PhantomCSS [1]: https://www.youtube.com/watch?v=VkTCL6Nqm6Y https://www.youtube.com/watch?v=VkTCL6Nqm6Y
hox 12y ago
Twilio follows a similar model for large scale changes, but at a distributed service level using a shadow proxy (https://github.com/twilio/shadow https://github.com/twilio/shadow) rather than having parallel code paths.
lnanek2 12y ago
Really distasteful slogan. I think every engineer who has worked in corp. America is used to management asking for mutually exclusive things due to ignorance.
no_future 12y ago
All this crap is easier said than done.
rgarcia 12y ago
> What do you consider good feedback? How can you promote understanding and positive approaches in your criticism of the code? How can you help the submitter learn and grow from this scenario? Unfortunately these questions don't get asked enough, which creates a self-perpetuating cycle of cynics and aggressive discussion. I'd be really interested to hear how teams codify this. To the extent that it's possible, I think feedback should be rooted in objective measures, e.g. styleguide/lint violations, test failures, etc. All too often, though, there are subjective things that come up: organization of code, interface design, using framework X instead of framework Y. These are the criticisms that most often lead to aggressive back and forths.
- blakehaswell 12y ago
  But those are the most important and interesting discussions to have, so I’m not convinced we should shy away from that sort of feedback. That said, they’re also the sort of discussions which should often happen before code is written. Decisions can be changed much more quickly at that point, and people take it much less personally when you talk about how the design of something can be improved when they haven’t spent the time to implement it. :-)
LeicaLatte 12y ago
Nicely done
davidw 12y ago
This pretty much sums it up: http://en.wikipedia.org/wiki/Project_management_triangle#.22Pick_any_two.22 http://en.wikipedia.org/wiki/Project_management_triangle#.22...
k__ 12y ago
"Move fast with stablity" That's why small companies have a chance. The big ones get slow and focus on different things. They have to change their maxims and their customers don't necessarily like that.
aleem 12y ago
Reading the title, I thought it would be a post about TDD. I don't do TDD, I wish I had the time and resources and I am moving toward adding a more test-driven approach. Adding unit tests to my key libs allows me to move fast without breaking anything (for the key libs, our test suite is stupendously comprehensive). Of course, during new feature dev this luxury isn't always available. The next best thing is to do a phased rollout where our users become our beta testers. This works really well because users can switch to the non-beta version through the menu bar very easily. There are other strategies for moving fast without breaking stuff, but these two provide easy, big wins.
hartzler 12y ago
For me the best line was "You can have the best, most comprehensive test suite in the world, but tests are still different from production.". Getting the business logic working is often the easy part of distributed apps in production environments with real users and unexpected data.
- cpsaltis 12y ago
  While it is true that most tests are different from production, having a solid test suite can go a long way in moving with startup speed but breaking stuff as little as possible. We (as in me and a couple of others from our startup) have written a howto on using Docker and Ansible to build your test suite a few months ago. Strangely enough, it has almost the same title: http://blog.mist.io/post/82383668190/move-fast-and-dont-break-things-testing-with http://blog.mist.io/post/82383668190/move-fast-and-dont-brea...
mcguire 12y ago
Personally, I think this is the most important piece of the article: "...I think the best way to get things done in a company isn't to bash it over your employee's heads every few hours, but to instead build an environment that helps foster those effects."

459 points by bpierre 12y ago

vdm 12y ago

Comparing the results of requests processed against both the new and old system is a pattern. http://www.alwaysagileconsulting.com/application-pattern-verify-branch-by-abstraction/ http://www.alwaysagileconsulting.com/application-pattern-ver... It was also done in mainframes at the processor instruction level.

danso 12y ago

Quite a bit OT, but I've never seen the transform-rotate property used as a styling in text...probably something that can/will be used to death, but it looks great here...gives the piece a very magazine feel (that, plus the handplaced-layout of floating embed images)

gknoy 12y ago
I was about to say the exact opposite. Every other slide had text that was Just a Little Bit Crooked, and it is driving me __batty__. If the angle were larger, and it weren't used on every other slide, I might think it was cooler (and it IS neat that you can do that with transformations!). As it is, it's really distracting from the message. edit 2: Now that I've read farther, it bothers me less. :-) If I were part of the live audience, I would likey not have noticed. Please don't let the text presentation distract you from the excellent message. :)
untog 12y ago
Unfortunately, it looks awful in Firefox.
- couchand 12y ago
  What version? I'm on 32 on Ubuntu and it looks clean and crisp.

lifeisstillgood 12y ago

This post just touches on so many of my personal "yes dammit yes" buttons I am purring like the kitten in the gif (go have a look) My new boss just gave a piece of advice that marries up very well with this - clients want Stability, Performance and Features - in that order. Consumers want features Then Stability and performance if they like it. So yes - hurry up and put the talk online please Zach - Itching to listen.

sanderjd 12y ago
Woah, I love your boss' advice! It is necessary to know your audience before you can prioritize. Also, I don't think consumers want features in the sense of "more", but in the sense of "key". Maybe it is Key Features, Stability, Performance, More Features, in that order.
- couchand 12y ago
  In my experience clients and consumers really just care about Feature Lists; power users maybe lean more towards Key Features. Performance and Stability are only a concern when they've already become a problem.
  - sanderjd 12y ago
    Really? I don't think consumers have shown at all that they care about (or are even aware of) feature lists. Even facebook is shifting toward a strategy of cobbling together single-feature apps instead of building one big feature-heavy platform.

Fuzzwah 12y ago

As always, I enjoyed reading Zach's thoughts. Spotted one tiny error: http://zachholman.com/images/talks/break-nothing/anpp.jpg http://zachholman.com/images/talks/break-nothing/anpp.jpg annp should be anpp in this image.

holman 12y ago
Actually caught this last night and fixed it literally everywhere but the important place: this image. :) Fixed! Thanks for pointing it out.
deleted 12y ago
[deleted]

Animats 12y ago

Does the article get any better after the cat video and the mention of Cheetos™? I bailed at that point.

holman 12y ago
It doesn't. The author goes on a long, ambiguous tirade about Skittles™ and how they aren't as aerodynamic as once initially thought. You did good by refusing to read the rest of the post and informing everyone of that fact on Hacker News. Cheers!
- trose 12y ago
  Fantastic response
therzathegza 12y ago
Wow, no fun not ever. To author: thanks for the reference to Coda Hale's Metrics talk. https://www.youtube.com/watch?v=czes-oa0yik https://www.youtube.com/watch?v=czes-oa0yik

mehrdada 12y ago

The problem with the new motto is that it is uncontroversial. As Zuck himself once mentioned at Startup School, corporate mottos are valuable only if they are controversial, that is, "be honest" is not a valuable motto, because it is obvious and uncontroversial, whereas "Move Fast and Break Things" is not so: it says something about prioritization that not everyone agrees with, and thereby communicates where the company stands on this issue. Everyone wants to move fast as long as nothing gets broken. I agree that it is possible to leverage technology to move faster while keeping things less broken, but it does not change the fundamental dynamic that at some point, you will have to trade _some_ stability for speed, and the new motto does not take a stance on which one it prefers at all.

deleted 12y ago
[deleted]
pm 12y ago
Move Fast (and Break Things) is the unwritten motto of every mediocre dev shop out there (breaking things being an unintended side effect of moving fast). It's not controversial.
- pshin45 12y ago
  > ...is the unwritten motto... Exactly, it's unwritten because no dev shop would explicitly market themselves as being willing to break things in the name of speed. It's an unfortunate side effect that they don't want to call attention to, whereas Facebook wears it as a badge of honor.
  - nahname 12y ago
    Facebook, where the users are the QA department
    - mapleoin 12y ago
      This is the most important point here. Facebook can afford to do this because their users don't care. No one really needs Facebook or depends on it to work at critical times. Most products are not like that. Imagine what happened if Gmail started doing move fast and break things. Oops some of your email vanished.

aresant 12y ago

I thoroughly enjoyed the visual design and layout of this site. That he took the deck and turned it into vibrant and unique feeling content made it feel more special than a normal blog post. Not sure where I'm going with this but I liked the balance of medium.com-like-focus + the design and style content of a NYT special feature. I'm assessing my reaction from a web analysts perspective, just interesting how much more engaged I was w/this from the first second than normal.

jgmmo 12y ago
It felt similar to _why's stuff Like this 'Nobody Knows Shoes' book https://cloud.github.com/downloads/shoes/shoes/nks.pdf https://cloud.github.com/downloads/shoes/shoes/nks.pdf

michaelochurch 12y ago

Sorry to be That Guy but this is one of the reasons I'm really getting into Haskell, the more I study it. Clojure's still stronger for exploratory work, but Haskell's the only language where I feel comfortable that I'm breaking nothing when I "move fast" (at least, on code I haven't touched in a while). Ok, I'm totally That Guy so I'm just going to shut up.

scott_karana 12y ago
I think you must have missed the point somewhere. Test coverage and metrics are important not just to catch "programming" errors but also logical and semantic errors. Haskell only partially helps with those other categories.
- michaelochurch 12y ago
  Absolutely true. I think that if you know how to use it, Haskell's type system can reduce your bug count substantially. It makes a difference, spending 30% of your time debugging as opposed to 70%. (These numbers are approximate and vary depending on the problem you're working on.) You still need to test, though. That is true. QuickCheck does a lot for that.
calibraxis 12y ago
How do you evaluate Clojure's core.typed and Prismatic Schema? (If you've had a chance.)
- michaelochurch 12y ago
  I haven't. I tried core.typed a year ago and it was still not very mature, but I'm glad it exists and I'm sure that it has advanced a long way. It's an ambitious project and I'm really glad it funded (I donated).

lifeisstillgood 12y ago

What interests me in companies similar to GitHub is how the organisational politics reflects the code (or coding process) I assume if I knew more about fluid dynamics one could make some decent correlations between blockages, eddies, and viscosity and middle management sign offs, manual testing and IM channels. I guess I am looking for justification that a flat collegiate structure produces higher quality long term code.And some idea how to persuade people to adopt such a format - open, free flowing discussion about code and a data driven "prove it not say it with authority" seems to conflict with hierarchies. Or maybe I am just fed up with organisational politics. :-) If a githuber is on could they comment on how design decisions and business decisions are taken ?

tsunamifury 12y ago

I'm surprised that few seem to understand that Zuckerberg chose a motto with a deeper meaning than 'breaking products and code'. He meant break the systems, calcified behaviors, and system-think that ends up killing a company over the long term. Move fast, and don't go in predictable directions based on existing corporate momentum. The only way to disrupt yourself is to 'break things' i.e. architectures, regular income, etc that may be holding you back from a better opportunity.

hawkice 12y ago
This is some deeply Talmudic reading of a fairly straight-forward description of engineering deploy cycles. But why stop after just putting those words in his mouth, when he could be retroactively understood to mean so much more! Move fast! Run to all meetings! Take typing lessons so you move faster while coding! Break things! Break your laptop to get a faster one from the company! Move your laptop fast at the hard surface to break it faster and while moving faster! Some may say, if you keep running around with your laptop, you might actually break it. I call that synergy.
jzwinck 12y ago
I agree. From the article: > I can take a pretty obvious guess as of the external manifestations of their new motto: it means they break fewer APIs on their platform. I found this absurd. The idea that "move fast and break things" referred to a reduced level of API compatibility seems much too narrow. Yes, it's understood as a license to break parts of the site in the interest of advancing the product. Facebook did much more than that: it broke/disrupted/redefined our social fabric, the way we understand each other. Along the way it broke promises, expectations, trust, fears, introversion, and boundaries. And it did this very, very quickly--ten years of Facebook and we are all different. Imagine if the Manhattan Project used the same slogan. It's good for some, bad for others, but developed quickly, taken to market, and damn the consequences we're going to figure out what this thing can do.

RickHull 12y ago

I agree that breaking a production user's experience is as verboten as it gets for agile software development. That's why we have massive test suites and CI infrastructure. However, you do want to avoid "CI handcuffs", where either because CI is too brittle or because you're simply integrating too many independent projects which, in order to push a change requiring coordinating changes elsewhere, requires massively outsized developer effort in order to push things through with zero CI breakage. This is more of a problem in CI systems with binary metrics like PASS/FAIL where every change that doesn't PASS is rejected. It's too easy to "wedge" a system like this, where project X's change can't go through without project Y being able to handle the change, and you end up introducing multiple code paths in both projects on a solely temporary basis just in order to keep CI green and happy. Rather than having a zero-tolerance CI failure policy, developers should be allowed to break CI temporarily, so long as they fix it in a timely manner (within an hour or two). Per-developer breakage metrics, to the extent they are needed, should not be in terms of breakage counts but instead breakage durations. That is, outside of production, it's fine to break stuff and quickly fix it, so long as you don't leave it broken. The big problems are where domain-siloed developers break a zero-tolerance policy because it was necessary to relax it "temporarily", and things stay broken because the policy stays relaxed and cannot be reinstated without sirens going off. Then restoring the CI policy is blocked for everyone by the one guy who knows the AIX quirks for debootstrap or whatever. Instead, breaking changes should be "allowed" where there is a window to fix the error and still move forward. Only when the window closes without a fix should the breaking change be rolled back (automatically). This line of thinking lends itself to formal, automated policy, but this depends first on judgment and cultural approval. As it happens, the best CI system I'm aware of for truly distributed, multiple project integration is OpenStack's Zuul: https://github.com/openstack-infra/zuul https://github.com/openstack-infra/zuul I'm not sure if it accommodates my prescription above, but if not, they probably have a better idea.

skybrian 12y ago
If both projects share a source control system, you can commit a change and update its downstream usages all at once, with no breakage even temporarily. However, this requires source control that scales company-wide. (Git doesn't qualify.)

S_A_P 12y ago

I think of "move fast and break things" in a different context that most people(maybe). It doesn't say move fast and SHIP broken things. To me it speaks to the fact that you shouldnt be afraid to completely change something if its to improve the product. Eliminate technical debt. Dont code around a workaround that doesnt always work.

malkia 12y ago

... Two code paths (old&new) - I've been doing this to some extent without trying to get any insight, or make it a practice (of my own first), but often the old code just stays before deploying the newer (day or two), or before submission, and then it's killed (or #ifdef-out). I liked the "empathy" bit though. Recently I've did the UCLA Extension TMP courses, and one of the lecturers - Jorge Cherbosque, PhD talked exactly about this and much more - here is the lecture (highly recommended) https://www.uclaextension.edu/tmp/Pages/79th/D1.aspx https://www.uclaextension.edu/tmp/Pages/79th/D1.aspx Is there a video behind the talk?

robgibbons 12y ago

You don't need to break things with good regression testing.

kcorbitt 12y ago

Tangential, but does anyone know what Github uses to match their css and html classes in the integration suite? I often find that I don't clean up my css as the free as I should because I'm scared of breaking another page by mistake, so something that tells me "you're really not using this anywhere anymore" would be great.

palcu 12y ago
I don't know what Github uses, but where I work, we do screenshot testing. We have PhantomCSS[0], which is a helper on top of CasperJs that visits pages, takes screenshots and compares them. There are a lot of other tools, Huxley and Selenium come to my mind, that do the same stuff. Even Facebook[1] tests this way. [0]: https://github.com/Huddle/PhantomCSS https://github.com/Huddle/PhantomCSS [1]: https://www.youtube.com/watch?v=VkTCL6Nqm6Y https://www.youtube.com/watch?v=VkTCL6Nqm6Y

hox 12y ago

Twilio follows a similar model for large scale changes, but at a distributed service level using a shadow proxy (https://github.com/twilio/shadow https://github.com/twilio/shadow) rather than having parallel code paths.

lnanek2 12y ago

Really distasteful slogan. I think every engineer who has worked in corp. America is used to management asking for mutually exclusive things due to ignorance.

no_future 12y ago

All this crap is easier said than done.

rgarcia 12y ago

> What do you consider good feedback? How can you promote understanding and positive approaches in your criticism of the code? How can you help the submitter learn and grow from this scenario? Unfortunately these questions don't get asked enough, which creates a self-perpetuating cycle of cynics and aggressive discussion. I'd be really interested to hear how teams codify this. To the extent that it's possible, I think feedback should be rooted in objective measures, e.g. styleguide/lint violations, test failures, etc. All too often, though, there are subjective things that come up: organization of code, interface design, using framework X instead of framework Y. These are the criticisms that most often lead to aggressive back and forths.

blakehaswell 12y ago
But those are the most important and interesting discussions to have, so I’m not convinced we should shy away from that sort of feedback. That said, they’re also the sort of discussions which should often happen before code is written. Decisions can be changed much more quickly at that point, and people take it much less personally when you talk about how the design of something can be improved when they haven’t spent the time to implement it. :-)

LeicaLatte 12y ago

Nicely done

davidw 12y ago

This pretty much sums it up: http://en.wikipedia.org/wiki/Project_management_triangle#.22Pick_any_two.22 http://en.wikipedia.org/wiki/Project_management_triangle#.22...

k__ 12y ago

"Move fast with stablity" That's why small companies have a chance. The big ones get slow and focus on different things. They have to change their maxims and their customers don't necessarily like that.

aleem 12y ago

Reading the title, I thought it would be a post about TDD. I don't do TDD, I wish I had the time and resources and I am moving toward adding a more test-driven approach. Adding unit tests to my key libs allows me to move fast without breaking anything (for the key libs, our test suite is stupendously comprehensive). Of course, during new feature dev this luxury isn't always available. The next best thing is to do a phased rollout where our users become our beta testers. This works really well because users can switch to the non-beta version through the menu bar very easily. There are other strategies for moving fast without breaking stuff, but these two provide easy, big wins.

hartzler 12y ago

For me the best line was "You can have the best, most comprehensive test suite in the world, but tests are still different from production.". Getting the business logic working is often the easy part of distributed apps in production environments with real users and unexpected data.

cpsaltis 12y ago
While it is true that most tests are different from production, having a solid test suite can go a long way in moving with startup speed but breaking stuff as little as possible. We (as in me and a couple of others from our startup) have written a howto on using Docker and Ansible to build your test suite a few months ago. Strangely enough, it has almost the same title: http://blog.mist.io/post/82383668190/move-fast-and-dont-break-things-testing-with http://blog.mist.io/post/82383668190/move-fast-and-dont-brea...

mcguire 12y ago

Personally, I think this is the most important piece of the article: "...I think the best way to get things done in a company isn't to bash it over your employee's heads every few hours, but to instead build an environment that helps foster those effects."