8 ms·

Move Fast without Breaking Things

86 points by devy 11y ago

cpfohl 11y ago
Great article! > Be Aware of What You Break I work at Rollbar (www.rollbar.com), and that point is probably the key to our existence. You can't afford not to know what's going wrong. And the best way to know what's going on is to get a full stack trace with all the information you need to reproduce into a platform that will notify the appropriate people of the breakage.
ktRolster 11y ago
A lot of the bugs in our code are completely avoidable, but people get in the mindset of, "I need to move fast, bugs don't matter!" Even if you don't have QA, when you write a bug, at least ask yourself, "what could I have done differently to avoid that bug in the future?" In some cases, it's as simple as, "put repeated code into a function so it doesn't get written twice." That will cut your chances of getting a typo in half.
- ArkyBeagle 11y ago
  So , really - you actually see typo bugs in the wild? I haven't seen one of those for 20 years. I do see lots of cases where people didn't test it or used the wrong mechanism. I have to go slowly because somebody already went fast and broke it. And a friend once told me "LZW can compress code faster than you can." :) Which does not go to the value of orthogonality of purpose in source code, but I enjoyed the irony of the comment very much.
  - collyw 11y ago
    Probably happens more frequently than you think in dynamic languages. Even in compiled languages I imagine its reasonably common to compare something to a hard coded string that could easily be mistyped.
    - ArkyBeagle 11y ago
      I wasn't thinking of dynamic languages at all. I'd agree.
  - chipsy 11y ago
    Math-heavy code tends to involve a lot of very similar algorithms, leading to a copy-paste-rename situation that isn't really helped by using a function. As such it's incredibly common to end up with bugs from assigning to "x, y, y" and not "x, y, z" etc. Sometimes a clever compiler will warn you that you have done something odd, other times you'll be left to discover the runtime error.
    - ArkyBeagle 11y ago
      Yeah, there's that. I've printed out code like that and colored different variables with different color highlighters to deal with it.
  - ktRolster 11y ago
    Here's one more recent than 20 years: http://www.theregister.co.uk/2009/07/30/typo_caused_massive_ms_bug/ http://www.theregister.co.uk/2009/07/30/typo_caused_massive_... You're right though, with care and skill, typos really shouldn't be an issue
- sbov 11y ago
  This seems like a really uncommon case. The vast majority of bugs I've ran across is an incorrect mental model of how something works: your system, some library, or an external system you're interfacing with. Aside from building a more understandable system (whatever that really means/entails), from my experience, the best way to counteract this is to, while writing code, always ask: "What if my understanding is wrong? What if someone else's understanding is wrong? What can I do to make the system fail in the loudest way possible if someone's understanding is wrong?"
  - ArkyBeagle 11y ago
    Generally, have the interface to the module under utilization be as simple as heck. As an example, I have a system for which almost all state is represented as XML/JSON/CSV text when it is outside the system. Inside the system it's tables of tuples with a master table of names cross references. Each "object" has an instantiator, a "process" callback and ... that's it. It's all done in a timer-based polling cycle ( after a select()/epoll() loop ). Each "object" owns a timer of its own that specifies the minimum poll time. There basically are no parameters after object creation, so you can't get parameters wrong. You have to send XML/JSON/CSV to it to change state. XML/JSON/CSV for initial configuration is controlled and shipped with the executable. For all settable state in the tables, there is a formal validation method and a controlled script to test them. The the client-writers, I provide correct scripts for each use case that they can then steal or modify. This way, I just don't have problems.
  - ktRolster 11y ago
    That goes with what Glenford Meyers said, that all bugs are transcription bugs, either from the requirements to the programmer, or from the programmers head to his fingers, or.....
thinkingkong 11y ago
I think its weird that weve put stability and speed on the same axis. The reality is that you can ship faster with more confidence by focusing on stability from the beginning. Simply slapping some code together and going "it works!" is - in isolation - a recipe for disaster.
- digi_owl 11y ago
  Its because on the web patches can be pushed as quickly as the user hits refresh, or at least so seem the thinking to be. The real problem comes when the same mentality is applied to software running locally, perhaps with patches never applied after initial install. This likely because developers are now so used to being always connected (this thanks to thinks like GIT that allow them to work from just about anywhere with a power outlet and a net connection), they can't grasp that not everything else is.
- nostrademons 11y ago
  Only if the requirements are known. If you have firm requirements, you can test for them, you can plan for them, and you move through them much faster when you don't get surprised by random bugs that need to be tracked down. If you don't have firm requirements, then each new fact you learn about your market potentially means you may have to invalidate large swaths of work you've already done. The more planning and testing you've done, and the more thoroughly you've covered your edge cases, the more there is to invalidate. Most of the big business ideas in the last decade were in areas of extreme market uncertainty. Hence, companies that "move fast and break things" have been at a big advantage. This may change in the future - VCs today are enamored with Perez, and IMHO one of the signals that we've crossed over from the installation to deployment phase of the Perez model is when performance, stability, and security become valued more than features & speed of execution. But I'd guess that we have a few more years of moving fast and breaking things first.
  - st3v3r 11y ago
    Having tests can help you ensure that, when you do change one part of your application, unrelated parts don't start breaking. And I don't really buy the argument that, because something might be invalidated later, we shouldn't put in the time now to make sure we get it right based on the info we do have.
JBiserkov 11y ago
Possibly related: "Move fast and fix things" by GitHub engineering. https://news.ycombinator.com/item?id=10739129 https://news.ycombinator.com/item?id=10739129
- deleted 11y ago
  [deleted]
teacup50 11y ago
I'm actually very happy to work somewhere that believes in "Move slow and don't break things". I believe that churn isn't progress, things with lasting value generally take time, and some problems can't be solved quickly. I enjoy working on problems that benefit from careful, considered, time-consuming thought. You might desire something different. That's OK, but nobody is obligated to accept fait accompli the universal necessity or value of moving "fast" (and hopefully not breaking things).
- vinceguidry 11y ago
  Moving slow isn't an option for many businesses.
  - teacup50 11y ago
    I believe it's an option far more often than people believe. Customers do follow dependability and lasting value; nobody trying to their solve day-to-day problems actually wants a treadmill.
    - vinceguidry 11y ago
      It's one of those cultural decisions you have to make early on in a company's formation and make sacrifices to hold on to because literally everyone else you're connected with will expect you to move at the same speed they're moving at.
  - Terr_ 11y ago
    Well, let's make sure we distinguish between what's viable for the company as a whole versus what's viable for the career-advancement of an individual executive :p
    - vinceguidry 11y ago
      If the firm hasn't yet succumbed to mediocrity, then the career goals of each executive should be well-aligned with overall company strategy. Ensuring that this is the case is a primary goal of executive recruiting. Middle managers and ICs, not necessarily, but a company generally can't afford to have its executives be out of step.
jbob2000 11y ago
I think this article misses what is meant by "Move Fast and Break Things". I never took it to mean 'physically (or programmatically) break things apart while moving quickly', I thought it was more about deconstructing things we considered "normal". Like another way of saying "think outside the box", and do that without hesitation or inhibition.
- icebraining 11y ago
  “Most companies mess up by moving too slowly and trying to be too precise. When you’re…moving quickly or doing anything like this you want to make mistakes evenly on both sides. We wanted to set up a culture so that we were equally messing up by moving too quickly and by moving too slowly some of the time. So that way, we’d know that we were in the middle.” https://www.youtube.com/watch?v=h7_UNu7zEVs https://www.youtube.com/watch?v=h7_UNu7zEVs
- digi_owl 11y ago
  My interpretation of it is to say that web devs should not be afraid of introducing bugs while implementing new features, as patches can be pushed just as fast. This may be true for a web site (or app as some like to call them these days). But the mentality has filtered down to actual software running locally, or even whole OSes (see Android for instance, where more than once Google has pushed a feature into the world before all the proverbial edges have been filed down).
swalsh 11y ago
You ever Notice how the engineering departments at these move fast and break things companies grow exponentially? Thinking of etsy, and Facebook, and a few others. Part of it is if they survive, they thrived... But I think the other part is that the code starts to become sharded across people's brains. The technical debt piles up so high in some places that it becomes impossible to maintain productivity with the same number of people.
- debacle 11y ago
  I think that's a non sequitur. Every startup engineering department grows exponentially. It's the only way for an engineering department to grow.
julie1 11y ago
Well, in physics we compute the speed with half life of what we want to keep alive regarding the costs. Mass transportation is an example of industry where half life is decades/century. Fast for this sector is slower than what the life expectancy of a coder represents. Games made to support a product ad are ephemeral, 1 month. Faster than a new release of firefox. Some dangerous radioactive isotopes are 10000 years. So nuclear plant related automation should not be using any CPU. A citizen data should last for the government as long a citizen live. Government if they aim at remaining stable should consider the half life of their automation to be long, way long. Hence slow changes. Very slow So do banks. But banks have to adapt with the speed of trades. Fast and slow can be set by doing something agile hates: careful business analysis. The problem is how we coupled all the businesses with insane costly rhythm that are forced down on every economical activities by the mean of noncompetitive business practices. Like useless obsolescence driven by HW monopolies, cheap energy, cheap regulated and poor education. Do we really need 1 Pb hard drive when entropy predicts we will not be able to find any relevant information given we have to much data without increasing the means involves in costly ways? We don't need fast changing technologies, we need boring, slow changing technologies that are reliable.
grandalf 11y ago
Move fast and break things for 0.001% of users. In other words, try bold things, bold refactorings, etc. within bounds. If a system is designed with good modularity, this can be done with very low risk. When someone deploys new code at Facebook, I recall reading that it initially goes out to internal users, then to a small percentage of actual users, then eventually to the entire system. This is simply a systems-based approach. The same applies when thinking about code in a testing environment, staging server, etc. At each level there are additional failure modes or adverse interactions. When done stupidly, a cowboy approach can lead to downtime, poorly-reasoned quick fixes, and blame within an institution. Often the same cowboy/girl who acted stupidly and caused the bug is lauded as a hero when he gets things working again a week later after the bug is discovered... when it was his/her bad judgment (or "move fast" mentality) that led the offending code to be shipped in the first place. We must be honest with ourselves and our teams about bad decisions (even in hindsight) and build processes that let us be bold when we need to.
white-flame 11y ago
Racing a car has a great analogy: In order to get faster lap times, you sometimes have to drive slower. Specifically, you need to slow down for corners in order to maintain your control over the vehicle. Try to drive too fast and you end up wrecked. Even if you don't wreck, correcting from having entered a corner too fast loses far more time than just slowing down properly. Similarly to technical debt, if you start a straightaway from a slower starting speed because you botched the prior corner, that damages your lap time throughout the entire stretch. In the overall scheme of being able to deliver software you're confident in, it is faster to consider stability inherently in your development process than try to blast out features and bolt on stability later once you're committed to the initial brittle implementation.
- nibs 11y ago
  Racing a car applies in two ways I think: the talent is not evenly distributed, and the masters are defined by their ability to decide when to go fast, and when to go slow.
  - sharps_xp 11y ago
    I agree. To be more explicit, the talent is indirectly correlated with how slow your team's "slow" is going to be. I think the degree of slow-ness is what management ultimately feels when they conduct retrospectives on technical debt reduction initiatives. We recently did a major refactor of our angular code (~1.5 months) and we've been pretty fast, releasing every 2 weeks. We've begun to outpace a lot of the backend development teams, so we now have more time to squash our backlog tickets. We have the prettiest Jira issue reports out of all teams.
gavinpc 11y ago
> In reality more often then not moving fast and breaking things will result in shipping scrappy software. This is because in the rush to get stuff out the fastest way time consuming things get skipped. Like user testing, automation, analytics, monitoring, manual testing - just to name a few. It sounds like the author could use a dose of his own medicine. The first sentence contains a (common) grammatical error. The third sentence is a fragment. The second is nearly unparseable.
mgrennan 11y ago
I'm not worried about the "Move Fast and Break Things Often" mentality. This will only last until it kills a much of people. Think code in cars, plans, X-ray machines, pill dispensers etc.
agentultra 11y ago
I'd add: think before you code. Programming is hard because thinking is hard. If you don't use tools to help you think rigorously you're going to ship software with bugs regardless of how good your unit tests are or any other strategy you use to cope with poorly designed software. I don't believe not thinking is going to help you move faster. You're just going to spend most of your time fixing the problems you hadn't thought of before you started shipping code. And yet this is what we're encouraged to do. We're asked to fix the bugs, do the least amount of work to get it to go, and fix the problems later. We're asked to not think and just code. There are probably a handful of people in the world who can write complex systems in code with consistently correct results. The rest of us need to be more mindful. We have wonderful tools available to us that can exhaustively check our designs. Using a system like TLA+ allows you to model your system in some pseudo-code and comes with a program that will exhaustively check your code for dead-locks, invariant violations, termination, etc. I've seen it used to find errors in graduate students' binary search algorithms 9 out of 10 times. Imagine what it can do for non-trivial components of your application. Part of moving fast is knowing how to avoid making errors in the first place. The only way I know how to do that is to think. You can't do this properly in code alone. You need higher-level tools to check your assumptions and find holes in your thinking. You don't see structural engineers slapping together the first thing that works and patching the building later when parts of it fall down. You don't sign waivers before crossing a bridge that claim no responsibility on behalf of the designers if it falls down while you're on it. Nor do you see engineers accepting more money from people to build private bridges that are less likely to fall down. So why do software engineers have no such liability for their creations? Why not take a few hours to write a proper specification for your distributed locking mechanism and save yourself days or weeks worth of debugging -- or worse -- liability suits when your mistakes harm the interests of your customers or the public at large? Why not use tools that check your thinking so that you can chase more interesting optimization and performance problems?
- matt_wulfeck 11y ago
  Maybe it's different for others, but for me the true nature of the problem sometimes only surfaces when you start coding. I can think forever and write the perfect spec, but when the rubber hits the road then things change. I think the reality is a virtuous cycle of thinking, coding, rethinking, recoding that gets the best results.
Animats 11y ago
Von Braun's team launched over 600 V-2 rockets before any of them reached a militarily useful target.
bobbygoodlatte 11y ago
I think this post misses the point. Company values are only meaningful when they differentiate that company from others. Everyone wants to "Move Fast and Not Break Things". "Move Fast and Break Things" was a value judgement. For Facebook, moving quickly was worth the penalty of sometimes breaking things. This is why most companies have lame values like "Do your best work" or "Be honest". Great values, but nobody disagrees with them—so they're meaningless.
- wdewind 11y ago
  I totally agree with this but would add a bit of nuance. I'd love to see all companies do a "X and the cost of Y" value system, but there is another part of stating you have values like "Integrity" which is the same as the Google "Don't be evil" thing. It's a claim that you can be called publicly on, and that's important as well. You'd be surprised how many companies out there aren't willing to make a claim for something like integrity. But yeah, who wouldn't say they value integrity, for instance, when asked? That being said, I'd also add that most values have those implicitly baked in. For instance, if you do say you value integrity, you're saying you're devaluing growth/money for the value/benefit of integrity.
  - bobbygoodlatte 11y ago
    Yep, I agree with your nuance. FWIW I think integrity is a pretty okay company value. Depending on your industry, valuing integrity may be a trade-off that other companies are unwilling to make
frogpelt 11y ago
Facebook/Zuckerberg have obviously been successful because this particular motto has been dissected countless times in blog posts and HN comments. Try this search: https://hn.algolia.com/?q=fast+break https://hn.algolia.com/?q=fast+break
aledesma 11y ago
I have found that using BDD with gherkin syntax can really cut down on many conserns when it comes to software stability and ensuring business worth. Many companies do not put enough focus on automated testing and even when they do they are only testing the surface level with simple funtional tests. If one simply throws the technical requirements into the BDD tests and have your system validate every affected integration prior to shipping then you simply have less debt to pay off as more bugs should be caught prior to shipping. Couple BDD with writing a negative functional test for every bug that is fixed and you should never trigger that specific bug again. In the end these tests will take more time, but if that time is bundled into the time required for the feature, and your code is already layed out in a testable and reusable way, then it is win-win.

86 points by devy 11y ago

Great article! > Be Aware of What You Break I work at Rollbar (www.rollbar.com), and that point is probably the key to our existence. You can't afford not to know what's going wrong. And the best way to know what's going on is to get a full stack trace with all the information you need to reproduce into a platform that will notify the appropriate people of the breakage.

ktRolster 11y ago

A lot of the bugs in our code are completely avoidable, but people get in the mindset of, "I need to move fast, bugs don't matter!" Even if you don't have QA, when you write a bug, at least ask yourself, "what could I have done differently to avoid that bug in the future?" In some cases, it's as simple as, "put repeated code into a function so it doesn't get written twice." That will cut your chances of getting a typo in half.

ArkyBeagle 11y ago
So , really - you actually see typo bugs in the wild? I haven't seen one of those for 20 years. I do see lots of cases where people didn't test it or used the wrong mechanism. I have to go slowly because somebody already went fast and broke it. And a friend once told me "LZW can compress code faster than you can." :) Which does not go to the value of orthogonality of purpose in source code, but I enjoyed the irony of the comment very much.
- collyw 11y ago
  Probably happens more frequently than you think in dynamic languages. Even in compiled languages I imagine its reasonably common to compare something to a hard coded string that could easily be mistyped.
  - ArkyBeagle 11y ago
    I wasn't thinking of dynamic languages at all. I'd agree.
- chipsy 11y ago
  Math-heavy code tends to involve a lot of very similar algorithms, leading to a copy-paste-rename situation that isn't really helped by using a function. As such it's incredibly common to end up with bugs from assigning to "x, y, y" and not "x, y, z" etc. Sometimes a clever compiler will warn you that you have done something odd, other times you'll be left to discover the runtime error.
  - ArkyBeagle 11y ago
    Yeah, there's that. I've printed out code like that and colored different variables with different color highlighters to deal with it.
- ktRolster 11y ago
  Here's one more recent than 20 years: http://www.theregister.co.uk/2009/07/30/typo_caused_massive_ms_bug/ http://www.theregister.co.uk/2009/07/30/typo_caused_massive_... You're right though, with care and skill, typos really shouldn't be an issue
sbov 11y ago
This seems like a really uncommon case. The vast majority of bugs I've ran across is an incorrect mental model of how something works: your system, some library, or an external system you're interfacing with. Aside from building a more understandable system (whatever that really means/entails), from my experience, the best way to counteract this is to, while writing code, always ask: "What if my understanding is wrong? What if someone else's understanding is wrong? What can I do to make the system fail in the loudest way possible if someone's understanding is wrong?"
- ArkyBeagle 11y ago
  Generally, have the interface to the module under utilization be as simple as heck. As an example, I have a system for which almost all state is represented as XML/JSON/CSV text when it is outside the system. Inside the system it's tables of tuples with a master table of names cross references. Each "object" has an instantiator, a "process" callback and ... that's it. It's all done in a timer-based polling cycle ( after a select()/epoll() loop ). Each "object" owns a timer of its own that specifies the minimum poll time. There basically are no parameters after object creation, so you can't get parameters wrong. You have to send XML/JSON/CSV to it to change state. XML/JSON/CSV for initial configuration is controlled and shipped with the executable. For all settable state in the tables, there is a formal validation method and a controlled script to test them. The the client-writers, I provide correct scripts for each use case that they can then steal or modify. This way, I just don't have problems.
- ktRolster 11y ago
  That goes with what Glenford Meyers said, that all bugs are transcription bugs, either from the requirements to the programmer, or from the programmers head to his fingers, or.....

thinkingkong 11y ago

I think its weird that weve put stability and speed on the same axis. The reality is that you can ship faster with more confidence by focusing on stability from the beginning. Simply slapping some code together and going "it works!" is - in isolation - a recipe for disaster.

digi_owl 11y ago
Its because on the web patches can be pushed as quickly as the user hits refresh, or at least so seem the thinking to be. The real problem comes when the same mentality is applied to software running locally, perhaps with patches never applied after initial install. This likely because developers are now so used to being always connected (this thanks to thinks like GIT that allow them to work from just about anywhere with a power outlet and a net connection), they can't grasp that not everything else is.
nostrademons 11y ago
Only if the requirements are known. If you have firm requirements, you can test for them, you can plan for them, and you move through them much faster when you don't get surprised by random bugs that need to be tracked down. If you don't have firm requirements, then each new fact you learn about your market potentially means you may have to invalidate large swaths of work you've already done. The more planning and testing you've done, and the more thoroughly you've covered your edge cases, the more there is to invalidate. Most of the big business ideas in the last decade were in areas of extreme market uncertainty. Hence, companies that "move fast and break things" have been at a big advantage. This may change in the future - VCs today are enamored with Perez, and IMHO one of the signals that we've crossed over from the installation to deployment phase of the Perez model is when performance, stability, and security become valued more than features & speed of execution. But I'd guess that we have a few more years of moving fast and breaking things first.
- st3v3r 11y ago
  Having tests can help you ensure that, when you do change one part of your application, unrelated parts don't start breaking. And I don't really buy the argument that, because something might be invalidated later, we shouldn't put in the time now to make sure we get it right based on the info we do have.

JBiserkov 11y ago

Possibly related: "Move fast and fix things" by GitHub engineering. https://news.ycombinator.com/item?id=10739129 https://news.ycombinator.com/item?id=10739129

deleted 11y ago
[deleted]

teacup50 11y ago

I'm actually very happy to work somewhere that believes in "Move slow and don't break things". I believe that churn isn't progress, things with lasting value generally take time, and some problems can't be solved quickly. I enjoy working on problems that benefit from careful, considered, time-consuming thought. You might desire something different. That's OK, but nobody is obligated to accept fait accompli the universal necessity or value of moving "fast" (and hopefully not breaking things).

vinceguidry 11y ago
Moving slow isn't an option for many businesses.
- teacup50 11y ago
  I believe it's an option far more often than people believe. Customers do follow dependability and lasting value; nobody trying to their solve day-to-day problems actually wants a treadmill.
  - vinceguidry 11y ago
    It's one of those cultural decisions you have to make early on in a company's formation and make sacrifices to hold on to because literally everyone else you're connected with will expect you to move at the same speed they're moving at.
- Terr_ 11y ago
  Well, let's make sure we distinguish between what's viable for the company as a whole versus what's viable for the career-advancement of an individual executive :p
  - vinceguidry 11y ago
    If the firm hasn't yet succumbed to mediocrity, then the career goals of each executive should be well-aligned with overall company strategy. Ensuring that this is the case is a primary goal of executive recruiting. Middle managers and ICs, not necessarily, but a company generally can't afford to have its executives be out of step.

jbob2000 11y ago

I think this article misses what is meant by "Move Fast and Break Things". I never took it to mean 'physically (or programmatically) break things apart while moving quickly', I thought it was more about deconstructing things we considered "normal". Like another way of saying "think outside the box", and do that without hesitation or inhibition.

icebraining 11y ago
“Most companies mess up by moving too slowly and trying to be too precise. When you’re…moving quickly or doing anything like this you want to make mistakes evenly on both sides. We wanted to set up a culture so that we were equally messing up by moving too quickly and by moving too slowly some of the time. So that way, we’d know that we were in the middle.” https://www.youtube.com/watch?v=h7_UNu7zEVs https://www.youtube.com/watch?v=h7_UNu7zEVs
digi_owl 11y ago
My interpretation of it is to say that web devs should not be afraid of introducing bugs while implementing new features, as patches can be pushed just as fast. This may be true for a web site (or app as some like to call them these days). But the mentality has filtered down to actual software running locally, or even whole OSes (see Android for instance, where more than once Google has pushed a feature into the world before all the proverbial edges have been filed down).

swalsh 11y ago

You ever Notice how the engineering departments at these move fast and break things companies grow exponentially? Thinking of etsy, and Facebook, and a few others. Part of it is if they survive, they thrived... But I think the other part is that the code starts to become sharded across people's brains. The technical debt piles up so high in some places that it becomes impossible to maintain productivity with the same number of people.

debacle 11y ago
I think that's a non sequitur. Every startup engineering department grows exponentially. It's the only way for an engineering department to grow.

julie1 11y ago

Well, in physics we compute the speed with half life of what we want to keep alive regarding the costs. Mass transportation is an example of industry where half life is decades/century. Fast for this sector is slower than what the life expectancy of a coder represents. Games made to support a product ad are ephemeral, 1 month. Faster than a new release of firefox. Some dangerous radioactive isotopes are 10000 years. So nuclear plant related automation should not be using any CPU. A citizen data should last for the government as long a citizen live. Government if they aim at remaining stable should consider the half life of their automation to be long, way long. Hence slow changes. Very slow So do banks. But banks have to adapt with the speed of trades. Fast and slow can be set by doing something agile hates: careful business analysis. The problem is how we coupled all the businesses with insane costly rhythm that are forced down on every economical activities by the mean of noncompetitive business practices. Like useless obsolescence driven by HW monopolies, cheap energy, cheap regulated and poor education. Do we really need 1 Pb hard drive when entropy predicts we will not be able to find any relevant information given we have to much data without increasing the means involves in costly ways? We don't need fast changing technologies, we need boring, slow changing technologies that are reliable.

grandalf 11y ago

Move fast and break things for 0.001% of users. In other words, try bold things, bold refactorings, etc. within bounds. If a system is designed with good modularity, this can be done with very low risk. When someone deploys new code at Facebook, I recall reading that it initially goes out to internal users, then to a small percentage of actual users, then eventually to the entire system. This is simply a systems-based approach. The same applies when thinking about code in a testing environment, staging server, etc. At each level there are additional failure modes or adverse interactions. When done stupidly, a cowboy approach can lead to downtime, poorly-reasoned quick fixes, and blame within an institution. Often the same cowboy/girl who acted stupidly and caused the bug is lauded as a hero when he gets things working again a week later after the bug is discovered... when it was his/her bad judgment (or "move fast" mentality) that led the offending code to be shipped in the first place. We must be honest with ourselves and our teams about bad decisions (even in hindsight) and build processes that let us be bold when we need to.

white-flame 11y ago

Racing a car has a great analogy: In order to get faster lap times, you sometimes have to drive slower. Specifically, you need to slow down for corners in order to maintain your control over the vehicle. Try to drive too fast and you end up wrecked. Even if you don't wreck, correcting from having entered a corner too fast loses far more time than just slowing down properly. Similarly to technical debt, if you start a straightaway from a slower starting speed because you botched the prior corner, that damages your lap time throughout the entire stretch. In the overall scheme of being able to deliver software you're confident in, it is faster to consider stability inherently in your development process than try to blast out features and bolt on stability later once you're committed to the initial brittle implementation.

nibs 11y ago
Racing a car applies in two ways I think: the talent is not evenly distributed, and the masters are defined by their ability to decide when to go fast, and when to go slow.
- sharps_xp 11y ago
  I agree. To be more explicit, the talent is indirectly correlated with how slow your team's "slow" is going to be. I think the degree of slow-ness is what management ultimately feels when they conduct retrospectives on technical debt reduction initiatives. We recently did a major refactor of our angular code (~1.5 months) and we've been pretty fast, releasing every 2 weeks. We've begun to outpace a lot of the backend development teams, so we now have more time to squash our backlog tickets. We have the prettiest Jira issue reports out of all teams.

gavinpc 11y ago

> In reality more often then not moving fast and breaking things will result in shipping scrappy software. This is because in the rush to get stuff out the fastest way time consuming things get skipped. Like user testing, automation, analytics, monitoring, manual testing - just to name a few. It sounds like the author could use a dose of his own medicine. The first sentence contains a (common) grammatical error. The third sentence is a fragment. The second is nearly unparseable.

mgrennan 11y ago

I'm not worried about the "Move Fast and Break Things Often" mentality. This will only last until it kills a much of people. Think code in cars, plans, X-ray machines, pill dispensers etc.

agentultra 11y ago

I'd add: think before you code. Programming is hard because thinking is hard. If you don't use tools to help you think rigorously you're going to ship software with bugs regardless of how good your unit tests are or any other strategy you use to cope with poorly designed software. I don't believe not thinking is going to help you move faster. You're just going to spend most of your time fixing the problems you hadn't thought of before you started shipping code. And yet this is what we're encouraged to do. We're asked to fix the bugs, do the least amount of work to get it to go, and fix the problems later. We're asked to not think and just code. There are probably a handful of people in the world who can write complex systems in code with consistently correct results. The rest of us need to be more mindful. We have wonderful tools available to us that can exhaustively check our designs. Using a system like TLA+ allows you to model your system in some pseudo-code and comes with a program that will exhaustively check your code for dead-locks, invariant violations, termination, etc. I've seen it used to find errors in graduate students' binary search algorithms 9 out of 10 times. Imagine what it can do for non-trivial components of your application. Part of moving fast is knowing how to avoid making errors in the first place. The only way I know how to do that is to think. You can't do this properly in code alone. You need higher-level tools to check your assumptions and find holes in your thinking. You don't see structural engineers slapping together the first thing that works and patching the building later when parts of it fall down. You don't sign waivers before crossing a bridge that claim no responsibility on behalf of the designers if it falls down while you're on it. Nor do you see engineers accepting more money from people to build private bridges that are less likely to fall down. So why do software engineers have no such liability for their creations? Why not take a few hours to write a proper specification for your distributed locking mechanism and save yourself days or weeks worth of debugging -- or worse -- liability suits when your mistakes harm the interests of your customers or the public at large? Why not use tools that check your thinking so that you can chase more interesting optimization and performance problems?

matt_wulfeck 11y ago
Maybe it's different for others, but for me the true nature of the problem sometimes only surfaces when you start coding. I can think forever and write the perfect spec, but when the rubber hits the road then things change. I think the reality is a virtuous cycle of thinking, coding, rethinking, recoding that gets the best results.

Animats 11y ago

Von Braun's team launched over 600 V-2 rockets before any of them reached a militarily useful target.

bobbygoodlatte 11y ago

I think this post misses the point. Company values are only meaningful when they differentiate that company from others. Everyone wants to "Move Fast and Not Break Things". "Move Fast and Break Things" was a value judgement. For Facebook, moving quickly was worth the penalty of sometimes breaking things. This is why most companies have lame values like "Do your best work" or "Be honest". Great values, but nobody disagrees with them—so they're meaningless.

wdewind 11y ago
I totally agree with this but would add a bit of nuance. I'd love to see all companies do a "X and the cost of Y" value system, but there is another part of stating you have values like "Integrity" which is the same as the Google "Don't be evil" thing. It's a claim that you can be called publicly on, and that's important as well. You'd be surprised how many companies out there aren't willing to make a claim for something like integrity. But yeah, who wouldn't say they value integrity, for instance, when asked? That being said, I'd also add that most values have those implicitly baked in. For instance, if you do say you value integrity, you're saying you're devaluing growth/money for the value/benefit of integrity.
- bobbygoodlatte 11y ago
  Yep, I agree with your nuance. FWIW I think integrity is a pretty okay company value. Depending on your industry, valuing integrity may be a trade-off that other companies are unwilling to make

frogpelt 11y ago

Facebook/Zuckerberg have obviously been successful because this particular motto has been dissected countless times in blog posts and HN comments. Try this search: https://hn.algolia.com/?q=fast+break https://hn.algolia.com/?q=fast+break

aledesma 11y ago

I have found that using BDD with gherkin syntax can really cut down on many conserns when it comes to software stability and ensuring business worth. Many companies do not put enough focus on automated testing and even when they do they are only testing the surface level with simple funtional tests. If one simply throws the technical requirements into the BDD tests and have your system validate every affected integration prior to shipping then you simply have less debt to pay off as more bugs should be caught prior to shipping. Couple BDD with writing a negative functional test for every bug that is fixed and you should never trigger that specific bug again. In the end these tests will take more time, but if that time is bundled into the time required for the feature, and your code is already layed out in a testable and reusable way, then it is win-win.