5 ms·

Bugs will always happen in production. The important part is not developing everything correctly. "Move fast and break stuff" is a decent philosophy, as long as

Bugs will always happen in production. The important part is not developing everything correctly. "Move fast and break stuff" is a decent philosophy, as long as you can learn from mistakes and fix recurring problems. When a bug in your application presents itself in production, something very specific should happen: 1) identify the symptoms, 2) isolate the causes, 3) address the issue in production, 4) perform a postmortem to identify the next steps, 5) fix the problem's causes throughout the entire supply chain so it cannot happen again. A lot of organizations get stuck only doing steps 1-3. That's not good. Operational processes are just as important, if not more so, than development processes. They're what ensure that when you find inefficiency or bugs, that they not only get fixed, but that they never happen again. This means actually making policy changes, training changes, etc in addition to just fixing code. This is the heart of lean manufacturing that supposedly influenced a lot of modern software development methodology. But some people may have forgotten that the lean manufacturing chain doesn't end at development.

nordsieck 8y ago
> 4) perform a postmortem to identify the next steps It takes an effective management team to make this step work. If the postmortem isn't blameless, this stem becomes completely ineffective as employees (not unreasonably) prioritize their own careers over the health of the company.
- sidlls 8y ago
  Sometimes there is blame, though. Sometimes a root cause is simply that a person or group was not competent in their execution of a task. I think there's a middle ground between never holding anyone accountable and the desperate situation you describe.
  - nordsieck 8y ago
    > Sometimes there is blame, though. Sometimes a root cause is simply that a person or group was not competent in their execution of a task. It is true that some people are not good at their jobs. It's also true that the only way to discover that people are bad at their jobs is their performance. However, I think the process approach is applicable to a much larger domain than many people give it credit. One of the greatest software development quotes I've read comes from DJB [0]: > For many years I have been systematically identifying error-prone programming habits—by reviewing the literature, analyzing other people’s mistakes, and analyzing my own mistakes—and redesigning my programming environment to eliminate those habits. The difference in security bugs found in qmail and sendmail speaks for itself. This is an example of the superiority of process: it shows that what some people consider a "root cause" is actually only a proximate cause - that the root cause is the programming environment that allows the bugs to occur. [0] https://cr.yp.to/qmail/qmailsec-20071101.pdf https://cr.yp.to/qmail/qmailsec-20071101.pdf
    - peterwwillis 8y ago
      Yep. In many ways the "competency" of a person or group comes down to whether there was a designated person whose job it was to ensure they were following good processes. DJB does it himself, but most people don't do it for themselves.
  - andrewaylett 8y ago
    In my experience, it's rare that any blame is sufficiently concentrated as to make assigning it sensible. A failure by a human is more commonly a failure in the process they were following, or in the supervision they should have received. And there's a big difference between "blame-free" and "unaccountable". It's not the fault of a team of developers if they aren't experienced enough to take responsibility for a critical production service, but the solution is not to berate them for being inexperienced: it's to give them the support they should (in hindsight) have had in the first place.
    - late2part 8y ago
      Sometimes that is translated as “a lack of training” which is often a euphemism for someone did something stupid and we have to put in place reactionary consequences (training on what not to do) if they or someone else does it again.
      - user5994461 8y ago
        Lack of experience is a nice word for incompetent. It remind me of an outage a few months ago. Lost a pair of systems, couldn't login and run anything anymore. Turns out, an operator ran a "rm -rf /home/user". He was following a procedure that includes cleaning a subdirectory. The subdirectory didn't exist so he figured out let's clean whatever top path exists!
      - andrewaylett 8y ago
        I typically push back on conclusions that only involve trying to change people and don't also put technical safeguards in place. There are definitely things you can't fully automate, but the goal should be that doing the right thing is really easy and making mistakes is hard.
  - thwarted 8y ago
    There's "blame" and there's "fault". Faults are correctable, blame tries to place responsibility on actors. Few faults can be corrected without acknowledging where responsibility lies. There are multiple kinds of responsibility however. Responsibility for the cause and responsibility for fixing it. The same party who caused the issue isn't necessarily qualified (or authorized) to address it. The idea behind blameless postmortems is to be descriptive without calling out bad action, and evaluating the fault holistically, and acknowledging the reason most faults occur is a failure of processes not necessarily a failure of a person. Fixing processes is much more scalable.
  - jasonlotito 8y ago
    > Sometimes a root cause is simply that a person or group was not competent in their execution of a task. If an incompetent person is able to take down a competent person's system, the "competent" person is to blame.
  - yc-kraln 8y ago
    Sure, sometimes someone made a mistake. But they're not to blame; the process was set up that someone could make a mistake. Change the process so if/when a mistake happens, it doesn't cause a problem.
- tonyedgecombe 8y ago
  In my experience that's quite rare, what is likely to happen is any postmortem will be used as a political stick to beat someone with.
gonzo41 8y ago
I bet this blog post was about javascript :p About your steps. The problem I often see around work is that A bug in prod doesn't get patched in the mainline then pushed all the way up ASAP so it can't accidentally promote up again as it exists somewhere in test or preprod as well. I like the CI/CD approach of just having everything go down the same pipe as it seems to be a decent way to at address point 5.
- pattrn 8y ago
  "I bet this blog post was about javascript :p" You should bet more often ;). That's quite correct. Mostly Javascript with a bit of python.
solatic 8y ago
Most orgs don't do 4-5 because they don't receive enough executive support to make meaningful organizational change. Sometimes the executives are clueless, sometimes they're absent, sometimes they're fighting budget battles against other executives and plotting high-level political plots with other executives and board members. Surprisingly few organizations genuinely care about product or process. The market opportunities for companies which do, are staggering.
- marktangotango 8y ago
  Can you talk more about the opportunities and how dev process is related to product and where the opportunity lies? Sound like a provacative yet vacuous statement to me?
  - solatic 8y ago
    Basic formula: 0) All companies are now software companies. See https://blogs.thomsonreuters.com/answerson/all-companies-are-technology-companies-now/ https://blogs.thomsonreuters.com/answerson/all-companies-are... etc. 1) Pick a well-established, non-high-tech industry, with poor technological adoption, which is not being dominated by a FAANG. Many options exist: banking, industry, food service, construction, transport, hotels, etc. 2) Start a small, local competitor. Differentiate through better technical service, using better dev process to respond quicker to market conditions. 3) Use that technical differentiation to ease scaling and enable growth. Ad infinitum. Many enterprises are still stuck in effectively yearly release cycles, not to mention the number of enterprises still encumbered by legacy systems e.g. mainframes etc. Pick a "target" by making some phone calls, doing some networking, find a business which is encumbered this way, and start a competitor. It should not be difficult to steal business from a competitor which a) will take a year to release a response b) whose response will be handicapped by needing to support internal legacy systems with its response. If you think easy targets like that don't exist, you need to get out of SV and spend some time with the rest of the world.
    - peterwwillis 8y ago
      Sometimes there's side considerations which are technically simple but logistically hard to work around. Technical differentiation is only one way to gain a competitive advantage, but it's also not necessarily the most important advantage. For example, it would be incredibly difficult to compete with Walmart unless you had a network large enough to compete with its effects on the entire production chain. Only someone else like Amazon can compete in that space, and it's not because they're technically superior, it's because they're friggin' huge.
zzzcpan 8y ago
> They're what ensure that when you find inefficiency or bugs, that they not only get fixed, but that they never happen again. This doesn't work. You shouldn't focus too much on making causes of bugs to never happen again, they will happen and they don't matter for operations as new code will bring back old causes of bugs and introduce new ones. What you should focus on is making sure that when bugs do happen they don't cause any trouble. Be realistic about bugs, expect bugs.
- MaulingMonkey 8y ago
  It's not a panacea but there's a lot you can do that does work. Maybe not 100% of the time, maybe not even 90% of the time for some bugs, but stuff that works nonetheless. Better unit test coverage, regression tests, new linters and static analysis passes, fuzzers, explosive runtime checks that catch problems earlier and more obvious ways, ...
  - abraham_lincoln 8y ago
    sentry.io
- peterwwillis 8y ago
  Of course it helps to make your application resilient to failure, but it does not help to ignore the causes of bugs. I would say most bugs are entropy, or a human doing something wrong, or processes that haven't accounted for some edge case. A lot of these can be prevented, and a common way to do that is to watch them happen, document them, and then implement a mechanism to fix the cause when it starts to happen.
sorokod 8y ago
The important part is not developing everything correctly. "Move fast and break stuff" is a decent philosophy, as long as you can learn from mistakes and fix recurring problems This strikes me as extremely arrogant and naive way of thinking. "Customers, stakeholders and investors. Your interests have been damaged but rest assured lessons have been learned. We will continue with this approach as it makes sense to is"
- ozim 8y ago
  not developing everything correctly is about caring about not damaging interest of people. Having data loss while migrating database to new release version is never acceptable. Db migrations should always work correctly, no discussion. Having some css issues where some menu is displayed in a crooked way is perfectly acceptable.
  - sorokod 8y ago
    The attraction of "Move fast and break stuff" is in its simplicity and focus. Once you start qualifying it with "it's only for cosmetic changes not for serious stuff" it stops being worthy of a discussion.
tmaly 8y ago
I would add, write automated tests to identify that the issue was fixed if it is possible.
ryandrake 8y ago
> Bugs will always happen in production. This is not some universal law. It’s true only when the project/team is not willing to make the trade-offs (like development speed) needed to deploy bug-free code. For most software it’s better to release with bugs, but not so for all software. For some software, like safety critical projects, things launched into space, etc, it’s better to take your time and release without bugs. Besides, good development practices, good requirements, good change control, and good testing can prevent the vast majority of bugs from reaching production. Sadly, this idea that bugs in production are inevitable is pervasive in industry.
- ambulancechaser 8y ago
  Can you point to a violation of this observation? Operating systems, CPUs, compilers, and software at large have all had bugs. Can you point to anything meaningful that has not been susceptible to bugs in production? > good development practices, good requirements, good change control, and good testing can prevent the vast majority of bugs prevention of the vast majority of bugs does not negate the claim that bugs will always happen in production.
- rifung 8y ago
  I am skeptical deploying a non trivial project to production without bugs is a viable strategy, even for safety critical projects. I imagine the bugs are just much less severe. It is not even easy to tell whether a piece of software has no bugs or just none you have seen.
- ljm 8y ago
  In order for that to happen, you would need perfect practices, perfect requirements, perfect testing and perfect ‘change control’, and an inhuman ability to never make a human mistake in the understanding of any one of those things within your team of equally perfect people, so add perfect communication on to the list. Anything less than perfection at any stage will be a vector for bugs, whether that bug is a syntax error, unexpected behavior, or a fully tested and functional feature that doesn’t meet expectation. Mistakes are inevitable and not necessarily bad when you handle them well (move fast and break things says nothing about what you do with all the breakage, so feels immature). Calling them bugs doesn’t change that.

nordsieck 8y ago

> 4) perform a postmortem to identify the next steps It takes an effective management team to make this step work. If the postmortem isn't blameless, this stem becomes completely ineffective as employees (not unreasonably) prioritize their own careers over the health of the company.

sidlls 8y ago
Sometimes there is blame, though. Sometimes a root cause is simply that a person or group was not competent in their execution of a task. I think there's a middle ground between never holding anyone accountable and the desperate situation you describe.
- nordsieck 8y ago
  > Sometimes there is blame, though. Sometimes a root cause is simply that a person or group was not competent in their execution of a task. It is true that some people are not good at their jobs. It's also true that the only way to discover that people are bad at their jobs is their performance. However, I think the process approach is applicable to a much larger domain than many people give it credit. One of the greatest software development quotes I've read comes from DJB [0]: > For many years I have been systematically identifying error-prone programming habits—by reviewing the literature, analyzing other people’s mistakes, and analyzing my own mistakes—and redesigning my programming environment to eliminate those habits. The difference in security bugs found in qmail and sendmail speaks for itself. This is an example of the superiority of process: it shows that what some people consider a "root cause" is actually only a proximate cause - that the root cause is the programming environment that allows the bugs to occur. [0] https://cr.yp.to/qmail/qmailsec-20071101.pdf https://cr.yp.to/qmail/qmailsec-20071101.pdf
  - peterwwillis 8y ago
    Yep. In many ways the "competency" of a person or group comes down to whether there was a designated person whose job it was to ensure they were following good processes. DJB does it himself, but most people don't do it for themselves.
- andrewaylett 8y ago
  In my experience, it's rare that any blame is sufficiently concentrated as to make assigning it sensible. A failure by a human is more commonly a failure in the process they were following, or in the supervision they should have received. And there's a big difference between "blame-free" and "unaccountable". It's not the fault of a team of developers if they aren't experienced enough to take responsibility for a critical production service, but the solution is not to berate them for being inexperienced: it's to give them the support they should (in hindsight) have had in the first place.
  - late2part 8y ago
    Sometimes that is translated as “a lack of training” which is often a euphemism for someone did something stupid and we have to put in place reactionary consequences (training on what not to do) if they or someone else does it again.
    - user5994461 8y ago
      Lack of experience is a nice word for incompetent. It remind me of an outage a few months ago. Lost a pair of systems, couldn't login and run anything anymore. Turns out, an operator ran a "rm -rf /home/user". He was following a procedure that includes cleaning a subdirectory. The subdirectory didn't exist so he figured out let's clean whatever top path exists!
    - andrewaylett 8y ago
      I typically push back on conclusions that only involve trying to change people and don't also put technical safeguards in place. There are definitely things you can't fully automate, but the goal should be that doing the right thing is really easy and making mistakes is hard.
- thwarted 8y ago
  There's "blame" and there's "fault". Faults are correctable, blame tries to place responsibility on actors. Few faults can be corrected without acknowledging where responsibility lies. There are multiple kinds of responsibility however. Responsibility for the cause and responsibility for fixing it. The same party who caused the issue isn't necessarily qualified (or authorized) to address it. The idea behind blameless postmortems is to be descriptive without calling out bad action, and evaluating the fault holistically, and acknowledging the reason most faults occur is a failure of processes not necessarily a failure of a person. Fixing processes is much more scalable.
- jasonlotito 8y ago
  > Sometimes a root cause is simply that a person or group was not competent in their execution of a task. If an incompetent person is able to take down a competent person's system, the "competent" person is to blame.
- yc-kraln 8y ago
  Sure, sometimes someone made a mistake. But they're not to blame; the process was set up that someone could make a mistake. Change the process so if/when a mistake happens, it doesn't cause a problem.
tonyedgecombe 8y ago
In my experience that's quite rare, what is likely to happen is any postmortem will be used as a political stick to beat someone with.

gonzo41 8y ago

I bet this blog post was about javascript :p About your steps. The problem I often see around work is that A bug in prod doesn't get patched in the mainline then pushed all the way up ASAP so it can't accidentally promote up again as it exists somewhere in test or preprod as well. I like the CI/CD approach of just having everything go down the same pipe as it seems to be a decent way to at address point 5.

pattrn 8y ago
"I bet this blog post was about javascript :p" You should bet more often ;). That's quite correct. Mostly Javascript with a bit of python.

solatic 8y ago

Most orgs don't do 4-5 because they don't receive enough executive support to make meaningful organizational change. Sometimes the executives are clueless, sometimes they're absent, sometimes they're fighting budget battles against other executives and plotting high-level political plots with other executives and board members. Surprisingly few organizations genuinely care about product or process. The market opportunities for companies which do, are staggering.

marktangotango 8y ago
Can you talk more about the opportunities and how dev process is related to product and where the opportunity lies? Sound like a provacative yet vacuous statement to me?
- solatic 8y ago
  Basic formula: 0) All companies are now software companies. See https://blogs.thomsonreuters.com/answerson/all-companies-are-technology-companies-now/ https://blogs.thomsonreuters.com/answerson/all-companies-are... etc. 1) Pick a well-established, non-high-tech industry, with poor technological adoption, which is not being dominated by a FAANG. Many options exist: banking, industry, food service, construction, transport, hotels, etc. 2) Start a small, local competitor. Differentiate through better technical service, using better dev process to respond quicker to market conditions. 3) Use that technical differentiation to ease scaling and enable growth. Ad infinitum. Many enterprises are still stuck in effectively yearly release cycles, not to mention the number of enterprises still encumbered by legacy systems e.g. mainframes etc. Pick a "target" by making some phone calls, doing some networking, find a business which is encumbered this way, and start a competitor. It should not be difficult to steal business from a competitor which a) will take a year to release a response b) whose response will be handicapped by needing to support internal legacy systems with its response. If you think easy targets like that don't exist, you need to get out of SV and spend some time with the rest of the world.
  - peterwwillis 8y ago
    Sometimes there's side considerations which are technically simple but logistically hard to work around. Technical differentiation is only one way to gain a competitive advantage, but it's also not necessarily the most important advantage. For example, it would be incredibly difficult to compete with Walmart unless you had a network large enough to compete with its effects on the entire production chain. Only someone else like Amazon can compete in that space, and it's not because they're technically superior, it's because they're friggin' huge.

zzzcpan 8y ago

> They're what ensure that when you find inefficiency or bugs, that they not only get fixed, but that they never happen again. This doesn't work. You shouldn't focus too much on making causes of bugs to never happen again, they will happen and they don't matter for operations as new code will bring back old causes of bugs and introduce new ones. What you should focus on is making sure that when bugs do happen they don't cause any trouble. Be realistic about bugs, expect bugs.

MaulingMonkey 8y ago
It's not a panacea but there's a lot you can do that does work. Maybe not 100% of the time, maybe not even 90% of the time for some bugs, but stuff that works nonetheless. Better unit test coverage, regression tests, new linters and static analysis passes, fuzzers, explosive runtime checks that catch problems earlier and more obvious ways, ...
- abraham_lincoln 8y ago
  sentry.io
peterwwillis 8y ago
Of course it helps to make your application resilient to failure, but it does not help to ignore the causes of bugs. I would say most bugs are entropy, or a human doing something wrong, or processes that haven't accounted for some edge case. A lot of these can be prevented, and a common way to do that is to watch them happen, document them, and then implement a mechanism to fix the cause when it starts to happen.

sorokod 8y ago

The important part is not developing everything correctly. "Move fast and break stuff" is a decent philosophy, as long as you can learn from mistakes and fix recurring problems This strikes me as extremely arrogant and naive way of thinking. "Customers, stakeholders and investors. Your interests have been damaged but rest assured lessons have been learned. We will continue with this approach as it makes sense to is"

ozim 8y ago
not developing everything correctly is about caring about not damaging interest of people. Having data loss while migrating database to new release version is never acceptable. Db migrations should always work correctly, no discussion. Having some css issues where some menu is displayed in a crooked way is perfectly acceptable.
- sorokod 8y ago
  The attraction of "Move fast and break stuff" is in its simplicity and focus. Once you start qualifying it with "it's only for cosmetic changes not for serious stuff" it stops being worthy of a discussion.

tmaly 8y ago

I would add, write automated tests to identify that the issue was fixed if it is possible.

ryandrake 8y ago

> Bugs will always happen in production. This is not some universal law. It’s true only when the project/team is not willing to make the trade-offs (like development speed) needed to deploy bug-free code. For most software it’s better to release with bugs, but not so for all software. For some software, like safety critical projects, things launched into space, etc, it’s better to take your time and release without bugs. Besides, good development practices, good requirements, good change control, and good testing can prevent the vast majority of bugs from reaching production. Sadly, this idea that bugs in production are inevitable is pervasive in industry.

ambulancechaser 8y ago
Can you point to a violation of this observation? Operating systems, CPUs, compilers, and software at large have all had bugs. Can you point to anything meaningful that has not been susceptible to bugs in production? > good development practices, good requirements, good change control, and good testing can prevent the vast majority of bugs prevention of the vast majority of bugs does not negate the claim that bugs will always happen in production.
rifung 8y ago
I am skeptical deploying a non trivial project to production without bugs is a viable strategy, even for safety critical projects. I imagine the bugs are just much less severe. It is not even easy to tell whether a piece of software has no bugs or just none you have seen.
ljm 8y ago
In order for that to happen, you would need perfect practices, perfect requirements, perfect testing and perfect ‘change control’, and an inhuman ability to never make a human mistake in the understanding of any one of those things within your team of equally perfect people, so add perfect communication on to the list. Anything less than perfection at any stage will be a vector for bugs, whether that bug is a syntax error, unexpected behavior, or a fully tested and functional feature that doesn’t meet expectation. Mistakes are inevitable and not necessarily bad when you handle them well (move fast and break things says nothing about what you do with all the breakage, so feels immature). Calling them bugs doesn’t change that.