9 ms·

Continuous Delivery Sounds Great, but Will It Work Here?

83 points by msolujic 9y ago

Shoothe 9y ago
Jez Humble's Lean Enterprise book was one of the best quality content I read. Highly recommend!
mabbo 9y ago
I once helped lead my team from a tedious manual process into full-CD. Took a lot of pain to get there, and we had to build some new tools to automate all of the steps we were doing, but we did it. Most of the pain was in realizing we didn't have enough UI regression tests. Every 'incident' sparked a new set of tests and over time the incidents stopped happening. Our velocity increased, time-to-market dropped, operational workload fell. It was great. Sadly, after a few years of the joy of full-CD, more conservative leadership took over our org. Now, all deployments of software must be accompanied by paperwork (digital), step-by-step plans of how you're going to deploy this software, approvals from senior managers, just general micromanagement. CD isn't possible in such environments.
- dominotw 9y ago
  Can the paperwork be automated. Automatically deploy software once ticket is approved.
  - Chyzwar 9y ago
    nope, otherwise said senior management will be no longer needed. These people need to create a veil of importance. They will create multi-level approval process. they will add QA team in India and customer signoff. When something happens blame is still on developers.
    - sidlls 9y ago
      If customers are involved maybe their signoff is necessary (not technically, but contractually). QA seems like an important, necessary step in any delivery process, "continuous" or not: tests aren't infallible and a combination of complementary automated and manual tests aren't a bad thing necessarily. I find often when developers think something that is process driven like this isn't necessary it's because they're not thinking outside their own direct and immediate concerns.
      - Chyzwar 9y ago
        QA is only needed for acceptance testing and should be automated. Not a tens of people doing checklist testing. The customer should sign off on during requirements gathering phase and on frequent demos, not before release. I often think that management was transplanted into the foreign ground of software development and have no clue. They will then come with platinum quarterly releases were testing cost more than development.
        valuearb 9y ago
        Automated testing is great, but there is something to be said for the value of having real eyes on the product. Often times front ends can pass all the tests, but a good QA person suddenly realizes, wait a second, do we really want it to work this way?
      - crdoconnor 9y ago
        I've been experimenting with auto-generating documentation (with screenshots) from tests as a means of getting quicker and more trusting customer sign off. The way I see it this is a communication feedback loop much like the one between tester and developer or product manager and developer and leaving that gap open means you're one step further away continuous delivery. I don't think it's unreasonable for customers to be somewhat distrustful of us, their black box that spits out code.
- Shoothe 9y ago
  You could build the sign off phases as part of the deployment pipeline so that each change automatically requests sign-offs from seniors and resumes when they are given. Of course if your management is actively against CD that could be hard but generally approvals are not a big problem.
- crdoconnor 9y ago
  >Most of the pain was in realizing we didn't have enough UI regression tests. Every 'incident' sparked a new set of tests and over time the incidents stopped happening. I noticed a process similar to this where bugs from production would appear to 'migrate' from tested scenarios to untested scenarios. We'd start off with UI tests on a browser, for example, and then the bugs would be clamped down on there but would seem now to crop up in API interactions or untested browsers. Then we started accommodating the tests to handle different browsers and mocking API interactions and the bugs would start cropping up somewhere else. What was counterintuitive was that, even though code coverage metrics were really quite low, most bugs cropped up in areas of the code with automated test coverage. "Environmental" coverage seemed to matter much, much more.
  - tacon 9y ago
    I really like to learn more about this scenario. Is it covered anywhere in more detail? Maybe some examples?
    - 0xcde4c3db 9y ago
      That description reminds me of the Sorcerer's Apprentice Syndrome in TFTP. https://en.wikipedia.org/wiki/Sorcerer%27s_Apprentice_Syndrome https://en.wikipedia.org/wiki/Sorcerer%27s_Apprentice_Syndro...
    - crdoconnor 9y ago
      Not that I know of, sorry.
- koolba 9y ago
  Fully automatic deploys are distinct from fully automated ones. I insist on the latter but not everyone is comfortable with the former. Large / legacy / lumbering institutions (think big banks) in particular want a change ticket, window or some other control before a release makes it's way to production. The middle ground I've found is to have fully automatic deploys in lower environments and automate the entire process for production. But rather than have releases flow there automatically, production is handled by pressing a single "Deploy whatever is in UAT to prod" button (i.e. the "Big Red Easy Button"). You then give access to that button to somebody else in the org (ex: a UAT team, a release team, your manager) and everybody's happy. It also conveniently gives you an excuse to never fill out production release paperwork as "It's not my job, I'm not allowed to release to production!"
  - wpietri 9y ago
    You raise an important point. I think there's a big distinction to be drawn between people's feelings and actual business needs. People everywhere like feelings of control. And the more layers of management a company has, the more executive feelings come to predominate over on-the-ground business facts. But in practice, bureaucratic change-control processes don't actually produce better code or safer systems. In my experience, they make things worse, because large batches and slow feedback loops make bugs easy to add and hard to find. One good next step after the middle ground you describe is continuous deployment of code, but using feature flags and A/B testing tools to control visibility and impact of features. As you say, if you give people some big, obvious controls, they feel in control. Decoupling code release with business impact lets me get on with what I care about. And it lets them learn to be bolder, more engaged in the process of delivering value to users.
  - hinkley 9y ago
    There’s a new group learning CD that mistakenly believes that automatic deploys are CD. Most of these came to the concept after it had supplanted discussion about CI. Automating a deploy is relatively straightforward but doesn’t provide much value if that’s all you do. Local reproduction of build steps, metrics, historical data, building out test infrastructure, all of this is important for scaling up a team and keeping regressions low.
- goliatone 9y ago
  What was the “reason” given to change the process? Were these changes and steps addressing something (regressions, stability, quality...)? Not saying your account is untrue, or that I can’t imagine management doing asinine things but it is still hard for me to imagine a) management willing to change something that’s not broken and b) developers that wouldn’t push back to said change.
  - mabbo 9y ago
    Not all teams had moved to full-CD. Not all teams had a good set of tests, automated safe deployment windows, alarms and auto-rollback, all the things you need to deploy any software safely. This lead to high severity incidents, customer dissatisfaction. The solution, of course, is more bureaucracy! And it worked, sort of. Everyone spends all the time writing documents asking for permission to deploy software changes rather than coding, so there's less changes overall, leading to less incidents. Never mind that velocity is a fraction of what it was or might have been. In short, management wants to solve the problem in a way they understand rather than a way that works. And they love a way that has them micromanage everything and nit-pick when you didn't fill out the paperwork correctly. Now, I should caveat all this- I'm a terrible pessimist sometimes and also I haven't had any caffeine today which can only make things worse.
- devonkim 9y ago
  I’m pretty done with old school change managers that act like ITIL is the only sane way to practice safe, reliable management of systems. In fact, I’ve never, ever seen anyone go full hog ITIL in IT leadership in the F500 that had ever worked at a high performance engineering or IT organization - it appears to be correlated like Six Sigma with low performing organizations after they adopt it (performance in the HBR study I saw for Six Sigma correlated negatively with the S&P 500). What I used to do for demos was to automate the manual processes of ITIL style change review including diffs of the code, test plans executed, test coverage, etc. and the dependency graph of CI impacted by a change (because CMDBs are broken in 99.9% of orgs, you need tools like CloudFormation, Terraform, etc. to give the granularity necessary for a thorough review even in bureaucratic hell). All the other parts of dealing with bureaucracy is pretty much filling out other fields like a list of approvers, signed commits, org, billing codes, etc. are all a matter of trivia rather than smarts. Your changesets need to become higher grained enough to warrant all the human eyes involved. But going backwards like that in processes is usually a sign that a place that is in decline though so that alone should be setting off warnings.
jakevoytko 9y ago
I work at a company mentioned in the article (Etsy). I've also worked in environments with daily cuts; these feel similar in practice. Having the ability to do continuous deployment is not the same as practicing continuous deployment. You don't buy much if developers still yolo-push 6000 line commits with fully baked features. You still need to foster a strong developer culture that encourages small changes. This is true outside of continuous deployment. But since you gain the option to deploy small and isolated changes to production, the consequences are much different. This has many implications. It needs to be OK to push back and ask engineers to split large code commits into smaller units. Engineers need a strong understanding of how to sequence changes: "I need to do the ORM changes, then the API endpoint, then the controller change, and then I can do the frontend." This means that senior engineers invest time in helping junior engineers understand why this has to be done and why this particular sequence is safe. Methods need to be devised to avoid holding up future pushes - for instance, a culture of rolling back to safety at the first sign of trouble instead of debugging in production. Anecdotally, the best developers seem to always be pushing small changes, or always have a few commits in the daily cut. Interpreting this is a bit of a chicken and egg problem - is throughput higher because the changes are smaller, or does high implementation volume just imply more changes? Or maybe it's just a free form of advertisement: "I always see that X is pushing, so they must be good"
- klenwell 9y ago
  > Anecdotally, the best developers seem to always be pushing small changes More formally, this is true, too. At least according to Donald G. Reinertsen, the author of this book: https://www.goodreads.com/book/show/6278270-the-principles-of-product-development-flow https://www.goodreads.com/book/show/6278270-the-principles-o... Batches and queues are primary themes of the book and Reinertsen waxes prosaic on the extensive benefits of making batches smaller all around during product development. I've noticed the benefits mainly with user story size and pull request size (which are generally correlated). Large batch sizes lead to longer delays which lead to cut corners which lead to more defects which lead to further delays. There are other underestimated penalties that Reinertsen discusses as well. Of course, too small and you're incurring disproportionate overhead with each batch. The key, I suppose, is figuring out mama-bear batch sizes for your team or organization and effectively paring down to them.
0xcde4c3db 9y ago
I don't feel like the article really addresses "highly regulated environments". To me, that means things like avionics, military equipment, medical devices, and nuclear power plants (cf. the list of things buried in most sufficiently-BigCo EULAs that you're not allowed to use the software for). Are there good examples of CD-like practices being used in these domains? I'm also a bit disappointed at how generic a lot of the wording is. It seems like you could do a search-and-replace of "continuous delivery" with almost any other process improvement term (e.g. agile, kaizen, Six Sigma) and have people nodding right along for 90% of it. Is there a concise explanation somewhere that really makes clear how this is something more fundamental than devops-flavored kaizen? Or am I expected to read the entire "canon" before I understand it enough to evaluate it?
- amarkov 9y ago
  Ensuring deployments are "predictable, routine affairs that can be performed on demand at any time" isn't just a generic business process problem. It imposes significant constraints on how code has to be written and tested.
denzil_correa 9y ago
One point missed in this entire calculation of "Continuous Delivery" of systems is the estimated "Cost of a Mistake" [0]. There are many applications where an small error in deployment could lead to a loss of a human life. Code for safety critical systems requires formal checks before being deployed and iterated [1]. There are indeed many many environments where continuous delivery won't work for very good reasons. [0] https://twitter.com/benhamner/status/832785333070606337 https://twitter.com/benhamner/status/832785333070606337 [1] https://en.wikipedia.org/wiki/Static_program_analysis https://en.wikipedia.org/wiki/Static_program_analysis
- icebraining 9y ago
  Continuous Delivery != Continuous Deployment. The first won't lead to any cost. Also, why can't those formal checks be applied in a CD fashion?
  - denzil_correa 9y ago
    The article does not distinguish between the terms delivery and deployment. Some relevant excerpts from the article below > It is often assumed that deploying software more frequently means accepting lower levels of stability and reliability in systems ... > While continuous deployment, the practice of continuously releasing every good build of your software, is mainly limited to cloud- or datacenter-hosted services, continuous delivery—the set of practices described here that enables continuous deployment—can be applied in any domain. ... > In continuous delivery, automated deployments to production-like environments are performed frequently throughout the deployment pipeline,
    - icebraining 9y ago
      The second quote is specifically distinguishing them :) it says that Continuous Delivery enables Continuous Deployment, while making it clear that they don't necessarily apply to the same cases. As for the third quote, notice it says production-like, not production itself. The idea is that you test in an environment that is as close to production as possible. Nobody will die if you deploy a bad build to a real pacemaker - as long as it's not inside an actual person :)
js8 9y ago
How does CD work with multiple vendors? For example, I work in a company on a product that reads data from many products of other companies. How do we tell our customers that we certified the product with the certain version of the products that we read data from? If vendors put change into the production continuously, how can we react to those changes before the customer gets them?

83 points by msolujic 9y ago

Jez Humble's Lean Enterprise book was one of the best quality content I read. Highly recommend!

I once helped lead my team from a tedious manual process into full-CD. Took a lot of pain to get there, and we had to build some new tools to automate all of the steps we were doing, but we did it. Most of the pain was in realizing we didn't have enough UI regression tests. Every 'incident' sparked a new set of tests and over time the incidents stopped happening. Our velocity increased, time-to-market dropped, operational workload fell. It was great. Sadly, after a few years of the joy of full-CD, more conservative leadership took over our org. Now, all deployments of software must be accompanied by paperwork (digital), step-by-step plans of how you're going to deploy this software, approvals from senior managers, just general micromanagement. CD isn't possible in such environments.

dominotw 9y ago
Can the paperwork be automated. Automatically deploy software once ticket is approved.
- Chyzwar 9y ago
  nope, otherwise said senior management will be no longer needed. These people need to create a veil of importance. They will create multi-level approval process. they will add QA team in India and customer signoff. When something happens blame is still on developers.
  - sidlls 9y ago
    If customers are involved maybe their signoff is necessary (not technically, but contractually). QA seems like an important, necessary step in any delivery process, "continuous" or not: tests aren't infallible and a combination of complementary automated and manual tests aren't a bad thing necessarily. I find often when developers think something that is process driven like this isn't necessary it's because they're not thinking outside their own direct and immediate concerns.
    - Chyzwar 9y ago
      QA is only needed for acceptance testing and should be automated. Not a tens of people doing checklist testing. The customer should sign off on during requirements gathering phase and on frequent demos, not before release. I often think that management was transplanted into the foreign ground of software development and have no clue. They will then come with platinum quarterly releases were testing cost more than development.
      - valuearb 9y ago
        Automated testing is great, but there is something to be said for the value of having real eyes on the product. Often times front ends can pass all the tests, but a good QA person suddenly realizes, wait a second, do we really want it to work this way?
    - crdoconnor 9y ago
      I've been experimenting with auto-generating documentation (with screenshots) from tests as a means of getting quicker and more trusting customer sign off. The way I see it this is a communication feedback loop much like the one between tester and developer or product manager and developer and leaving that gap open means you're one step further away continuous delivery. I don't think it's unreasonable for customers to be somewhat distrustful of us, their black box that spits out code.
Shoothe 9y ago
You could build the sign off phases as part of the deployment pipeline so that each change automatically requests sign-offs from seniors and resumes when they are given. Of course if your management is actively against CD that could be hard but generally approvals are not a big problem.
crdoconnor 9y ago
>Most of the pain was in realizing we didn't have enough UI regression tests. Every 'incident' sparked a new set of tests and over time the incidents stopped happening. I noticed a process similar to this where bugs from production would appear to 'migrate' from tested scenarios to untested scenarios. We'd start off with UI tests on a browser, for example, and then the bugs would be clamped down on there but would seem now to crop up in API interactions or untested browsers. Then we started accommodating the tests to handle different browsers and mocking API interactions and the bugs would start cropping up somewhere else. What was counterintuitive was that, even though code coverage metrics were really quite low, most bugs cropped up in areas of the code with automated test coverage. "Environmental" coverage seemed to matter much, much more.
- tacon 9y ago
  I really like to learn more about this scenario. Is it covered anywhere in more detail? Maybe some examples?
  - 0xcde4c3db 9y ago
    That description reminds me of the Sorcerer's Apprentice Syndrome in TFTP. https://en.wikipedia.org/wiki/Sorcerer%27s_Apprentice_Syndrome https://en.wikipedia.org/wiki/Sorcerer%27s_Apprentice_Syndro...
  - crdoconnor 9y ago
    Not that I know of, sorry.
koolba 9y ago
Fully automatic deploys are distinct from fully automated ones. I insist on the latter but not everyone is comfortable with the former. Large / legacy / lumbering institutions (think big banks) in particular want a change ticket, window or some other control before a release makes it's way to production. The middle ground I've found is to have fully automatic deploys in lower environments and automate the entire process for production. But rather than have releases flow there automatically, production is handled by pressing a single "Deploy whatever is in UAT to prod" button (i.e. the "Big Red Easy Button"). You then give access to that button to somebody else in the org (ex: a UAT team, a release team, your manager) and everybody's happy. It also conveniently gives you an excuse to never fill out production release paperwork as "It's not my job, I'm not allowed to release to production!"
- wpietri 9y ago
  You raise an important point. I think there's a big distinction to be drawn between people's feelings and actual business needs. People everywhere like feelings of control. And the more layers of management a company has, the more executive feelings come to predominate over on-the-ground business facts. But in practice, bureaucratic change-control processes don't actually produce better code or safer systems. In my experience, they make things worse, because large batches and slow feedback loops make bugs easy to add and hard to find. One good next step after the middle ground you describe is continuous deployment of code, but using feature flags and A/B testing tools to control visibility and impact of features. As you say, if you give people some big, obvious controls, they feel in control. Decoupling code release with business impact lets me get on with what I care about. And it lets them learn to be bolder, more engaged in the process of delivering value to users.
- hinkley 9y ago
  There’s a new group learning CD that mistakenly believes that automatic deploys are CD. Most of these came to the concept after it had supplanted discussion about CI. Automating a deploy is relatively straightforward but doesn’t provide much value if that’s all you do. Local reproduction of build steps, metrics, historical data, building out test infrastructure, all of this is important for scaling up a team and keeping regressions low.
goliatone 9y ago
What was the “reason” given to change the process? Were these changes and steps addressing something (regressions, stability, quality...)? Not saying your account is untrue, or that I can’t imagine management doing asinine things but it is still hard for me to imagine a) management willing to change something that’s not broken and b) developers that wouldn’t push back to said change.
- mabbo 9y ago
  Not all teams had moved to full-CD. Not all teams had a good set of tests, automated safe deployment windows, alarms and auto-rollback, all the things you need to deploy any software safely. This lead to high severity incidents, customer dissatisfaction. The solution, of course, is more bureaucracy! And it worked, sort of. Everyone spends all the time writing documents asking for permission to deploy software changes rather than coding, so there's less changes overall, leading to less incidents. Never mind that velocity is a fraction of what it was or might have been. In short, management wants to solve the problem in a way they understand rather than a way that works. And they love a way that has them micromanage everything and nit-pick when you didn't fill out the paperwork correctly. Now, I should caveat all this- I'm a terrible pessimist sometimes and also I haven't had any caffeine today which can only make things worse.
devonkim 9y ago
I’m pretty done with old school change managers that act like ITIL is the only sane way to practice safe, reliable management of systems. In fact, I’ve never, ever seen anyone go full hog ITIL in IT leadership in the F500 that had ever worked at a high performance engineering or IT organization - it appears to be correlated like Six Sigma with low performing organizations after they adopt it (performance in the HBR study I saw for Six Sigma correlated negatively with the S&P 500). What I used to do for demos was to automate the manual processes of ITIL style change review including diffs of the code, test plans executed, test coverage, etc. and the dependency graph of CI impacted by a change (because CMDBs are broken in 99.9% of orgs, you need tools like CloudFormation, Terraform, etc. to give the granularity necessary for a thorough review even in bureaucratic hell). All the other parts of dealing with bureaucracy is pretty much filling out other fields like a list of approvers, signed commits, org, billing codes, etc. are all a matter of trivia rather than smarts. Your changesets need to become higher grained enough to warrant all the human eyes involved. But going backwards like that in processes is usually a sign that a place that is in decline though so that alone should be setting off warnings.

jakevoytko 9y ago

I work at a company mentioned in the article (Etsy). I've also worked in environments with daily cuts; these feel similar in practice. Having the ability to do continuous deployment is not the same as practicing continuous deployment. You don't buy much if developers still yolo-push 6000 line commits with fully baked features. You still need to foster a strong developer culture that encourages small changes. This is true outside of continuous deployment. But since you gain the option to deploy small and isolated changes to production, the consequences are much different. This has many implications. It needs to be OK to push back and ask engineers to split large code commits into smaller units. Engineers need a strong understanding of how to sequence changes: "I need to do the ORM changes, then the API endpoint, then the controller change, and then I can do the frontend." This means that senior engineers invest time in helping junior engineers understand why this has to be done and why this particular sequence is safe. Methods need to be devised to avoid holding up future pushes - for instance, a culture of rolling back to safety at the first sign of trouble instead of debugging in production. Anecdotally, the best developers seem to always be pushing small changes, or always have a few commits in the daily cut. Interpreting this is a bit of a chicken and egg problem - is throughput higher because the changes are smaller, or does high implementation volume just imply more changes? Or maybe it's just a free form of advertisement: "I always see that X is pushing, so they must be good"

klenwell 9y ago
> Anecdotally, the best developers seem to always be pushing small changes More formally, this is true, too. At least according to Donald G. Reinertsen, the author of this book: https://www.goodreads.com/book/show/6278270-the-principles-of-product-development-flow https://www.goodreads.com/book/show/6278270-the-principles-o... Batches and queues are primary themes of the book and Reinertsen waxes prosaic on the extensive benefits of making batches smaller all around during product development. I've noticed the benefits mainly with user story size and pull request size (which are generally correlated). Large batch sizes lead to longer delays which lead to cut corners which lead to more defects which lead to further delays. There are other underestimated penalties that Reinertsen discusses as well. Of course, too small and you're incurring disproportionate overhead with each batch. The key, I suppose, is figuring out mama-bear batch sizes for your team or organization and effectively paring down to them.

0xcde4c3db 9y ago

I don't feel like the article really addresses "highly regulated environments". To me, that means things like avionics, military equipment, medical devices, and nuclear power plants (cf. the list of things buried in most sufficiently-BigCo EULAs that you're not allowed to use the software for). Are there good examples of CD-like practices being used in these domains? I'm also a bit disappointed at how generic a lot of the wording is. It seems like you could do a search-and-replace of "continuous delivery" with almost any other process improvement term (e.g. agile, kaizen, Six Sigma) and have people nodding right along for 90% of it. Is there a concise explanation somewhere that really makes clear how this is something more fundamental than devops-flavored kaizen? Or am I expected to read the entire "canon" before I understand it enough to evaluate it?

amarkov 9y ago
Ensuring deployments are "predictable, routine affairs that can be performed on demand at any time" isn't just a generic business process problem. It imposes significant constraints on how code has to be written and tested.

denzil_correa 9y ago

One point missed in this entire calculation of "Continuous Delivery" of systems is the estimated "Cost of a Mistake" [0]. There are many applications where an small error in deployment could lead to a loss of a human life. Code for safety critical systems requires formal checks before being deployed and iterated [1]. There are indeed many many environments where continuous delivery won't work for very good reasons. [0] https://twitter.com/benhamner/status/832785333070606337 https://twitter.com/benhamner/status/832785333070606337 [1] https://en.wikipedia.org/wiki/Static_program_analysis https://en.wikipedia.org/wiki/Static_program_analysis

icebraining 9y ago
Continuous Delivery != Continuous Deployment. The first won't lead to any cost. Also, why can't those formal checks be applied in a CD fashion?
- denzil_correa 9y ago
  The article does not distinguish between the terms delivery and deployment. Some relevant excerpts from the article below > It is often assumed that deploying software more frequently means accepting lower levels of stability and reliability in systems ... > While continuous deployment, the practice of continuously releasing every good build of your software, is mainly limited to cloud- or datacenter-hosted services, continuous delivery—the set of practices described here that enables continuous deployment—can be applied in any domain. ... > In continuous delivery, automated deployments to production-like environments are performed frequently throughout the deployment pipeline,
  - icebraining 9y ago
    The second quote is specifically distinguishing them :) it says that Continuous Delivery enables Continuous Deployment, while making it clear that they don't necessarily apply to the same cases. As for the third quote, notice it says production-like, not production itself. The idea is that you test in an environment that is as close to production as possible. Nobody will die if you deploy a bad build to a real pacemaker - as long as it's not inside an actual person :)

js8 9y ago

How does CD work with multiple vendors? For example, I work in a company on a product that reads data from many products of other companies. How do we tell our customers that we certified the product with the certain version of the products that we read data from? If vendors put change into the production continuously, how can we react to those changes before the customer gets them?