6 ms·

Benefits of Continuous Delivery

82 points by henrik_w 9y ago

vsupalov 9y ago
Great article! A tiny nitpick: the distinction between continuous delivery and continuous deployment, is that in the first case you could deploy anytime your want, but the triggering is still up to a human. With continuous deployment, everything is shipped to prod automatically, given that all conditions are met. If you want to learn more quickly - I did a talk on the topic last week, and did my best to provide a concise overview of the most essential terms. Check out the slides for a high-level view on ci/cd [1] and deployment pipelines in general [2] if you want to learn more. [1] https://www.slideshare.net/VladislavSupalov/automated-testing-environments-with-kubernetes-gitlab/12 https://www.slideshare.net/VladislavSupalov/automated-testin... [2] https://www.slideshare.net/VladislavSupalov/automated-testing-environments-with-kubernetes-gitlab/17 https://www.slideshare.net/VladislavSupalov/automated-testin...
- korzun 9y ago
  I think you are nitpicking just to nitpick. Technically, continuous deployment is still 100% up to a human.
- rhizome 9y ago
  Your "continuous delivery" definition sounds like CI to me.
atsaloli 9y ago
If anyone wants to learn how to set up CI/CD, I have a free self-paced class at https://gitpitch.com/atsaloli/cicd/master?grs=gitlab#/ https://gitpitch.com/atsaloli/cicd/master?grs=gitlab#/ -- all you need to follow along is an Ubuntu VM for the hands-on exercises.
deleted 9y ago
[deleted]
zeroz 9y ago
Good summary. One missing argument IMHO against continuous deployment: Fear of consequences in strong regulated sectors, like finance and insurance. Continuous delivery is highly encouraged, but for deployments I see strong preferences to test everything (in some parts automation is still weak) and therefor some bundling of deployments or special dates are still preferred. I think costs of bad reputation or being watch by regulators because of failed or illegal 'transactions' are in these businesses much higher than e.g. in retail, gaming, etc.
- brightball 9y ago
  Continuous deployment is something that you can only truly achieve when you have a very high degree of confidence in your automated testing IMHO.
- beat 9y ago
  And meanwhile, we have Equifax losing tons of valuable data due to a breach caused largely by how slowly they deploy, and how difficult it is for them to get rid of antiquated technology. After many years in big enterprise, I've learned something important - the appearance of risk is more important than the existence of risk. Continuous delivery looks "risky". Slow, deliberate release cycles on a quarterly or even yearly basis look "safe", because "testing". In practice, those quarterly deployments have far too many changes embedded in them all at once. Worse, teams race to get their features in under the deadline, knowing it can be months before they'll get another chance, leading to careless coding and inadequate testing. So, based on both my experience and a little beyond-common-sense logic, slow release cycles are more risky than fast ones.
  - zeroz 9y ago
    I absolutely agree with you! These huge quarterly updates with last minute hop on's and changes ("otherwise we have to wait another three months") which large enterprises did or are still doing are more risky than smaller ones. Does this require to jump directly to continuous deployment? I don't think so. What's wrong with continuous delivery to user acceptance stage and e.g. two weekly deployments after ~98% automated tests ~2% manuel test (especially penetration).
ryanbrunner 9y ago
I like that this article doesn't focus too much on the technical aspects of auto-deploys and CI. I've been in a lot of places where the concepts were lumped together, and while we could have easily delivered software continuously, we falsely believed we needed every last thing to be perfectly automated before we did. It's important that your deployment process is repeatable and simple (i.e. typing one or two commands into a terminal), but if a human is still kicking it off that's still a net positive over big timed releases.
- vsupalov 9y ago
  Good point. But that's not limited to automation. As with data engineering, or software development, you're likely to overdo it and build something which does not agree with reality if you "try to build the whole thing" at once. Building the smallest possible deployment pipeline (a skeleton pipeline, as it's called in the continuous delivery book (from 2010 but still the best thing on the topic out there) [1]). You start covering the most essential needs just so they become useful and usable. Then you go ahead and iterate from there, building something which suits your team, company & product. Learning and adapting in the process. [1] https://www.amazon.de/Continuous-Delivery-Deployment-Automation-Addison-Wesley/dp/0321601912 https://www.amazon.de/Continuous-Delivery-Deployment-Automat...
- perlgeek 9y ago
  We have automated our deployments using https://gocd.org/ https://gocd.org/ (which mostly runs Ansible scripts that do the actual deployment), and we use manual approval steps to trigger production deployments. This is a very nice escape hatch for when you want to do manual steps (like manual testing) before a deployment, but you can still rely on the fact that deployment to prod will use the same playbook as deploying to staging did.
- jbattle 9y ago
  I agree that reaching perfect automation isn't critical - but one thing that is critical is to ensure that local uncommitted changes do not get deployed. I've been burned a couple of times where changes from a developer's machine ended up in production
OtterCoder 9y ago
I'm working on CD for a project I've been working on, but one difficulty I'm facing is that the clients and server are being developed on parallel tracks, and aren't always in feature parity at any given moment. Which repo should own the integration tests? How do I synchronize the releases of matching front and back ends?
- beat 9y ago
  Are you versioning your api? I assume the clients are communicating with the server via some sort of api. Document the api in machine-readable form (like, say, Swagger), and have separate test suites for both client and server, to specific api versions. Since both are working against the same (complete and consistent!) spec, they should be able to talk to each other, and integration testing can be pretty minimal. For REST apis, use semantic versioning, and you can embed specific versions in the request headers. The server should be able to respond correctly to a range of different versions (an api call to say what range of versions the server speaks isn't a bad idea). Coarse-grained api revisions can be embedded in the url (the ever popular /v1 and /v2, etc).
- wpietri 9y ago
  This is a strong sign that your client/server distinction is artificial. If it were in a true client/server environment, where you had a variety of different client versions rolled out, then I expect you would have already found the obvious answer: the server has to support multiple versions of clients and clients use a mechanism like feature switches or capability detection to enable functionality as it becomes available server side. Both repos have their own integration tests: the server's make sure the server supports multiple client versions, and the client's make sure that the client degrades gracefully in the face of server variation. From the way you talk, though, it sounds to me like you expect client and server to always be released at the same time (e.g., where it's a web front end and web back end). If that's really the case, then I'd just have everybody work as one team, working off a common set of features switches. Is that helpful?
  - OtterCoder 9y ago
    The distinction is only artificial because we are in early stages of development. Our MVP will require a web client and an intermittently offline mobile client. It is helpful, but certainly doesn't sound simple. Feature switches would require the messages to already be designed, which is most of the work already done. It also sounds like an edge-case nightmare.
coldcode 9y ago
I find continuous delivery to the mobile app stores to be rather silly and wasteful, updating your app every two weeks for example consumes vast bandwidth especially for people with automatic updates on. The benefits of changing apps on such a quick basis makes it unlikely customers will even notice changes or be able to adapt to what's new or different. Being able to delivery quickly is not the same as having it be automatically useful, just as being able to easily add some new functionality is not the same as having that be useful or desirable to the end user.
- chaosphere2112 9y ago
  On the bandwidth end, both Android and iOS do use incremental updates ([1], [2]); if the changes are something that you would be releasing eventually anyway, you're not wasting any bandwidth, and are instead loadbalancing it over multiple payment periods. [1]: http://www.androidpolice.com/2016/07/23/new-play-store-tools-help-developers-to-shrink-the-size-of-app-updates/ http://www.androidpolice.com/2016/07/23/new-play-store-tools... [2]: https://developer.apple.com/library/content/qa/qa1779/_index.html https://developer.apple.com/library/content/qa/qa1779/_index...
- perlgeek 9y ago
  In such cases, you can practice Continuous Delivery to a beta program, where users explicitly sign up to live on the (b)leading edge. That way you get some of the benefits of CD, but you can still do relatively infrequent releases to the "real" app store.
korzun 9y ago
Good article. I want to offer my thoughts on a couple of things from my personal experience. > If the change deployed is small, there is less code to look through in case of a problem. If you only deploy new software every three weeks, there is a lot more code that could be causing a problem. That's relative. Pushing out an accumulated amount of small changes once a week will most likely have the same end the result. The difference is, if you commit more than one breaking change you are dynamically expanding the window of service degradation. One release with three breaking changes is better than three broken pushes. > If a problem can’t be found or fixed quickly, it is also a lot easier to revert a small deploy than a large deploy. It is also harder to revert two non-consecutive deploys out of three. > If I deploy a new feature as soon as it is ready, everything about it is fresh in my mind. So if there is a problem, trouble shooting is easier than if I have worked on other features in between. Personally, I favor stability vs. easier troubleshooting. This works for some products and not others. > It is also frees up mental energy to be completely done with a feature (including deployed to production). Anecdotal evidence, but my team would usually catch and correct bugs when they have to come back to green light a production push. Engineers that ship clean and fast are rare. > All things being equal, the faster a feature reaches the customer, the better. Having a feature ready for production, but not deploying it, is wasteful. Something like this would usually be pushed out manually to align with other non-engineering parties within your company. Pushing broken features to the customer faster is not a good thing. Unless you can assume 100% success rate; which is not possible. > The sooner the customer starts using the new feature, the sooner you hear what works, what doesn’t work, and what improvements they would like. This depends on the stage of the company, the product, and your customers. > Furthermore, as valuable as testing is, it is never as good as running new code in production. The configuration and data in the production environment will reveal problems that you would never find in testing. All of the environments I govern match production 1:1 (sans data sanitation) in every way possible. I feel pretty strongly about this, if you can't test your code without pushing it into production, you should not be automating anything. > Continuous delivery works best when the developers creating the new features are the ones deploying them. There are no hand-offs – the same person writes the code, tests, deploys and debugs if necessary. This quote (from Werner Vogels, CTO of Amazon) sums it up perfectly: “You built it, you run it.” Don't compare a start-up to Amazon. Amazon has dedicated teams to govern the process and you most likely not replicate that. Also, hiring people that 'just send it' without doing damage takes money, time and a lot of training. It's expensive.
github-cat 9y ago
Hmm, another big challenge for continuous delivery is actually human factor. You need to have people who can work in continuous delivery mode. This factor is more critical than that of traditional software development based on my experience.

82 points by henrik_w 9y ago

Great article! A tiny nitpick: the distinction between continuous delivery and continuous deployment, is that in the first case you could deploy anytime your want, but the triggering is still up to a human. With continuous deployment, everything is shipped to prod automatically, given that all conditions are met. If you want to learn more quickly - I did a talk on the topic last week, and did my best to provide a concise overview of the most essential terms. Check out the slides for a high-level view on ci/cd [1] and deployment pipelines in general [2] if you want to learn more. [1] https://www.slideshare.net/VladislavSupalov/automated-testing-environments-with-kubernetes-gitlab/12 https://www.slideshare.net/VladislavSupalov/automated-testin... [2] https://www.slideshare.net/VladislavSupalov/automated-testing-environments-with-kubernetes-gitlab/17 https://www.slideshare.net/VladislavSupalov/automated-testin...

korzun 9y ago
I think you are nitpicking just to nitpick. Technically, continuous deployment is still 100% up to a human.
rhizome 9y ago
Your "continuous delivery" definition sounds like CI to me.

atsaloli 9y ago

If anyone wants to learn how to set up CI/CD, I have a free self-paced class at https://gitpitch.com/atsaloli/cicd/master?grs=gitlab#/ https://gitpitch.com/atsaloli/cicd/master?grs=gitlab#/ -- all you need to follow along is an Ubuntu VM for the hands-on exercises.

deleted 9y ago

[deleted]

zeroz 9y ago

Good summary. One missing argument IMHO against continuous deployment: Fear of consequences in strong regulated sectors, like finance and insurance. Continuous delivery is highly encouraged, but for deployments I see strong preferences to test everything (in some parts automation is still weak) and therefor some bundling of deployments or special dates are still preferred. I think costs of bad reputation or being watch by regulators because of failed or illegal 'transactions' are in these businesses much higher than e.g. in retail, gaming, etc.

brightball 9y ago
Continuous deployment is something that you can only truly achieve when you have a very high degree of confidence in your automated testing IMHO.
beat 9y ago
And meanwhile, we have Equifax losing tons of valuable data due to a breach caused largely by how slowly they deploy, and how difficult it is for them to get rid of antiquated technology. After many years in big enterprise, I've learned something important - the appearance of risk is more important than the existence of risk. Continuous delivery looks "risky". Slow, deliberate release cycles on a quarterly or even yearly basis look "safe", because "testing". In practice, those quarterly deployments have far too many changes embedded in them all at once. Worse, teams race to get their features in under the deadline, knowing it can be months before they'll get another chance, leading to careless coding and inadequate testing. So, based on both my experience and a little beyond-common-sense logic, slow release cycles are more risky than fast ones.
- zeroz 9y ago
  I absolutely agree with you! These huge quarterly updates with last minute hop on's and changes ("otherwise we have to wait another three months") which large enterprises did or are still doing are more risky than smaller ones. Does this require to jump directly to continuous deployment? I don't think so. What's wrong with continuous delivery to user acceptance stage and e.g. two weekly deployments after ~98% automated tests ~2% manuel test (especially penetration).

ryanbrunner 9y ago

I like that this article doesn't focus too much on the technical aspects of auto-deploys and CI. I've been in a lot of places where the concepts were lumped together, and while we could have easily delivered software continuously, we falsely believed we needed every last thing to be perfectly automated before we did. It's important that your deployment process is repeatable and simple (i.e. typing one or two commands into a terminal), but if a human is still kicking it off that's still a net positive over big timed releases.

vsupalov 9y ago
Good point. But that's not limited to automation. As with data engineering, or software development, you're likely to overdo it and build something which does not agree with reality if you "try to build the whole thing" at once. Building the smallest possible deployment pipeline (a skeleton pipeline, as it's called in the continuous delivery book (from 2010 but still the best thing on the topic out there) [1]). You start covering the most essential needs just so they become useful and usable. Then you go ahead and iterate from there, building something which suits your team, company & product. Learning and adapting in the process. [1] https://www.amazon.de/Continuous-Delivery-Deployment-Automation-Addison-Wesley/dp/0321601912 https://www.amazon.de/Continuous-Delivery-Deployment-Automat...
perlgeek 9y ago
We have automated our deployments using https://gocd.org/ https://gocd.org/ (which mostly runs Ansible scripts that do the actual deployment), and we use manual approval steps to trigger production deployments. This is a very nice escape hatch for when you want to do manual steps (like manual testing) before a deployment, but you can still rely on the fact that deployment to prod will use the same playbook as deploying to staging did.
jbattle 9y ago
I agree that reaching perfect automation isn't critical - but one thing that is critical is to ensure that local uncommitted changes do not get deployed. I've been burned a couple of times where changes from a developer's machine ended up in production

OtterCoder 9y ago

I'm working on CD for a project I've been working on, but one difficulty I'm facing is that the clients and server are being developed on parallel tracks, and aren't always in feature parity at any given moment. Which repo should own the integration tests? How do I synchronize the releases of matching front and back ends?

beat 9y ago
Are you versioning your api? I assume the clients are communicating with the server via some sort of api. Document the api in machine-readable form (like, say, Swagger), and have separate test suites for both client and server, to specific api versions. Since both are working against the same (complete and consistent!) spec, they should be able to talk to each other, and integration testing can be pretty minimal. For REST apis, use semantic versioning, and you can embed specific versions in the request headers. The server should be able to respond correctly to a range of different versions (an api call to say what range of versions the server speaks isn't a bad idea). Coarse-grained api revisions can be embedded in the url (the ever popular /v1 and /v2, etc).
wpietri 9y ago
This is a strong sign that your client/server distinction is artificial. If it were in a true client/server environment, where you had a variety of different client versions rolled out, then I expect you would have already found the obvious answer: the server has to support multiple versions of clients and clients use a mechanism like feature switches or capability detection to enable functionality as it becomes available server side. Both repos have their own integration tests: the server's make sure the server supports multiple client versions, and the client's make sure that the client degrades gracefully in the face of server variation. From the way you talk, though, it sounds to me like you expect client and server to always be released at the same time (e.g., where it's a web front end and web back end). If that's really the case, then I'd just have everybody work as one team, working off a common set of features switches. Is that helpful?
- OtterCoder 9y ago
  The distinction is only artificial because we are in early stages of development. Our MVP will require a web client and an intermittently offline mobile client. It is helpful, but certainly doesn't sound simple. Feature switches would require the messages to already be designed, which is most of the work already done. It also sounds like an edge-case nightmare.

coldcode 9y ago

I find continuous delivery to the mobile app stores to be rather silly and wasteful, updating your app every two weeks for example consumes vast bandwidth especially for people with automatic updates on. The benefits of changing apps on such a quick basis makes it unlikely customers will even notice changes or be able to adapt to what's new or different. Being able to delivery quickly is not the same as having it be automatically useful, just as being able to easily add some new functionality is not the same as having that be useful or desirable to the end user.

chaosphere2112 9y ago
On the bandwidth end, both Android and iOS do use incremental updates ([1], [2]); if the changes are something that you would be releasing eventually anyway, you're not wasting any bandwidth, and are instead loadbalancing it over multiple payment periods. [1]: http://www.androidpolice.com/2016/07/23/new-play-store-tools-help-developers-to-shrink-the-size-of-app-updates/ http://www.androidpolice.com/2016/07/23/new-play-store-tools... [2]: https://developer.apple.com/library/content/qa/qa1779/_index.html https://developer.apple.com/library/content/qa/qa1779/_index...
perlgeek 9y ago
In such cases, you can practice Continuous Delivery to a beta program, where users explicitly sign up to live on the (b)leading edge. That way you get some of the benefits of CD, but you can still do relatively infrequent releases to the "real" app store.

korzun 9y ago

Good article. I want to offer my thoughts on a couple of things from my personal experience. > If the change deployed is small, there is less code to look through in case of a problem. If you only deploy new software every three weeks, there is a lot more code that could be causing a problem. That's relative. Pushing out an accumulated amount of small changes once a week will most likely have the same end the result. The difference is, if you commit more than one breaking change you are dynamically expanding the window of service degradation. One release with three breaking changes is better than three broken pushes. > If a problem can’t be found or fixed quickly, it is also a lot easier to revert a small deploy than a large deploy. It is also harder to revert two non-consecutive deploys out of three. > If I deploy a new feature as soon as it is ready, everything about it is fresh in my mind. So if there is a problem, trouble shooting is easier than if I have worked on other features in between. Personally, I favor stability vs. easier troubleshooting. This works for some products and not others. > It is also frees up mental energy to be completely done with a feature (including deployed to production). Anecdotal evidence, but my team would usually catch and correct bugs when they have to come back to green light a production push. Engineers that ship clean and fast are rare. > All things being equal, the faster a feature reaches the customer, the better. Having a feature ready for production, but not deploying it, is wasteful. Something like this would usually be pushed out manually to align with other non-engineering parties within your company. Pushing broken features to the customer faster is not a good thing. Unless you can assume 100% success rate; which is not possible. > The sooner the customer starts using the new feature, the sooner you hear what works, what doesn’t work, and what improvements they would like. This depends on the stage of the company, the product, and your customers. > Furthermore, as valuable as testing is, it is never as good as running new code in production. The configuration and data in the production environment will reveal problems that you would never find in testing. All of the environments I govern match production 1:1 (sans data sanitation) in every way possible. I feel pretty strongly about this, if you can't test your code without pushing it into production, you should not be automating anything. > Continuous delivery works best when the developers creating the new features are the ones deploying them. There are no hand-offs – the same person writes the code, tests, deploys and debugs if necessary. This quote (from Werner Vogels, CTO of Amazon) sums it up perfectly: “You built it, you run it.” Don't compare a start-up to Amazon. Amazon has dedicated teams to govern the process and you most likely not replicate that. Also, hiring people that 'just send it' without doing damage takes money, time and a lot of training. It's expensive.

github-cat 9y ago

Hmm, another big challenge for continuous delivery is actually human factor. You need to have people who can work in continuous delivery mode. This factor is more critical than that of traditional software development based on my experience.