6 ms·

"Very few teams are actually practicing CI/CD" is mentioned in the article. In an enterprise setting, due to segregation of duties, it's often true that CD is

by mborch 6y ago

"Very few teams are actually practicing CI/CD" is mentioned in the article. In an enterprise setting, due to segregation of duties, it's often true that CD is impossible due to restrictive change management processes that result in unpredictable and typically week-long cycles.

digikata 6y ago
I guess I think CD could still be valuable in that kind of setting in terms maintaining a deployment ready release that is ready to enter an externally imposed release cycle. (including delivering change mgmt artifacts etc...).
- vp8989 6y ago
  That's not CD though.
  - dunreith 6y ago
    Depends on which CD you're talking about. Continuous Delivery: artifacts get delivered and are ready to deploy when someone is ready to do it. Continuous Deployment: built artifacts get deployed automatically.
    - jasonpeacock 6y ago
      > when someone is ready to do it That's not automated. CD (however you (re)define it) is about automation and removing people from the loop. As long as you have manual processes, it's not continuous.
    - vp8989 6y ago
      The article talks about Continuous Deployment. IMO Continuous Delivery is kind of a nothingburger vague process term that lots of people can (and do) talk themselves into saying/thinking they are doing. That's kind of the theme of this blog post BTW. Continuous Deployment is the actual indisputable thing (you either automatically deploy new commits in mainline or you don't) that gives you the real benefits but which lots of orgs aren't yet doing.
      - galangalalgol 6y ago
        Can anyone besides web get away with continuous deployment? IOS already wants to update too often, I don't want CD to my phones or probably my desktop or server OS. Especially in enterprise. And in cases where hardware hasn't even been released it is impossible, much less undesirable. It makes sense to talk about continuous delevery amd how to squeeze out extrabenefits from that model. I suspect it is mainly about automating as much as possible between CDel and CDep but I have obly ever worked in CDel on hardware that doesn't even exist yet.
        necovek 6y ago
        Things like classic Ubuntu get continuously updated with security fixes, and with livepatch, even kernel gets updated without a reboot. Even their process is very much like true CI/CD but packages only get into "proposed" archive (still accessible to the public) without further human vetting (I mean, a code review is human vetting too). I personally want my desktop/server apps to automatically update according to traditional LTS rules (no breaking changes, just bugfixes).
    - dunreith 6y ago
      > As long as you have manual processes, it's not continuous. Agreed. I'm all-in for Continuous Deployment. Continuous Delivery is a half-measure at best.
signal11 6y ago
It depends on the nature of the business. My experience (regulated financial institution) is that both front office and back office have teams which release hundreds of times a week. It does depend on the maturity of the team (including the business which it’s part of), however. And not all teams are at this level, because of their own particular limitations. Don’t do CD as a tech/ego benchmark thing, however (or Facebook/Google envy). Do it if you can agree that this brings some benefits to the business — eg reduced risk, higher throughput, etc. Also delivering faster really forces you to examine your process for bottlenecks and strip away overhead. I really like this line from the blog: > The teams who have achieved CI/CD have not done so because they are better engineers than the rest of us. I promise you. They are teams that pay more attention to process than the rest of us.
- detaro 6y ago
  Delivery method also matters. E.g. having actual continuous deployment to customer devices in a typical embedded context is very different from continuously deploying a web app. Depending on the circumstances not always impossible, but the factors feeding into the evaluation if it's worth it are very different.
- lhorie 6y ago
  > Don’t do CD as a tech/ego benchmark thing, however (or Facebook/Google envy). Do it if you can agree that this brings some benefits to the business This is a really important point. Rapid releases are not always inherently better. I used to work in a company that dealt with heavy regulatory bureaucracy (in the sense that putting out wrong/misleading information could result in class action lawsuits), and because of that, workflows were highly optimized for waterfall delivery. This meant aggressive reliance on classical project management tools and techniques, and hyper-specialization (as opposed to the sort of full stack engineer approach taken by many bay area tech companies) - and this has worked well for that company for decades. You can also do hybrid approaches, e.g. continuous deployment to a QA environment, and have dedicated QA teams, while delivering to client on a waterfall schedule.
  - bmhin 6y ago
vp8989 6y ago
Anecdotally this has largely been my experience, too. TL;DR ... everyone has Jenkins but they can't just let it deploy stuff because there is tons of "professionalism theatre" bureaucracy baked into the SDLC implemented as crappy homegrown scripts that check you filled in fields in JIRA. On top of that the org likely has incorrect thinking that the way to reduce bad outcomes from software is to change it less often.
- dunreith 6y ago
  > incorrect thinking that the way to reduce bad outcomes from software is to change it less often This. So much this. The way you turn a bug into a "bad outcome" is to make it slower to change. If you can deploy quickly and easily, you can fix any software issue before it becomes a big problem.
jdbernard 6y ago
Agreed. In the data points from my professional career it's only been small companies and teams that have actual CD to prod. The closest I've been able to come in enterprise environments is CD to a lower environment (got all the way to UAT once!) with manual promotion to prod. But that's not the same.
- dunreith 6y ago
  I'm currently at a Fortune 500 and we're doing CD for our eCommerce site. I count myself very lucky to have been involved in the process.
- signal11 6y ago
  My workplace is probably one of the larger ones in HN (tens of thousands of people in technology), and we do CD all the way to production. We’re also extremely regulated. I agree that we’ve been lucky though, to have great tech leadership on this. And also a very supportive business. The regulators have not had any problem with CD or high cadence at all — because we can also show that our systems actually got more stable as we adopted CD “in spirit” and increased release cadence. Also, because we no longer do manual promotions, we’ve improved our Cyber-sec position by a lot — just one of the many advantages of automation.
arminiusreturns 6y ago
I would like to hear more talk of how we can approach this problem. When you are in a place like that, whats the way to get management and customer buy-in, especially in the certain industries (fin, gov, etc) where they seem to think monthly deploys are almost too fast.

by mborch 6y ago

digikata 6y ago

I guess I think CD could still be valuable in that kind of setting in terms maintaining a deployment ready release that is ready to enter an externally imposed release cycle. (including delivering change mgmt artifacts etc...).

vp8989 6y ago
That's not CD though.
- dunreith 6y ago
  Depends on which CD you're talking about. Continuous Delivery: artifacts get delivered and are ready to deploy when someone is ready to do it. Continuous Deployment: built artifacts get deployed automatically.
  - jasonpeacock 6y ago
    > when someone is ready to do it That's not automated. CD (however you (re)define it) is about automation and removing people from the loop. As long as you have manual processes, it's not continuous.
  - vp8989 6y ago
    The article talks about Continuous Deployment. IMO Continuous Delivery is kind of a nothingburger vague process term that lots of people can (and do) talk themselves into saying/thinking they are doing. That's kind of the theme of this blog post BTW. Continuous Deployment is the actual indisputable thing (you either automatically deploy new commits in mainline or you don't) that gives you the real benefits but which lots of orgs aren't yet doing.
    - galangalalgol 6y ago
      Can anyone besides web get away with continuous deployment? IOS already wants to update too often, I don't want CD to my phones or probably my desktop or server OS. Especially in enterprise. And in cases where hardware hasn't even been released it is impossible, much less undesirable. It makes sense to talk about continuous delevery amd how to squeeze out extrabenefits from that model. I suspect it is mainly about automating as much as possible between CDel and CDep but I have obly ever worked in CDel on hardware that doesn't even exist yet.
      - necovek 6y ago
        Things like classic Ubuntu get continuously updated with security fixes, and with livepatch, even kernel gets updated without a reboot. Even their process is very much like true CI/CD but packages only get into "proposed" archive (still accessible to the public) without further human vetting (I mean, a code review is human vetting too). I personally want my desktop/server apps to automatically update according to traditional LTS rules (no breaking changes, just bugfixes).
  - dunreith 6y ago
    > As long as you have manual processes, it's not continuous. Agreed. I'm all-in for Continuous Deployment. Continuous Delivery is a half-measure at best.

signal11 6y ago

It depends on the nature of the business. My experience (regulated financial institution) is that both front office and back office have teams which release hundreds of times a week. It does depend on the maturity of the team (including the business which it’s part of), however. And not all teams are at this level, because of their own particular limitations. Don’t do CD as a tech/ego benchmark thing, however (or Facebook/Google envy). Do it if you can agree that this brings some benefits to the business — eg reduced risk, higher throughput, etc. Also delivering faster really forces you to examine your process for bottlenecks and strip away overhead. I really like this line from the blog: > The teams who have achieved CI/CD have not done so because they are better engineers than the rest of us. I promise you. They are teams that pay more attention to process than the rest of us.

detaro 6y ago
Delivery method also matters. E.g. having actual continuous deployment to customer devices in a typical embedded context is very different from continuously deploying a web app. Depending on the circumstances not always impossible, but the factors feeding into the evaluation if it's worth it are very different.
lhorie 6y ago
> Don’t do CD as a tech/ego benchmark thing, however (or Facebook/Google envy). Do it if you can agree that this brings some benefits to the business This is a really important point. Rapid releases are not always inherently better. I used to work in a company that dealt with heavy regulatory bureaucracy (in the sense that putting out wrong/misleading information could result in class action lawsuits), and because of that, workflows were highly optimized for waterfall delivery. This meant aggressive reliance on classical project management tools and techniques, and hyper-specialization (as opposed to the sort of full stack engineer approach taken by many bay area tech companies) - and this has worked well for that company for decades. You can also do hybrid approaches, e.g. continuous deployment to a QA environment, and have dedicated QA teams, while delivering to client on a waterfall schedule.
- bmhin 6y ago

vp8989 6y ago

Anecdotally this has largely been my experience, too. TL;DR ... everyone has Jenkins but they can't just let it deploy stuff because there is tons of "professionalism theatre" bureaucracy baked into the SDLC implemented as crappy homegrown scripts that check you filled in fields in JIRA. On top of that the org likely has incorrect thinking that the way to reduce bad outcomes from software is to change it less often.

dunreith 6y ago
> incorrect thinking that the way to reduce bad outcomes from software is to change it less often This. So much this. The way you turn a bug into a "bad outcome" is to make it slower to change. If you can deploy quickly and easily, you can fix any software issue before it becomes a big problem.

jdbernard 6y ago

Agreed. In the data points from my professional career it's only been small companies and teams that have actual CD to prod. The closest I've been able to come in enterprise environments is CD to a lower environment (got all the way to UAT once!) with manual promotion to prod. But that's not the same.

dunreith 6y ago
I'm currently at a Fortune 500 and we're doing CD for our eCommerce site. I count myself very lucky to have been involved in the process.
signal11 6y ago
My workplace is probably one of the larger ones in HN (tens of thousands of people in technology), and we do CD all the way to production. We’re also extremely regulated. I agree that we’ve been lucky though, to have great tech leadership on this. And also a very supportive business. The regulators have not had any problem with CD or high cadence at all — because we can also show that our systems actually got more stable as we adopted CD “in spirit” and increased release cadence. Also, because we no longer do manual promotions, we’ve improved our Cyber-sec position by a lot — just one of the many advantages of automation.

arminiusreturns 6y ago

I would like to hear more talk of how we can approach this problem. When you are in a place like that, whats the way to get management and customer buy-in, especially in the certain industries (fin, gov, etc) where they seem to think monthly deploys are almost too fast.