6 ms·

I once helped lead my team from a tedious manual process into full-CD. Took a lot of pain to get there, and we had to build some new tools to automate all of th

by mabbo 9y ago

I once helped lead my team from a tedious manual process into full-CD. Took a lot of pain to get there, and we had to build some new tools to automate all of the steps we were doing, but we did it. Most of the pain was in realizing we didn't have enough UI regression tests. Every 'incident' sparked a new set of tests and over time the incidents stopped happening. Our velocity increased, time-to-market dropped, operational workload fell. It was great. Sadly, after a few years of the joy of full-CD, more conservative leadership took over our org. Now, all deployments of software must be accompanied by paperwork (digital), step-by-step plans of how you're going to deploy this software, approvals from senior managers, just general micromanagement. CD isn't possible in such environments.

dominotw 9y ago
Can the paperwork be automated. Automatically deploy software once ticket is approved.
- Chyzwar 9y ago
  nope, otherwise said senior management will be no longer needed. These people need to create a veil of importance. They will create multi-level approval process. they will add QA team in India and customer signoff. When something happens blame is still on developers.
  - sidlls 9y ago
    If customers are involved maybe their signoff is necessary (not technically, but contractually). QA seems like an important, necessary step in any delivery process, "continuous" or not: tests aren't infallible and a combination of complementary automated and manual tests aren't a bad thing necessarily. I find often when developers think something that is process driven like this isn't necessary it's because they're not thinking outside their own direct and immediate concerns.
    - Chyzwar 9y ago
      QA is only needed for acceptance testing and should be automated. Not a tens of people doing checklist testing. The customer should sign off on during requirements gathering phase and on frequent demos, not before release. I often think that management was transplanted into the foreign ground of software development and have no clue. They will then come with platinum quarterly releases were testing cost more than development.
      - valuearb 9y ago
        Automated testing is great, but there is something to be said for the value of having real eyes on the product. Often times front ends can pass all the tests, but a good QA person suddenly realizes, wait a second, do we really want it to work this way?
    - crdoconnor 9y ago
      I've been experimenting with auto-generating documentation (with screenshots) from tests as a means of getting quicker and more trusting customer sign off. The way I see it this is a communication feedback loop much like the one between tester and developer or product manager and developer and leaving that gap open means you're one step further away continuous delivery. I don't think it's unreasonable for customers to be somewhat distrustful of us, their black box that spits out code.
Shoothe 9y ago
You could build the sign off phases as part of the deployment pipeline so that each change automatically requests sign-offs from seniors and resumes when they are given. Of course if your management is actively against CD that could be hard but generally approvals are not a big problem.
crdoconnor 9y ago
>Most of the pain was in realizing we didn't have enough UI regression tests. Every 'incident' sparked a new set of tests and over time the incidents stopped happening. I noticed a process similar to this where bugs from production would appear to 'migrate' from tested scenarios to untested scenarios. We'd start off with UI tests on a browser, for example, and then the bugs would be clamped down on there but would seem now to crop up in API interactions or untested browsers. Then we started accommodating the tests to handle different browsers and mocking API interactions and the bugs would start cropping up somewhere else. What was counterintuitive was that, even though code coverage metrics were really quite low, most bugs cropped up in areas of the code with automated test coverage. "Environmental" coverage seemed to matter much, much more.
- tacon 9y ago
  I really like to learn more about this scenario. Is it covered anywhere in more detail? Maybe some examples?
  - 0xcde4c3db 9y ago
    That description reminds me of the Sorcerer's Apprentice Syndrome in TFTP. https://en.wikipedia.org/wiki/Sorcerer%27s_Apprentice_Syndrome https://en.wikipedia.org/wiki/Sorcerer%27s_Apprentice_Syndro...
  - crdoconnor 9y ago
    Not that I know of, sorry.
koolba 9y ago
Fully automatic deploys are distinct from fully automated ones. I insist on the latter but not everyone is comfortable with the former. Large / legacy / lumbering institutions (think big banks) in particular want a change ticket, window or some other control before a release makes it's way to production. The middle ground I've found is to have fully automatic deploys in lower environments and automate the entire process for production. But rather than have releases flow there automatically, production is handled by pressing a single "Deploy whatever is in UAT to prod" button (i.e. the "Big Red Easy Button"). You then give access to that button to somebody else in the org (ex: a UAT team, a release team, your manager) and everybody's happy. It also conveniently gives you an excuse to never fill out production release paperwork as "It's not my job, I'm not allowed to release to production!"
- wpietri 9y ago
  You raise an important point. I think there's a big distinction to be drawn between people's feelings and actual business needs. People everywhere like feelings of control. And the more layers of management a company has, the more executive feelings come to predominate over on-the-ground business facts. But in practice, bureaucratic change-control processes don't actually produce better code or safer systems. In my experience, they make things worse, because large batches and slow feedback loops make bugs easy to add and hard to find. One good next step after the middle ground you describe is continuous deployment of code, but using feature flags and A/B testing tools to control visibility and impact of features. As you say, if you give people some big, obvious controls, they feel in control. Decoupling code release with business impact lets me get on with what I care about. And it lets them learn to be bolder, more engaged in the process of delivering value to users.
- hinkley 9y ago
  There’s a new group learning CD that mistakenly believes that automatic deploys are CD. Most of these came to the concept after it had supplanted discussion about CI. Automating a deploy is relatively straightforward but doesn’t provide much value if that’s all you do. Local reproduction of build steps, metrics, historical data, building out test infrastructure, all of this is important for scaling up a team and keeping regressions low.
goliatone 9y ago
What was the “reason” given to change the process? Were these changes and steps addressing something (regressions, stability, quality...)? Not saying your account is untrue, or that I can’t imagine management doing asinine things but it is still hard for me to imagine a) management willing to change something that’s not broken and b) developers that wouldn’t push back to said change.
- mabbo 9y ago
  Not all teams had moved to full-CD. Not all teams had a good set of tests, automated safe deployment windows, alarms and auto-rollback, all the things you need to deploy any software safely. This lead to high severity incidents, customer dissatisfaction. The solution, of course, is more bureaucracy! And it worked, sort of. Everyone spends all the time writing documents asking for permission to deploy software changes rather than coding, so there's less changes overall, leading to less incidents. Never mind that velocity is a fraction of what it was or might have been. In short, management wants to solve the problem in a way they understand rather than a way that works. And they love a way that has them micromanage everything and nit-pick when you didn't fill out the paperwork correctly. Now, I should caveat all this- I'm a terrible pessimist sometimes and also I haven't had any caffeine today which can only make things worse.
devonkim 9y ago
I’m pretty done with old school change managers that act like ITIL is the only sane way to practice safe, reliable management of systems. In fact, I’ve never, ever seen anyone go full hog ITIL in IT leadership in the F500 that had ever worked at a high performance engineering or IT organization - it appears to be correlated like Six Sigma with low performing organizations after they adopt it (performance in the HBR study I saw for Six Sigma correlated negatively with the S&P 500). What I used to do for demos was to automate the manual processes of ITIL style change review including diffs of the code, test plans executed, test coverage, etc. and the dependency graph of CI impacted by a change (because CMDBs are broken in 99.9% of orgs, you need tools like CloudFormation, Terraform, etc. to give the granularity necessary for a thorough review even in bureaucratic hell). All the other parts of dealing with bureaucracy is pretty much filling out other fields like a list of approvers, signed commits, org, billing codes, etc. are all a matter of trivia rather than smarts. Your changesets need to become higher grained enough to warrant all the human eyes involved. But going backwards like that in processes is usually a sign that a place that is in decline though so that alone should be setting off warnings.

by mabbo 9y ago

dominotw 9y ago

Can the paperwork be automated. Automatically deploy software once ticket is approved.

Chyzwar 9y ago
nope, otherwise said senior management will be no longer needed. These people need to create a veil of importance. They will create multi-level approval process. they will add QA team in India and customer signoff. When something happens blame is still on developers.
- sidlls 9y ago
  If customers are involved maybe their signoff is necessary (not technically, but contractually). QA seems like an important, necessary step in any delivery process, "continuous" or not: tests aren't infallible and a combination of complementary automated and manual tests aren't a bad thing necessarily. I find often when developers think something that is process driven like this isn't necessary it's because they're not thinking outside their own direct and immediate concerns.
  - Chyzwar 9y ago
    QA is only needed for acceptance testing and should be automated. Not a tens of people doing checklist testing. The customer should sign off on during requirements gathering phase and on frequent demos, not before release. I often think that management was transplanted into the foreign ground of software development and have no clue. They will then come with platinum quarterly releases were testing cost more than development.
    - valuearb 9y ago
      Automated testing is great, but there is something to be said for the value of having real eyes on the product. Often times front ends can pass all the tests, but a good QA person suddenly realizes, wait a second, do we really want it to work this way?
  - crdoconnor 9y ago
    I've been experimenting with auto-generating documentation (with screenshots) from tests as a means of getting quicker and more trusting customer sign off. The way I see it this is a communication feedback loop much like the one between tester and developer or product manager and developer and leaving that gap open means you're one step further away continuous delivery. I don't think it's unreasonable for customers to be somewhat distrustful of us, their black box that spits out code.

Shoothe 9y ago

You could build the sign off phases as part of the deployment pipeline so that each change automatically requests sign-offs from seniors and resumes when they are given. Of course if your management is actively against CD that could be hard but generally approvals are not a big problem.

crdoconnor 9y ago

>Most of the pain was in realizing we didn't have enough UI regression tests. Every 'incident' sparked a new set of tests and over time the incidents stopped happening. I noticed a process similar to this where bugs from production would appear to 'migrate' from tested scenarios to untested scenarios. We'd start off with UI tests on a browser, for example, and then the bugs would be clamped down on there but would seem now to crop up in API interactions or untested browsers. Then we started accommodating the tests to handle different browsers and mocking API interactions and the bugs would start cropping up somewhere else. What was counterintuitive was that, even though code coverage metrics were really quite low, most bugs cropped up in areas of the code with automated test coverage. "Environmental" coverage seemed to matter much, much more.

tacon 9y ago
I really like to learn more about this scenario. Is it covered anywhere in more detail? Maybe some examples?
- 0xcde4c3db 9y ago
  That description reminds me of the Sorcerer's Apprentice Syndrome in TFTP. https://en.wikipedia.org/wiki/Sorcerer%27s_Apprentice_Syndrome https://en.wikipedia.org/wiki/Sorcerer%27s_Apprentice_Syndro...
- crdoconnor 9y ago
  Not that I know of, sorry.

koolba 9y ago

Fully automatic deploys are distinct from fully automated ones. I insist on the latter but not everyone is comfortable with the former. Large / legacy / lumbering institutions (think big banks) in particular want a change ticket, window or some other control before a release makes it's way to production. The middle ground I've found is to have fully automatic deploys in lower environments and automate the entire process for production. But rather than have releases flow there automatically, production is handled by pressing a single "Deploy whatever is in UAT to prod" button (i.e. the "Big Red Easy Button"). You then give access to that button to somebody else in the org (ex: a UAT team, a release team, your manager) and everybody's happy. It also conveniently gives you an excuse to never fill out production release paperwork as "It's not my job, I'm not allowed to release to production!"

wpietri 9y ago
You raise an important point. I think there's a big distinction to be drawn between people's feelings and actual business needs. People everywhere like feelings of control. And the more layers of management a company has, the more executive feelings come to predominate over on-the-ground business facts. But in practice, bureaucratic change-control processes don't actually produce better code or safer systems. In my experience, they make things worse, because large batches and slow feedback loops make bugs easy to add and hard to find. One good next step after the middle ground you describe is continuous deployment of code, but using feature flags and A/B testing tools to control visibility and impact of features. As you say, if you give people some big, obvious controls, they feel in control. Decoupling code release with business impact lets me get on with what I care about. And it lets them learn to be bolder, more engaged in the process of delivering value to users.
hinkley 9y ago
There’s a new group learning CD that mistakenly believes that automatic deploys are CD. Most of these came to the concept after it had supplanted discussion about CI. Automating a deploy is relatively straightforward but doesn’t provide much value if that’s all you do. Local reproduction of build steps, metrics, historical data, building out test infrastructure, all of this is important for scaling up a team and keeping regressions low.

goliatone 9y ago

What was the “reason” given to change the process? Were these changes and steps addressing something (regressions, stability, quality...)? Not saying your account is untrue, or that I can’t imagine management doing asinine things but it is still hard for me to imagine a) management willing to change something that’s not broken and b) developers that wouldn’t push back to said change.

mabbo 9y ago
Not all teams had moved to full-CD. Not all teams had a good set of tests, automated safe deployment windows, alarms and auto-rollback, all the things you need to deploy any software safely. This lead to high severity incidents, customer dissatisfaction. The solution, of course, is more bureaucracy! And it worked, sort of. Everyone spends all the time writing documents asking for permission to deploy software changes rather than coding, so there's less changes overall, leading to less incidents. Never mind that velocity is a fraction of what it was or might have been. In short, management wants to solve the problem in a way they understand rather than a way that works. And they love a way that has them micromanage everything and nit-pick when you didn't fill out the paperwork correctly. Now, I should caveat all this- I'm a terrible pessimist sometimes and also I haven't had any caffeine today which can only make things worse.

devonkim 9y ago

I’m pretty done with old school change managers that act like ITIL is the only sane way to practice safe, reliable management of systems. In fact, I’ve never, ever seen anyone go full hog ITIL in IT leadership in the F500 that had ever worked at a high performance engineering or IT organization - it appears to be correlated like Six Sigma with low performing organizations after they adopt it (performance in the HBR study I saw for Six Sigma correlated negatively with the S&P 500). What I used to do for demos was to automate the manual processes of ITIL style change review including diffs of the code, test plans executed, test coverage, etc. and the dependency graph of CI impacted by a change (because CMDBs are broken in 99.9% of orgs, you need tools like CloudFormation, Terraform, etc. to give the granularity necessary for a thorough review even in bureaucratic hell). All the other parts of dealing with bureaucracy is pretty much filling out other fields like a list of approvers, signed commits, org, billing codes, etc. are all a matter of trivia rather than smarts. Your changesets need to become higher grained enough to warrant all the human eyes involved. But going backwards like that in processes is usually a sign that a place that is in decline though so that alone should be setting off warnings.