4 ms·

To paraphrase the maxim, working automated systems evolve from working manual systems. But only some manual systems work. I start CI/CD by doing the whole proc

To paraphrase the maxim, working automated systems evolve from working manual systems. But only some manual systems work. I start CI/CD by doing the whole process manually. For example, type the commands to build a Docker image, or spawn a VM, or obtain a secret. I encode all this in pseudo code, then put it in Bash (or Python). When a conditional branch appears (a different environment with different credentials), I treat it like any other code. Separate the bits that stay the same, and inject the bits that change. The problem with most CI/CD systems is that people tightly couple themselves to the tool without really understanding it - the point the article is making. They over-complicate the solution because the documentation encourages you to do that. When they want to customise, debug, or even migrate away from it, it’s very difficult.

codeflo 5y ago
> Separate the bits that stay the same, and inject the bits that change. Basically, treat your build/deployment scripts like you would any other code. Strive to have a single source of truth for any fact that can change etc. That seems obvious, but in my experience was at first counterintuitive for many people who otherwise had no trouble applying these principles in application code. A lot of modern CI/CD principles boil down to “it’s just code”.
- fennecfoxen 5y ago
  That includes the part where “wouldn’t it be great if we could run just this one piece of code by itself sometimes, instead of having it embedded in a big ball of mud with tons of shared state and invisible dependencies?”
harperlee 5y ago
Very related to the idea of “Do-nothing scripting”: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=Do%20nothing%20script&sort=byPopularity&type=story https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
kqr 5y ago
People vastly underestimate manual processes. A properly optimised, trimmed, practiced manual process can be executed much more cheaply than most people think. What's the point? Flexibility. Humans are amazing at adapting to circumstances that invalidate portions of the process. In a fast-changing world, this is critical. Really solid automation is notoriously expensive to set up, and about as expensive to change when the circumstances fundamentally change. When, not if. (You can usually still scrape by on an old automated process that's no longer relevant, but you're leaving oodles of value on the floor.) Why do people underestimate manual processes? They usually don't know how to optimise them (hint: it involves restructuring things you think of as outside the process in question) so they associate manual with inefficient and expensive. When you have a time-tested optimised manual process, it might not even make economic sense to automate it. But if it does, you don't have to do it all at once. You can start by automating the dumbest, simplest, most tedious, and most stable step of the process first. Then you can continue to introduce automated steps only for as long as it makes economic sense to do so. ---- Another common complaint about manual processes is that they invite mistakes. I have three things to say on that: - There are ways to mistake-proof manual processes. "Poka-yoke" is a good starting word for googling. - Automated processes make mistakes too. The difference is that an automated process makes mistakes more consistently. (Which may or may not be a good thing, depending on context.) - Manual processes can avoid mistakes that are about to happen. In fact, I'd argue this happens at about the same rate as mistakes are made.
- Teletio 5y ago
  Funny that you advocate for manual things while I have the feeling we do too much manual and advocate for the opposite. Especially the error scenarios are bad. They cost you time no one measures. Manual processes are also difficult to sync across multiple team members and you need tooling around it to make sure manual things happen. My mantra / priority looks more like this: 1. Try not to do it at all 2. Make it automated 3. Do it manually with a heartbeat system I don't want to do things manually I prefer to be able to go to a beer garden in the summer and being flexible. And as an endpoint: automation for me is the necessary base of adding additional value with high return. Only with an automation base you can extend it by fixing more and more issues automatically. While you fix the full disk issue a 100 times, I fix it once.
nonrandomstring 5y ago
A practice I've found valuable in the past is the script command. SCRIPT(1) script - make typescript of terminal session Manually run through a process over and over, checking the intermediate outcomes until its right and tight. Then take the last iteration from script and make that into a shell script. After testing an tweaking few more times now make that portable in a system language of your choice. A kludge I frequently used was just copy it to a Perl file as back-quoted script lines, then replace each with a proper statement, same thing works with python calling process/system/exec things that return proper exit values. Dirty, mechanical, industrial programming. No elegance in sight. But it works reliably in most cases and leaves an understandable trail for debug.
arinlen 5y ago
> The problem with most CI/CD systems is that people tightly couple themselves to the tool without really understanding it - the point the article is making. It's worth noting that tight coupling to CICD services doesn't happen by accident. Not only do CICD services encourage that as a way to hold on to customers but also adding abstraction layers increases the cost of maintaining stuff. I mean, CICD services have their happy path, which implies tight coupling, and doing anything that steps out of this path won't lead to a smooth experience.
pjc50 5y ago
> The problem with most CI/CD systems is that people tightly couple themselves to the tool without really understanding it Related to this, it can be a genuine benefit to adapt your process to the tools rather than the other way round, especially if they're really widely used tools, because then you know you're adopting a process that works for someone. It does however require humility. The absolute worst excesses of "Enterprise" are "custom COTS": buy some software which is built around a standard process, then try to crowbar that to match a completely different one which is inflexible not for business reasons but for organisational-politics reasons.
- headsoup 5y ago
  I find on most occasions "custom COTS" tends to mean "we don't actually know what we do, but off the shelf - top Gartner Quadrant quartile in particular - is best practice and we'll figure it out from there," to the completely expected drawn out suffering that follows. Until the next person does it again with further abstraction from the actual needs.

codeflo 5y ago

> Separate the bits that stay the same, and inject the bits that change. Basically, treat your build/deployment scripts like you would any other code. Strive to have a single source of truth for any fact that can change etc. That seems obvious, but in my experience was at first counterintuitive for many people who otherwise had no trouble applying these principles in application code. A lot of modern CI/CD principles boil down to “it’s just code”.

fennecfoxen 5y ago
That includes the part where “wouldn’t it be great if we could run just this one piece of code by itself sometimes, instead of having it embedded in a big ball of mud with tons of shared state and invisible dependencies?”

harperlee 5y ago

Very related to the idea of “Do-nothing scripting”: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=Do%20nothing%20script&sort=byPopularity&type=story https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

kqr 5y ago

People vastly underestimate manual processes. A properly optimised, trimmed, practiced manual process can be executed much more cheaply than most people think. What's the point? Flexibility. Humans are amazing at adapting to circumstances that invalidate portions of the process. In a fast-changing world, this is critical. Really solid automation is notoriously expensive to set up, and about as expensive to change when the circumstances fundamentally change. When, not if. (You can usually still scrape by on an old automated process that's no longer relevant, but you're leaving oodles of value on the floor.) Why do people underestimate manual processes? They usually don't know how to optimise them (hint: it involves restructuring things you think of as outside the process in question) so they associate manual with inefficient and expensive. When you have a time-tested optimised manual process, it might not even make economic sense to automate it. But if it does, you don't have to do it all at once. You can start by automating the dumbest, simplest, most tedious, and most stable step of the process first. Then you can continue to introduce automated steps only for as long as it makes economic sense to do so. ---- Another common complaint about manual processes is that they invite mistakes. I have three things to say on that: - There are ways to mistake-proof manual processes. "Poka-yoke" is a good starting word for googling. - Automated processes make mistakes too. The difference is that an automated process makes mistakes more consistently. (Which may or may not be a good thing, depending on context.) - Manual processes can avoid mistakes that are about to happen. In fact, I'd argue this happens at about the same rate as mistakes are made.

Teletio 5y ago
Funny that you advocate for manual things while I have the feeling we do too much manual and advocate for the opposite. Especially the error scenarios are bad. They cost you time no one measures. Manual processes are also difficult to sync across multiple team members and you need tooling around it to make sure manual things happen. My mantra / priority looks more like this: 1. Try not to do it at all 2. Make it automated 3. Do it manually with a heartbeat system I don't want to do things manually I prefer to be able to go to a beer garden in the summer and being flexible. And as an endpoint: automation for me is the necessary base of adding additional value with high return. Only with an automation base you can extend it by fixing more and more issues automatically. While you fix the full disk issue a 100 times, I fix it once.

nonrandomstring 5y ago

A practice I've found valuable in the past is the script command. SCRIPT(1) script - make typescript of terminal session Manually run through a process over and over, checking the intermediate outcomes until its right and tight. Then take the last iteration from script and make that into a shell script. After testing an tweaking few more times now make that portable in a system language of your choice. A kludge I frequently used was just copy it to a Perl file as back-quoted script lines, then replace each with a proper statement, same thing works with python calling process/system/exec things that return proper exit values. Dirty, mechanical, industrial programming. No elegance in sight. But it works reliably in most cases and leaves an understandable trail for debug.

arinlen 5y ago

> The problem with most CI/CD systems is that people tightly couple themselves to the tool without really understanding it - the point the article is making. It's worth noting that tight coupling to CICD services doesn't happen by accident. Not only do CICD services encourage that as a way to hold on to customers but also adding abstraction layers increases the cost of maintaining stuff. I mean, CICD services have their happy path, which implies tight coupling, and doing anything that steps out of this path won't lead to a smooth experience.

pjc50 5y ago

> The problem with most CI/CD systems is that people tightly couple themselves to the tool without really understanding it Related to this, it can be a genuine benefit to adapt your process to the tools rather than the other way round, especially if they're really widely used tools, because then you know you're adopting a process that works for someone. It does however require humility. The absolute worst excesses of "Enterprise" are "custom COTS": buy some software which is built around a standard process, then try to crowbar that to match a completely different one which is inflexible not for business reasons but for organisational-politics reasons.

headsoup 5y ago
I find on most occasions "custom COTS" tends to mean "we don't actually know what we do, but off the shelf - top Gartner Quadrant quartile in particular - is best practice and we'll figure it out from there," to the completely expected drawn out suffering that follows. Until the next person does it again with further abstraction from the actual needs.