3 ms·

Similar notion: Never, ever have more than a simple shell script invocation in your CI configuration. So many projects I've worked on have had nontrivial build

Similar notion: Never, ever have more than a simple shell script invocation in your CI configuration. So many projects I've worked on have had nontrivial build-logic or implicit assumptions in their CI configuration which leads to distrust of local reproducibility. Eliminate as many variables and steps to reproducibility as possible. At the very least, create a `ci.sh` script in your project and make all your CI invocations go through that. If you avoid putting "raw" npm or whatever invocations in your CI config in the first place you won't be tempted to add "just one more option" to your CI config and forget to add that option when running locally. Such a script also becomes a de-facto API for how to use your project. This can help immensely if all of a sudden, for example, you switch programming languages (or major versions) environment assumptions, etc.

chubot 6y ago
Yes definitely, many lines of shell in YAML (or Docker files) is one of my pet peeves. It's always better to invoke a shell script. Then you can use ShellCheck on it, your editor will syntax highlight it, you can parse it with Oil (osh -n) [1], etc. And yes there shouldn't be too much of a difference between what developers do locally and what the CI does. It's sort of an anti-pattern if you have a bunch of automation that can only the CI can run. The developer should be able to run it on their machine too, without CI, and without YAML ... [1] http://www.oilshell.org/why.html#what-you-can-use-right-now http://www.oilshell.org/why.html#what-you-can-use-right-now
paulryanrogers 6y ago
This is the ideal. In practice the CI environment may need optimizations that don't make sense locally without a lot more work.
candiddevmike 6y ago
I try to have CI only run make targets. Helps to reproduce things really easily.
jolux 6y ago
Often when you use the CI config instead of shell scripts it gives you deeper integration with the CI provider, like better error messages and such. I wish there was a way to bridge that gap.
- rmetzler 6y ago
  I used to run gitlab-runner locally to debug the CI pipeline but my team now uses a lot of gitlab features to make the pipelines more DRY and this means CI code is scattered over projects. The easiest way I found to debug it, is to put set -x into the code to print the command with substituted variables.
a1369209993 6y ago
> have had nontrivial build-logic or implicit assumptions in their CI configuration Wait, what? If your CI isn't testing the builds your bugs are (possibly) in then what's the point of CI?
kmarc 6y ago
This. OTOH, the monster I'm currently implementing as a set of ci.sh-like scripts is already way too big. It all started as a couple `kubectl apply`, but now I wish it was an ansible playbook. Some other people on the same project abused groovy to the fullest to create some fancypants Jenkins pipeline. Local reproducibility? Zero.

chubot 6y ago

Yes definitely, many lines of shell in YAML (or Docker files) is one of my pet peeves. It's always better to invoke a shell script. Then you can use ShellCheck on it, your editor will syntax highlight it, you can parse it with Oil (osh -n) [1], etc. And yes there shouldn't be too much of a difference between what developers do locally and what the CI does. It's sort of an anti-pattern if you have a bunch of automation that can only the CI can run. The developer should be able to run it on their machine too, without CI, and without YAML ... [1] http://www.oilshell.org/why.html#what-you-can-use-right-now http://www.oilshell.org/why.html#what-you-can-use-right-now

paulryanrogers 6y ago

This is the ideal. In practice the CI environment may need optimizations that don't make sense locally without a lot more work.

candiddevmike 6y ago

I try to have CI only run make targets. Helps to reproduce things really easily.

jolux 6y ago

Often when you use the CI config instead of shell scripts it gives you deeper integration with the CI provider, like better error messages and such. I wish there was a way to bridge that gap.

rmetzler 6y ago
I used to run gitlab-runner locally to debug the CI pipeline but my team now uses a lot of gitlab features to make the pipelines more DRY and this means CI code is scattered over projects. The easiest way I found to debug it, is to put set -x into the code to print the command with substituted variables.

a1369209993 6y ago

> have had nontrivial build-logic or implicit assumptions in their CI configuration Wait, what? If your CI isn't testing the builds your bugs are (possibly) in then what's the point of CI?

kmarc 6y ago

This. OTOH, the monster I'm currently implementing as a set of ci.sh-like scripts is already way too big. It all started as a couple `kubectl apply`, but now I wish it was an ansible playbook. Some other people on the same project abused groovy to the fullest to create some fancypants Jenkins pipeline. Local reproducibility? Zero.