6 ms·

The rule for CI/CD and DevOps in general is boil your entire build process down to one line: ./build.sh If you want to ship containers somewhere, do it in

The rule for CI/CD and DevOps in general is boil your entire build process down to one line: ./build.sh If you want to ship containers somewhere, do it in your build script where you check to see if you’re running in “CI”. No fancy pants workflow yamls to vendor lock yourself into whatever CI platform you’re using today, or tomorrow. Just checkout, build w/ params, point your coverage checker at it. This is also the same for onboarding new hires. They should be able to checkout, and build, no issues or caveats, setup for local environment. This ensures they are ready to PR by end of the day. (Fmr Director of DevOps for a Fortune 500)

pxc 1y ago
You still inevitably need a bunch of CI platform-specific bullshit for determining "is this a pull request? which branch am I running on?", etc. Depending on what you're trying to do and what tools you're working with, you may need such logic both in an accursed YAML DSL and in your build script. And if you want your CI jobs to do things like report cute little statuses, integrate with your source forge's static analysis results viewer, or block PRs, you have to integrate with the forge at a deeper level. There aren't good tools today for translating between the environment variables or other things that various CI platforms expose, managing secrets (if you use CI to deploy things) that are exposed in platform-specific ways, etc. If all you're doing with CI is spitting out some binaries, sure, I guess. But if you actually ask developers what they want out of CI, it's typically more than that.
- michaelmior 1y ago
  A lot of CI platforms (such as GitHub) spit out a lot of environment variables automatically that can help you with the logic in your build script. If they don't, they should give you a way to set them. One approach is to keep the majority of the logic in your build script and just use the platform-specific stuff to configure the environment for the build script. Of course, as you mention, if you want to do things like comment on PRs or report detailed status information, you have to dig deeper.
  - merb 1y ago
    how can you build your containers in parallel? over multiple machines? I'm not sure that a sh script can do that with github
    - pxc 1y ago
      If you build them with Nix, you can. Just call `nix build` with a trailing `&` a bunch of times. But it's kind of cheating, because the Nix daemon actually handles per-machine scheduling and cross-machine orchestration for you. Just set up some self-hosted runners with Nix and an appropriately configured remote builders configuration to get started. If you really want to, you can graduate after that to a Kubernetes cluster where Nix is available on the nodes. Pass the Nix daemon socket through to your rootless containers, and you'll get caching in the Nix store for free even with your ephemeral containers. But you probably don't need all that anyway. Just buy or rent a big build server. Nix will use as many cores as you have by default. It will be a long time before you can't easily buy or rent a build server big enough.
    - reactordev 1y ago
      these problems are general ones and the solution is the same as running programs in parallel or across machines. When needing to build different architectures (and needing a host to provide the toolchains), what's stopping you from issuing more than 1 command in your CI/CD pipeline? Most pipelines have a way of running something on a specific host. So does k8s, ecs, <pick your provider>, and probably your IT team. My experience, when it gets time to actually build the thing. A one-liner (with args if you need them) is the best approach. If you really REALLY need to, you can have more than one script for doing it - depending on what path down the pipeline you take. Maybe it's 1) ./build.sh -config Release 2) ./deploy.sh -docker -registry=<$REGISTRY> --kick Just try not to go too crazy. The larger the org, the larger this wrangling task can be. Look at Google and gclient/gn. Not saying it's bad, just saying it's complicated for a reason. You don't need that (you'll know if you do). The point I made is I hate when I see 42 lines in a build workflow yaml that isn't syntax highlighted because it's been |'d in there. I think the yaml's of your pipelines, etc, should be configuration for the pipeline and the actual execution should be outsourced to a script you provide.
- oftenwrong 1y ago
  There are some very basic tools that can help with portability, such as https://github.com/milesj/rust-cicd-env https://github.com/milesj/rust-cicd-env , but I agree that there is a lot of proprietary, vendor-specific, valuable functionality available in the average "CI" system that you cannot make effective use of with this approach. Still, it's the approach I generally favor for a number of reasons.
HPsquared 1y ago
Sounds like the Lotus philosophy, "simplify and add lightness".
XorNot 1y ago
The other rule is that script should run as a user. Solely on that working directory. There are too many scripts like that which start, ask for sudo and then it's off to implementing someones "great idea" about your systems network interfaces.
- reactordev 1y ago
  sudo should not be required to build software. If there’s something you require that requires sudo, it’s a pre-build environment setup on your machine. On the host. Or wherever. It’s not part of the build. If you need credentials, get them from secrets or environment variables.
  - immibis 1y ago
    For use cases like making tar files with contents owned by root, Debian developed the tool "fakeroot", which intercepts standard library functions so that when the build script sets a file to be owned by root and then reads the ownership later, it sees it's owned by root, so it records that in the tar file.
    - reactordev 1y ago
      Debian takes the You can’t touch this approach to things to solve their issues. Instead of work arounds, they just hack at the lower kernel level and trace all you do. It’s a flex. fakeroot isn’t the only tool like this. I love me some Debian.
marsven_422 1y ago
[dead]
mrbombastic 1y ago
How do you get caching of build steps with this approach? Or do you just not?
- arianvanp 1y ago
  Use a modern hermetic build system with remote caching or remote execution. Nix, Bazel, buck, pants. Many options
  - pwnna 1y ago
    This is like fighting complexity with even more complexity. Nix and bazel are definitely not close to actually achieving hermetic build at scale. And when they break the complexity increases exponentially to fix.
    - pxc 1y ago
      What's not hermetic with Nix? Are you talking about running with the sandbox disabled, or and macOS quirks? It's pretty damn hard to accidentally depend on the underlying system in an unexpected way with Nix.
      - wredcoll 1y ago
        My experience with nix, at a smaller scale than what you're talking about, is that it only worked as long as every. single. thing. was reimplemented inside nix. Once one thing was outside of nix, everything exploded and writing a workaround was miserable because the nix configuration did not make it easy.
        pxc 1y ago
        > every. single. thing. was reimplemented inside nix That's kinda what hermetic means, though, isn't it? Whether that's painful or not, that's pretty much exactly what GGP was asking for! > Once one thing was outside of nix, everything exploded and writing a workaround was miserable because the nix configuration did not make it easy. Nix doesn't make it easy to have Nix builds depend on non-Nix things (this is required for hermeticity), but the other way around is usually less troublesome. Still, I know what you mean. What languages were you working in?
maratc 1y ago
Yeah, that's a good rule. Except, do you want to build Debug or Release? Or maybe RelWithDebugInfo? And do you want that with sanitizers maybe? And what the sanitizers' options should be? Do you want to compile your tests too, if you want to run them later on a different machine? And what about that dependency that takes two hours to compile, maybe you just want to reuse the previous compilation of it? And if so, where to take that from? Etc. etc. Before long, you need another script that will output the train of options to your `build.sh`. (If Fortune 500 companies can do a one-line build with zero parameters, I suspect I'd be very bored there.)
- reactordev 1y ago
  Of course we had parameters but we never ship debug builds. Treat everything like production. If you want to debug, docker compose or add logs and metrics to seek what you find.
BobbyTables2 1y ago
You’re not wrong but your suggestion also throws away a lot of major benefits of CI. I agree jobs should be one liners but we still need more than one… The single job pipeline doesn’t tell you what failed. It doesn’t parallelize unit and integration test suites while dealing with the combinatorial matrix of build type, target device, etc. At some point, a few CI runners become more powerful than a developer’s workstation. Parallelization can really matter for reducing CI times. I’d argue the root of the problem is that we are stuck on using “make” and scripts for local build automation. We need something descriptive enough to describe a meaningful CI pipeline but also allow local execution. Sure, one can develop a bespoke solution, but reinventing the wheel each time gets tiring and eventually becomes a sizable time sink. In principle, we should be able to execute pieces of .gitlab-ci.yml locally, but even that becomes non trivial with all the nonstandard YAML behaviors done in gitlab, not to mention the varied executor types. Instead we have a CI workflow and a local workflow and hope the two are manually kept in sync. In some sense, the current CI-only automation tools shouldn’t even need to exist (gitlab, Jenkins, etc) — why didn’t we just use a cron job running “build.sh” ? I argue these tools should mainly only have to focus on the “reporting/artifacts” with the pipeline execution parts handled elsewhere (or also locally for a developer). Shame on you GitLab!
- reactordev 1y ago
  You are mistaking a build for a pipeline. I still believe in pipelines and configuring the right hosts/runners to produce your artifacts. Your actual build on that host/runner should be a one-liner.

pxc 1y ago

You still inevitably need a bunch of CI platform-specific bullshit for determining "is this a pull request? which branch am I running on?", etc. Depending on what you're trying to do and what tools you're working with, you may need such logic both in an accursed YAML DSL and in your build script. And if you want your CI jobs to do things like report cute little statuses, integrate with your source forge's static analysis results viewer, or block PRs, you have to integrate with the forge at a deeper level. There aren't good tools today for translating between the environment variables or other things that various CI platforms expose, managing secrets (if you use CI to deploy things) that are exposed in platform-specific ways, etc. If all you're doing with CI is spitting out some binaries, sure, I guess. But if you actually ask developers what they want out of CI, it's typically more than that.

michaelmior 1y ago
A lot of CI platforms (such as GitHub) spit out a lot of environment variables automatically that can help you with the logic in your build script. If they don't, they should give you a way to set them. One approach is to keep the majority of the logic in your build script and just use the platform-specific stuff to configure the environment for the build script. Of course, as you mention, if you want to do things like comment on PRs or report detailed status information, you have to dig deeper.
- merb 1y ago
  how can you build your containers in parallel? over multiple machines? I'm not sure that a sh script can do that with github
  - pxc 1y ago
    If you build them with Nix, you can. Just call `nix build` with a trailing `&` a bunch of times. But it's kind of cheating, because the Nix daemon actually handles per-machine scheduling and cross-machine orchestration for you. Just set up some self-hosted runners with Nix and an appropriately configured remote builders configuration to get started. If you really want to, you can graduate after that to a Kubernetes cluster where Nix is available on the nodes. Pass the Nix daemon socket through to your rootless containers, and you'll get caching in the Nix store for free even with your ephemeral containers. But you probably don't need all that anyway. Just buy or rent a big build server. Nix will use as many cores as you have by default. It will be a long time before you can't easily buy or rent a build server big enough.
  - reactordev 1y ago
    these problems are general ones and the solution is the same as running programs in parallel or across machines. When needing to build different architectures (and needing a host to provide the toolchains), what's stopping you from issuing more than 1 command in your CI/CD pipeline? Most pipelines have a way of running something on a specific host. So does k8s, ecs, <pick your provider>, and probably your IT team. My experience, when it gets time to actually build the thing. A one-liner (with args if you need them) is the best approach. If you really REALLY need to, you can have more than one script for doing it - depending on what path down the pipeline you take. Maybe it's 1) ./build.sh -config Release 2) ./deploy.sh -docker -registry=<$REGISTRY> --kick Just try not to go too crazy. The larger the org, the larger this wrangling task can be. Look at Google and gclient/gn. Not saying it's bad, just saying it's complicated for a reason. You don't need that (you'll know if you do). The point I made is I hate when I see 42 lines in a build workflow yaml that isn't syntax highlighted because it's been |'d in there. I think the yaml's of your pipelines, etc, should be configuration for the pipeline and the actual execution should be outsourced to a script you provide.
oftenwrong 1y ago
There are some very basic tools that can help with portability, such as https://github.com/milesj/rust-cicd-env https://github.com/milesj/rust-cicd-env , but I agree that there is a lot of proprietary, vendor-specific, valuable functionality available in the average "CI" system that you cannot make effective use of with this approach. Still, it's the approach I generally favor for a number of reasons.

HPsquared 1y ago

Sounds like the Lotus philosophy, "simplify and add lightness".

XorNot 1y ago

The other rule is that script should run as a user. Solely on that working directory. There are too many scripts like that which start, ask for sudo and then it's off to implementing someones "great idea" about your systems network interfaces.

reactordev 1y ago
sudo should not be required to build software. If there’s something you require that requires sudo, it’s a pre-build environment setup on your machine. On the host. Or wherever. It’s not part of the build. If you need credentials, get them from secrets or environment variables.
- immibis 1y ago
  For use cases like making tar files with contents owned by root, Debian developed the tool "fakeroot", which intercepts standard library functions so that when the build script sets a file to be owned by root and then reads the ownership later, it sees it's owned by root, so it records that in the tar file.
  - reactordev 1y ago
    Debian takes the You can’t touch this approach to things to solve their issues. Instead of work arounds, they just hack at the lower kernel level and trace all you do. It’s a flex. fakeroot isn’t the only tool like this. I love me some Debian.

marsven_422 1y ago

[dead]

mrbombastic 1y ago

How do you get caching of build steps with this approach? Or do you just not?

arianvanp 1y ago
Use a modern hermetic build system with remote caching or remote execution. Nix, Bazel, buck, pants. Many options
- pwnna 1y ago
  This is like fighting complexity with even more complexity. Nix and bazel are definitely not close to actually achieving hermetic build at scale. And when they break the complexity increases exponentially to fix.
  - pxc 1y ago
    What's not hermetic with Nix? Are you talking about running with the sandbox disabled, or and macOS quirks? It's pretty damn hard to accidentally depend on the underlying system in an unexpected way with Nix.
    - wredcoll 1y ago
      My experience with nix, at a smaller scale than what you're talking about, is that it only worked as long as every. single. thing. was reimplemented inside nix. Once one thing was outside of nix, everything exploded and writing a workaround was miserable because the nix configuration did not make it easy.
      - pxc 1y ago
        > every. single. thing. was reimplemented inside nix That's kinda what hermetic means, though, isn't it? Whether that's painful or not, that's pretty much exactly what GGP was asking for! > Once one thing was outside of nix, everything exploded and writing a workaround was miserable because the nix configuration did not make it easy. Nix doesn't make it easy to have Nix builds depend on non-Nix things (this is required for hermeticity), but the other way around is usually less troublesome. Still, I know what you mean. What languages were you working in?

maratc 1y ago

Yeah, that's a good rule. Except, do you want to build Debug or Release? Or maybe RelWithDebugInfo? And do you want that with sanitizers maybe? And what the sanitizers' options should be? Do you want to compile your tests too, if you want to run them later on a different machine? And what about that dependency that takes two hours to compile, maybe you just want to reuse the previous compilation of it? And if so, where to take that from? Etc. etc. Before long, you need another script that will output the train of options to your `build.sh`. (If Fortune 500 companies can do a one-line build with zero parameters, I suspect I'd be very bored there.)

reactordev 1y ago
Of course we had parameters but we never ship debug builds. Treat everything like production. If you want to debug, docker compose or add logs and metrics to seek what you find.

BobbyTables2 1y ago

You’re not wrong but your suggestion also throws away a lot of major benefits of CI. I agree jobs should be one liners but we still need more than one… The single job pipeline doesn’t tell you what failed. It doesn’t parallelize unit and integration test suites while dealing with the combinatorial matrix of build type, target device, etc. At some point, a few CI runners become more powerful than a developer’s workstation. Parallelization can really matter for reducing CI times. I’d argue the root of the problem is that we are stuck on using “make” and scripts for local build automation. We need something descriptive enough to describe a meaningful CI pipeline but also allow local execution. Sure, one can develop a bespoke solution, but reinventing the wheel each time gets tiring and eventually becomes a sizable time sink. In principle, we should be able to execute pieces of .gitlab-ci.yml locally, but even that becomes non trivial with all the nonstandard YAML behaviors done in gitlab, not to mention the varied executor types. Instead we have a CI workflow and a local workflow and hope the two are manually kept in sync. In some sense, the current CI-only automation tools shouldn’t even need to exist (gitlab, Jenkins, etc) — why didn’t we just use a cron job running “build.sh” ? I argue these tools should mainly only have to focus on the “reporting/artifacts” with the pipeline execution parts handled elsewhere (or also locally for a developer). Shame on you GitLab!

reactordev 1y ago
You are mistaking a build for a pipeline. I still believe in pipelines and configuring the right hosts/runners to produce your artifacts. Your actual build on that host/runner should be a one-liner.