6 ms·

Rather than replacing your Makefile with GH Actions, replace your GH Actions with a Makefile, and make your GH Actions run `make` in a script task. Do you real

Rather than replacing your Makefile with GH Actions, replace your GH Actions with a Makefile, and make your GH Actions run `make` in a script task. Do you really need that GH Action for pulling Docker images / installing $language_compiler / creating cloud resources ? A `docker` / `curl` / `sudo apt-get install` invocation in a Makefile / script needs to be written once and is the same in CI as on your dev machines. The Action is GH-specific, requires you to look it up, requires you to learn the DSL for invoking it, requires you to keep it up-to-date, requires you to worry about it getting hacked and stealing your secrets, ... A Makefile already supports dependencies between targets. A shell script is a simple DSL for executing processes, piping the output of one to another, and detecting and acting on failure of subcommands. How much YAML spaghetti do you need to write to do those same things in a workflow file?

Gigachad 4y ago
With actions you can run multiple tasks in parallel, restart failed portions without retrying the whole CI, run certain sections with conditions like which branch you are on (different tasks when commit to master vs feature branch)
- Arnavion 4y ago
  >With actions you can run multiple tasks in parallel, If you mean "I want to run one build with the foo feature and one build with the bar feature. Actions lets me run those in parallel", then that is the "strategy" part of the workflow, not the "tasks" part. My comment was about the latter. ("and make your GH Actions run `make` in a script task.") If you mean "I want to run two steps of a job in parallel and then run the rest of the job after they're complete", then shell is a much simpler DSL for that. Running things in parallel is literally a single `&` character. >restart failed portions without retrying the whole CI, run certain sections with conditions like which branch you are on (different tasks when commit to master vs feature branch) So can a shell script.
  - FooBarWidget 4y ago
    > Running things in parallel is literally a single `&` character. Yes now try waiting for all parallel tasks, and error out if at least one errors. And try separating their output so that they don't get interleaved into a big mess. And try by default hiding the outputs of commands that succeeded, only showing those that failed, except when the user explicitly asks for it. Your "simple" shell script now suddenly isn't so simple anymore.
    - stabbles 4y ago
      Using GNU make you can get very nice output sync: all: variant-1 variant-2 @echo done with all variant-%: @echo starting $@ @sleep 1 @echo done $@ Use `make -j2 --output-sync=target`.
      - notpushkin 4y ago
        brew install make gmake -j2 --output-sync=target on macOS.
    - cormacrelf 4y ago
      That’s what GNU Parallel is for. Or Pueue, which gives you a GH-Actions-level feature set but it’s less likely to be installed on any particular machine. Pretty sure you could fetch a pueue binary at the start of an actions script and do everything that way. (These can’t project their tasks over multiple GH Actions runners, eg for multiple OSes. For that you will need to use the YAML. Good compilers will already do work in parallel and max out however many cores they are given, so multi machine is the main use case. Unfortunate.)
    - Arnavion 4y ago
      >Yes now try waiting for all parallel tasks, `wait` >and error out if at least one errors. `wait` propagates the exit status of the waited task. `set -e` triggers the script to exit on any command failure. >And try separating their output so that they don't get interleaved into a big mess. `| while read -r line; do echo "task X: $line"; done` Learn the tools instead of being so confident that it can't be done or that it's complicated.
de_keyboard 4y ago
I agree. GitHub Actions should call your scripts but your scripts should not depend on GitHub Actions API. I also suggest Bazel as a consideration alongside Make. With Bazel, you get two advantages over Make: 1. It is easier to ensure that what GitHub Actions runs and builds is the same as what you have locally, since Bazel can fetch toolchains as part of the build process 2. Using a Bazel remote cache, you do not have to repeat work if the build fails halfway and you need to make some changes before running it again.
- toastal 4y ago
  Same can be said about Nix and as a bonus you get Nix instead of Starlark
  - intelVISA 4y ago
    Nix & Make is the True Engineer(tm)'s Bazel.
    - de_keyboard 4y ago
      Are Nix and Bazel mutually exclusive? https://www.tweag.io/blog/2018-03-15-bazel-nix/ https://www.tweag.io/blog/2018-03-15-bazel-nix/
ilammy 4y ago
This is a lesson that I've learned after going all-out on actions once. Now my makefiles in addition to the usual "make" and "make test" also support "make prerequisites" to prepare a build environment by installing everything necessary and "make ci" to run everything that CI should check. With actual implementation being scripts placed under "scripts/ci". The scripts do provide some goodies when they are run by GitHub Actions – like folding the build logs or installing dependencies differently – but these targets also work on the developer machine.
- jaxn 4y ago
  What about caching to reduce ci time? GH setup scripts cache dependencies in a way that would seem hard to replicate in a make file.
  - makapuf 4y ago
    You can split your make file in two CI steps, one cached, and the other one depending on it.
  - ilammy 4y ago
    If it’s manageable – just don’t. Build from scratch. Make sure your build works from scratch and completes in acceptable timeframe. If it’s painful, treat the root cause and not the symptoms. If it’s unbearable due to circumstances out of your control, there’s nothing wrong with adding some actions/cache steps to .github/workflows – this goes around the build: fetch previous cache before, update the cache after if needed. The build is still reproducible outside of GitHub Actions, but a pinch of magic salt makes it go faster sometimes without being an essential part of the build pipeline married to GitHub. If you need to install a whole host of mostly static dependencies, GitHub Actions support running steps in arbitrary Docker container. Prepare an image beforehand, it can be cached too, now you have a predictable environment. (The only downside is that it doesn’t work on macOS and Windows.)
    - KronisLV 4y ago
      > If you need to install a whole host of mostly static dependencies, GitHub Actions support running steps in arbitrary Docker container. Prepare an image beforehand, it can be cached too, now you have a predictable environment. (The only downside is that it doesn’t work on macOS and Windows.) Actually I use a similar workflow for some of the other projects that I have to work with, keeping everything CI agnostic. I incrementally add various types of dependencies to the container images where the application will be built. For example: 1. common base image (e.g. Debian, Ubuntu, or something like Alpine, or maybe RPM based ones) 2. previous + common tools (optional, if you want to include curl or other tools in the container for debugging or testing stuff quickly) 3. previous + project runtime (depending on tech stack, for example OpenJDK for Java projects) 4. previous + development tools (depending on the tech stack, typically for pulling in dependencies, like Maven, or npm, or whatever) 5. previous + project dependencies (if the project is large and the dependencies change rarely, you can install them once here and the changing 5% or so later) 6. previous + project build (including things like running tests, typically multi-stage with the build and tests, and built app handled separately) Compared to the more "common" way to do things, step #5 probably jumps out the most here, I do a pass of installing all of the dependencies, say, every morning, or hourly in the background, so that later when the project is built the CI can just go: "Hmm, it seems like 95% of the things I need here are already present, I'll just pull the remaining packages (if any)." Clean installs only need to be done when packages are removed, which is also reasonably easy to do. Though the benefits of this aren't quite as staggering, if you use a self-hosted package repository like Sonatype Nexus, which can cache any dependencies that you've used previously and make everything faster on the network I/O side. This only doesn't hold true when actually installing the packages takes up the majority of the time (e.g. compiling native code), in which case the above is still very useful. So, an example of how the stages might look, is as follows: Builder: Ubuntu + tools (optional) + OpenJDK + Maven + project dependencies + project build (and run tests) Runner: Ubuntu + tools (optional) + OpenJDK + built project from last image (using COPY with --from, typically .jar file or app directory) Of course, things are less comfortable when you don't have all of your app's dependencies packaged statically but need them "on the system" instead, like Python packages or Ruby Gems, but then your builder and runner will simply look more alike. For my own personal stuff I also use a slightly simplified version of this, about which I wrote on my blog here, the drawbacks included: https://blog.kronis.dev/articles/using-ubuntu-as-the-base-for-all-of-my-containers https://blog.kronis.dev/articles/using-ubuntu-as-the-base-fo...
tvaughan 4y ago
This so hard. I like to think of the make targets, e.g. build, test, install, etc. as an API that should be consistent across repos. This really helps with cross team collaboration. The details of how these tasks happen is free to change at will without the need to “distribute” these changes to developers. There’s no disruption to anyone’s flow. Plus, with a little documentation, on boarding new developers is so much more simple.
intelVISA 4y ago
I use Makefiles anywhere I can fit them they're a brief respite from YAML hell. This issue was solved in 1976 -- I appreciate there's a lot of VC money in reinventing the wheel (and coming full circle) but I digress.
- fbdab103 4y ago
  I too use Make everywhere, but what I would give for an improved tool that had better syntax, composability, and simultaneously deployed everywhere. Sadly, it is good enough, so we shall suffer forever.
mikepurvis 4y ago
I’ve run into this with overly complicated Jenkins pipeline files as well. I think the root cause is just that a single entry point pipeline is boring— everyone wants a CI config that sets statuses and posts results and does things in parallel and interacts with plugins, and every one of those steps is something that is at least semi unique to the CI execution environment. I think the method you describe is still absolutely how it should be, but this types of interactions are why there’s gravity in the other direction.
- pydry 4y ago
  >every one of those steps is something that is at least semi unique to the CI execution environment. Apart from triggers and environment set up none of those things have to be unique. I often push complex CI logic in YAML into code where it is more easily debugged and I dont have to scratch my head to figure out how to use conditionals. Sending slack messages should always be in code IMHO.
silverwind 4y ago
This. All my action files are just a `make test` call which installs dependencies on-demand and most importantly, makes the process reproducible locally which is invaluable to debug.
FooBarWidget 4y ago
A Makefile really sucks at displaying outputs/logs of commands, especially when there are lots of commands and when they run concurrently. It also really sucks at communicating what the overall progress is: how many jobs have finished, how many left, how much time has elapsed. Heck make can make all this much better by just prepending each output line with some colored prefix and timestamp. But make hasn't changed in 30 years and likely won't change. People are proud that it "solves" things since 1976. Yes if your requirements never changed since 1976. I'm not holding by breath that it will deliver basic usability-enhancing features that one can reasonably expect nowadays.
- doliveira 4y ago
  You can just invoke make build, make test within the specific CI stages.
  - FooBarWidget 4y ago
    Then you reduce make to a simple small-scale task runner, basically admitting that it's unusable for large numbers of heterogeneous tasks or concurrency.
    - deleted 4y ago
      [deleted]
- MuffinFlavored 4y ago
  https://www.stevefenton.co.uk/blog/2022/06/run-a-bash-script-with-arguments-in-github-actions/ https://www.stevefenton.co.uk/blog/2022/06/run-a-bash-script...
kodisha 4y ago
Can anyone point me out to a GH repo with this or similar setup. Eager to learn, especially if it runs a bit more complex node/next.js build.
- nerdyadventurer 4y ago
  me too, examples please!
nerdyadventurer 4y ago
Any examples, blog posts on how to use Makefiles for CI?
deleted 4y ago
[deleted]
naikrovek 4y ago
forget make, just put it all in a docker container and let the docker container be your CI. that's what the tool linked in the post does, and if you have a docker container you can run it unmodified in just about any CI system.
nickjj 4y ago
This is what I do except I use a shell script instead of a Makefile. A working example of this is at: https://github.com/nickjj/docker-flask-example/blob/912388f31f5cae03499dc82e2139e7c8f4b3168c/.github/workflows/ci.yml#L19-L29 https://github.com/nickjj/docker-flask-example/blob/912388f3... Those ./run ci:XXX commands are in: https://github.com/nickjj/docker-flask-example/blob/912388f31f5cae03499dc82e2139e7c8f4b3168c/run#L144-L179 https://github.com/nickjj/docker-flask-example/blob/912388f3... I like it because if CI ever happens to be down I can still run that shell script locally.
- sneak 4y ago
  You should still have a Makefile that calls your shell scripts when "make" or "make test" is run. Every person who writes shell scripts has a different filename and arguments. "make" and "make test" are always the same everywhere.
solatic 4y ago
> installing $language_compiler Yes, because proper use of the tool cache (and other caches) significantly speed up GitHub Actions builds. Look, I'm all for putting logic in Makefiles instead of in YAML. But not at the expense of slower (and therefore more expensive) builds!
deleted 4y ago
[deleted]
rubicks 4y ago
Strongly concur. I'll concede some cons: * GNU make is sometimes unavailable * syntax is an acquired taste * not everything fits in a rule body Imho, these are far outweighed by the flexibility, portability, and "least surprise" convention embodied in a Makefile.

Gigachad 4y ago

With actions you can run multiple tasks in parallel, restart failed portions without retrying the whole CI, run certain sections with conditions like which branch you are on (different tasks when commit to master vs feature branch)

Arnavion 4y ago
>With actions you can run multiple tasks in parallel, If you mean "I want to run one build with the foo feature and one build with the bar feature. Actions lets me run those in parallel", then that is the "strategy" part of the workflow, not the "tasks" part. My comment was about the latter. ("and make your GH Actions run `make` in a script task.") If you mean "I want to run two steps of a job in parallel and then run the rest of the job after they're complete", then shell is a much simpler DSL for that. Running things in parallel is literally a single `&` character. >restart failed portions without retrying the whole CI, run certain sections with conditions like which branch you are on (different tasks when commit to master vs feature branch) So can a shell script.
- FooBarWidget 4y ago
  > Running things in parallel is literally a single `&` character. Yes now try waiting for all parallel tasks, and error out if at least one errors. And try separating their output so that they don't get interleaved into a big mess. And try by default hiding the outputs of commands that succeeded, only showing those that failed, except when the user explicitly asks for it. Your "simple" shell script now suddenly isn't so simple anymore.
  - stabbles 4y ago
    Using GNU make you can get very nice output sync: all: variant-1 variant-2 @echo done with all variant-%: @echo starting $@ @sleep 1 @echo done $@ Use `make -j2 --output-sync=target`.
    - notpushkin 4y ago
      brew install make gmake -j2 --output-sync=target on macOS.
  - cormacrelf 4y ago
    That’s what GNU Parallel is for. Or Pueue, which gives you a GH-Actions-level feature set but it’s less likely to be installed on any particular machine. Pretty sure you could fetch a pueue binary at the start of an actions script and do everything that way. (These can’t project their tasks over multiple GH Actions runners, eg for multiple OSes. For that you will need to use the YAML. Good compilers will already do work in parallel and max out however many cores they are given, so multi machine is the main use case. Unfortunate.)
  - Arnavion 4y ago
    >Yes now try waiting for all parallel tasks, `wait` >and error out if at least one errors. `wait` propagates the exit status of the waited task. `set -e` triggers the script to exit on any command failure. >And try separating their output so that they don't get interleaved into a big mess. `| while read -r line; do echo "task X: $line"; done` Learn the tools instead of being so confident that it can't be done or that it's complicated.

de_keyboard 4y ago

I agree. GitHub Actions should call your scripts but your scripts should not depend on GitHub Actions API. I also suggest Bazel as a consideration alongside Make. With Bazel, you get two advantages over Make: 1. It is easier to ensure that what GitHub Actions runs and builds is the same as what you have locally, since Bazel can fetch toolchains as part of the build process 2. Using a Bazel remote cache, you do not have to repeat work if the build fails halfway and you need to make some changes before running it again.

toastal 4y ago
Same can be said about Nix and as a bonus you get Nix instead of Starlark
- intelVISA 4y ago
  Nix & Make is the True Engineer(tm)'s Bazel.
  - de_keyboard 4y ago
    Are Nix and Bazel mutually exclusive? https://www.tweag.io/blog/2018-03-15-bazel-nix/ https://www.tweag.io/blog/2018-03-15-bazel-nix/

ilammy 4y ago

This is a lesson that I've learned after going all-out on actions once. Now my makefiles in addition to the usual "make" and "make test" also support "make prerequisites" to prepare a build environment by installing everything necessary and "make ci" to run everything that CI should check. With actual implementation being scripts placed under "scripts/ci". The scripts do provide some goodies when they are run by GitHub Actions – like folding the build logs or installing dependencies differently – but these targets also work on the developer machine.

jaxn 4y ago
What about caching to reduce ci time? GH setup scripts cache dependencies in a way that would seem hard to replicate in a make file.
- makapuf 4y ago
  You can split your make file in two CI steps, one cached, and the other one depending on it.
- ilammy 4y ago
  If it’s manageable – just don’t. Build from scratch. Make sure your build works from scratch and completes in acceptable timeframe. If it’s painful, treat the root cause and not the symptoms. If it’s unbearable due to circumstances out of your control, there’s nothing wrong with adding some actions/cache steps to .github/workflows – this goes around the build: fetch previous cache before, update the cache after if needed. The build is still reproducible outside of GitHub Actions, but a pinch of magic salt makes it go faster sometimes without being an essential part of the build pipeline married to GitHub. If you need to install a whole host of mostly static dependencies, GitHub Actions support running steps in arbitrary Docker container. Prepare an image beforehand, it can be cached too, now you have a predictable environment. (The only downside is that it doesn’t work on macOS and Windows.)
  - KronisLV 4y ago
    > If you need to install a whole host of mostly static dependencies, GitHub Actions support running steps in arbitrary Docker container. Prepare an image beforehand, it can be cached too, now you have a predictable environment. (The only downside is that it doesn’t work on macOS and Windows.) Actually I use a similar workflow for some of the other projects that I have to work with, keeping everything CI agnostic. I incrementally add various types of dependencies to the container images where the application will be built. For example: 1. common base image (e.g. Debian, Ubuntu, or something like Alpine, or maybe RPM based ones) 2. previous + common tools (optional, if you want to include curl or other tools in the container for debugging or testing stuff quickly) 3. previous + project runtime (depending on tech stack, for example OpenJDK for Java projects) 4. previous + development tools (depending on the tech stack, typically for pulling in dependencies, like Maven, or npm, or whatever) 5. previous + project dependencies (if the project is large and the dependencies change rarely, you can install them once here and the changing 5% or so later) 6. previous + project build (including things like running tests, typically multi-stage with the build and tests, and built app handled separately) Compared to the more "common" way to do things, step #5 probably jumps out the most here, I do a pass of installing all of the dependencies, say, every morning, or hourly in the background, so that later when the project is built the CI can just go: "Hmm, it seems like 95% of the things I need here are already present, I'll just pull the remaining packages (if any)." Clean installs only need to be done when packages are removed, which is also reasonably easy to do. Though the benefits of this aren't quite as staggering, if you use a self-hosted package repository like Sonatype Nexus, which can cache any dependencies that you've used previously and make everything faster on the network I/O side. This only doesn't hold true when actually installing the packages takes up the majority of the time (e.g. compiling native code), in which case the above is still very useful. So, an example of how the stages might look, is as follows: Builder: Ubuntu + tools (optional) + OpenJDK + Maven + project dependencies + project build (and run tests) Runner: Ubuntu + tools (optional) + OpenJDK + built project from last image (using COPY with --from, typically .jar file or app directory) Of course, things are less comfortable when you don't have all of your app's dependencies packaged statically but need them "on the system" instead, like Python packages or Ruby Gems, but then your builder and runner will simply look more alike. For my own personal stuff I also use a slightly simplified version of this, about which I wrote on my blog here, the drawbacks included: https://blog.kronis.dev/articles/using-ubuntu-as-the-base-for-all-of-my-containers https://blog.kronis.dev/articles/using-ubuntu-as-the-base-fo...

tvaughan 4y ago

This so hard. I like to think of the make targets, e.g. build, test, install, etc. as an API that should be consistent across repos. This really helps with cross team collaboration. The details of how these tasks happen is free to change at will without the need to “distribute” these changes to developers. There’s no disruption to anyone’s flow. Plus, with a little documentation, on boarding new developers is so much more simple.

intelVISA 4y ago

I use Makefiles anywhere I can fit them they're a brief respite from YAML hell. This issue was solved in 1976 -- I appreciate there's a lot of VC money in reinventing the wheel (and coming full circle) but I digress.

fbdab103 4y ago
I too use Make everywhere, but what I would give for an improved tool that had better syntax, composability, and simultaneously deployed everywhere. Sadly, it is good enough, so we shall suffer forever.

mikepurvis 4y ago

I’ve run into this with overly complicated Jenkins pipeline files as well. I think the root cause is just that a single entry point pipeline is boring— everyone wants a CI config that sets statuses and posts results and does things in parallel and interacts with plugins, and every one of those steps is something that is at least semi unique to the CI execution environment. I think the method you describe is still absolutely how it should be, but this types of interactions are why there’s gravity in the other direction.

pydry 4y ago
>every one of those steps is something that is at least semi unique to the CI execution environment. Apart from triggers and environment set up none of those things have to be unique. I often push complex CI logic in YAML into code where it is more easily debugged and I dont have to scratch my head to figure out how to use conditionals. Sending slack messages should always be in code IMHO.

silverwind 4y ago

This. All my action files are just a `make test` call which installs dependencies on-demand and most importantly, makes the process reproducible locally which is invaluable to debug.

FooBarWidget 4y ago

A Makefile really sucks at displaying outputs/logs of commands, especially when there are lots of commands and when they run concurrently. It also really sucks at communicating what the overall progress is: how many jobs have finished, how many left, how much time has elapsed. Heck make can make all this much better by just prepending each output line with some colored prefix and timestamp. But make hasn't changed in 30 years and likely won't change. People are proud that it "solves" things since 1976. Yes if your requirements never changed since 1976. I'm not holding by breath that it will deliver basic usability-enhancing features that one can reasonably expect nowadays.

doliveira 4y ago
You can just invoke make build, make test within the specific CI stages.
- FooBarWidget 4y ago
  Then you reduce make to a simple small-scale task runner, basically admitting that it's unusable for large numbers of heterogeneous tasks or concurrency.
  - deleted 4y ago
    [deleted]
MuffinFlavored 4y ago
https://www.stevefenton.co.uk/blog/2022/06/run-a-bash-script-with-arguments-in-github-actions/ https://www.stevefenton.co.uk/blog/2022/06/run-a-bash-script...

kodisha 4y ago

Can anyone point me out to a GH repo with this or similar setup. Eager to learn, especially if it runs a bit more complex node/next.js build.

nerdyadventurer 4y ago
me too, examples please!

nerdyadventurer 4y ago

Any examples, blog posts on how to use Makefiles for CI?

deleted 4y ago

[deleted]

naikrovek 4y ago

forget make, just put it all in a docker container and let the docker container be your CI. that's what the tool linked in the post does, and if you have a docker container you can run it unmodified in just about any CI system.

nickjj 4y ago

This is what I do except I use a shell script instead of a Makefile. A working example of this is at: https://github.com/nickjj/docker-flask-example/blob/912388f31f5cae03499dc82e2139e7c8f4b3168c/.github/workflows/ci.yml#L19-L29 https://github.com/nickjj/docker-flask-example/blob/912388f3... Those ./run ci:XXX commands are in: https://github.com/nickjj/docker-flask-example/blob/912388f31f5cae03499dc82e2139e7c8f4b3168c/run#L144-L179 https://github.com/nickjj/docker-flask-example/blob/912388f3... I like it because if CI ever happens to be down I can still run that shell script locally.

sneak 4y ago
You should still have a Makefile that calls your shell scripts when "make" or "make test" is run. Every person who writes shell scripts has a different filename and arguments. "make" and "make test" are always the same everywhere.

solatic 4y ago

> installing $language_compiler Yes, because proper use of the tool cache (and other caches) significantly speed up GitHub Actions builds. Look, I'm all for putting logic in Makefiles instead of in YAML. But not at the expense of slower (and therefore more expensive) builds!

deleted 4y ago

[deleted]

rubicks 4y ago

Strongly concur. I'll concede some cons: * GNU make is sometimes unavailable * syntax is an acquired taste * not everything fits in a rule body Imho, these are far outweighed by the flexibility, portability, and "least surprise" convention embodied in a Makefile.