4 ms·

With GitHub actions you can quite easily specify a workflow for parts of your repo (simple file path filter https://docs.github.com/en/actions/reference/workflo

by csnweb 5y ago

With GitHub actions you can quite easily specify a workflow for parts of your repo (simple file path filter https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#onpushpull_requestpaths https://docs.github.com/en/actions/reference/workflow-syntax...). So you can basically just write one workflow for each project in the monorepo and have only those running where changes occured.

throwaway894345 5y ago
That’s what I’m doing now, but you still have to push artifacts up to a repository and downstream jobs have to pull them down. Further still, managing the boilerplate for GitHub actions is no small task (e.g., you have a dozen Python projects and you want the actions for each to look similar), so to solve for that you have to build some hack that involves generating the Actions YAML before you commit or similar. Not the end of the world, but still far from my ideal state.
- CameronNemo 5y ago
  >have to build some hack that involves generating the Actions YAML before you commit or similar You could probably use jsonnet expressions and invoke them from a pre-commit hook. That seems fairly elegant to me.
  - throwaway894345 5y ago
    I’m not doing json net, but I am generating them effectively from a pre-commit hook. One of the jobs that I generate validates that the actions that were committed are up-to-date. This works well because I don’t have too many dependencies, but if you have deep dependency trees then the builds will end up taking a long time and you end up reinventing Nix or Bazel poorly.
    - CameronNemo 5y ago
      Are nix and Bazel in the same category? I have used nix very little and never used Bazel. But I have used Make and jsonnet pretty often. Can you re-run Make in a pipeline to verify that the actions in use match what would be generated based on the repo state?
      - throwaway894345 5y ago
        > Are nix and Bazel in the same category? Nix bills itself as a fully reproducible package manager and Bazel as a fully reproducible build tool. In the abstract, both of these reproducibly build software, so that puts them in the same category in general. > Can you re-run Make in a pipeline to verify that the actions in use match what would be generated based on the repo state? Yes, and I’m doing something similar although not with Make. The problem is that this home-grown build system doesn’t do incremental builds, so everything will be fully built from source every time, and that can take a while for very deep dependency graphs. Conceivably your “Actions generator” script could generate only the jobs that need to be run based on what changed since the previous commit, referencing artifacts from the previous commit for those build steps which weren’t invalidated. This is a neat concept, but we’re well on our way to reinventing Nix or Bazel.

by csnweb 5y ago

throwaway894345 5y ago

That’s what I’m doing now, but you still have to push artifacts up to a repository and downstream jobs have to pull them down. Further still, managing the boilerplate for GitHub actions is no small task (e.g., you have a dozen Python projects and you want the actions for each to look similar), so to solve for that you have to build some hack that involves generating the Actions YAML before you commit or similar. Not the end of the world, but still far from my ideal state.

CameronNemo 5y ago
>have to build some hack that involves generating the Actions YAML before you commit or similar You could probably use jsonnet expressions and invoke them from a pre-commit hook. That seems fairly elegant to me.
- throwaway894345 5y ago
  I’m not doing json net, but I am generating them effectively from a pre-commit hook. One of the jobs that I generate validates that the actions that were committed are up-to-date. This works well because I don’t have too many dependencies, but if you have deep dependency trees then the builds will end up taking a long time and you end up reinventing Nix or Bazel poorly.
  - CameronNemo 5y ago
    Are nix and Bazel in the same category? I have used nix very little and never used Bazel. But I have used Make and jsonnet pretty often. Can you re-run Make in a pipeline to verify that the actions in use match what would be generated based on the repo state?
    - throwaway894345 5y ago
      > Are nix and Bazel in the same category? Nix bills itself as a fully reproducible package manager and Bazel as a fully reproducible build tool. In the abstract, both of these reproducibly build software, so that puts them in the same category in general. > Can you re-run Make in a pipeline to verify that the actions in use match what would be generated based on the repo state? Yes, and I’m doing something similar although not with Make. The problem is that this home-grown build system doesn’t do incremental builds, so everything will be fully built from source every time, and that can take a while for very deep dependency graphs. Conceivably your “Actions generator” script could generate only the jobs that need to be run based on what changed since the previous commit, referencing artifacts from the previous commit for those build steps which weren’t invalidated. This is a neat concept, but we’re well on our way to reinventing Nix or Bazel.