3 ms·

The reason why many CI configs devolve into such a mess isn't typically that they don't extract complicated logic into scripts, it's about all the interactions

The reason why many CI configs devolve into such a mess isn't typically that they don't extract complicated logic into scripts, it's about all the interactions with the CI system itself. This includes caching, sharing of artifacts, generating reports, configuring permissions, ordering of jobs, deciding when which jobs will run, deciding what to do when jobs fail, etc. All of this can get quite messy in a large enough project.

pydry 2y ago
It never becomes unbearably messy this way though. The reason it gets unbearably messy is because most people google "how to do x in github actions" (e.g. send a slack message) and there is a way and it's almost always worse than scripting it yourself.
- SOLAR_FIELDS 2y ago
  The reason it gets unbearably messy is that GitHub has constructed an ecosystem that encourages developers to write Turing complete imperative behavior into YAML without providing the same language constructs/tooling that a proper adult language provides to encourage code reuse and debugging. Without tooling like this any sufficiently complex system is guaranteed to evolve into a spaghetti mess, because no sane way exists to maintain such a system at scale without proper tooling, which one would need to hand roll themselves against a giant, ever changing mostly undocumented black box proprietary system (GitHub Actions). Someone tried to do this, the project is called “act”. The results are described by the author in the article as “subpar”. The only sane way to use GitHub Actions at scale is to take the subset of its features that you leverage to perform the execution (event triggers, runs-on, etc) and only use those features, and farm out all the rest of the work in something that is actually maintainable eg Buildkit, Bazel, Gradle etc
  - WorldMaker 2y ago
    Which I feel is a recurring lesson in CI in general. CI systems have to be scriptable to get their job done because they can't anticipate every build system. With even terrible scriptability comes Turing completeness, because it is hard to avoid, a natural law. Eventually someone figures out how to make it do wild things. Eventually those wild things becomes someone's requirements in their CI pipeline. Eventually someone is blog posting about how terrible that entire CI system is because of how baroque their pipeline has become and how many crazy scripts it has that are hard to test and harder to fix. It's like the circle of life for CI systems.
  - Tainnor 2y ago
    It's not just Github, Gitlab uses the same mess of YAML-programming with custom extensions. So do, I believe, many other systems. Ironically, Jenkins does this one correctly, they just give you a regular programming language. It's a shame there are so many other pitfalls, though.
    - SOLAR_FIELDS 2y ago
      Jenkins has almost the opposite problem: they give you the freedom to do everything, but point the gun right at your foot in the process of doing so
bluGill 2y ago
You should generate your report with regular scripts. You need ci config to deploy them but that is the only part that should be different.
- evilduck 2y ago
  This doesn’t really work when you start sharding tests.
  - bluGill 2y ago
    It works just fine - you have ci scripts for tests-group-1, test-group-2, and so on. You data collection will need to aggregate data from them all, but that is something most data collection systems have (and at least the ones I know of also allow individuals to upload their local test data thus meaning you can test that upload locally). If you break those test groups up right most developers will know which they should run as a result of their changes (if your tests are not so long that developers wouldn't run them all before pushing then you shouldn't shard anyway, though it may be reasonable to say your CI shards are still longer than what a developer would run locally) I have had many tests which manipulate the global environment (integration tests should do this, though I'm not convinced the distinction between integration and unit tests is valuable) so the ability run the same tests as CI is very helpful in finding and verifying you fixed these.
DanielHB 2y ago
Caching and sharing artifacts is usually the main culprit. My company has been using https://nx.dev/ https://nx.dev/ for that. It works locally as well and CI and it just works. Our NX is pointed to store artifacts in GHA, but our GHA scripts don't do any caching directly, it is all handled by NX. It works so well I would even consider pulling a nodejs environment to run it in non-nodejs projects (although I haven't tried, probably would run into some problems). It is somewhat heavy on configuration, but it just moves the complexity from CI configuration to NX configuration (which is nicer IMO). Our CI pipelines are super fast if you don't hit one of one of our slow compilling parts of the codebase. With NX your local dev environment can pull cached items that were built from previous CI ran-jobs or other devs. We have some native C++ dependencies that are kind of a pain to build locally, our dev machines can pull the built binaries built by other devs (since all devs and CI also share the same cache-artifacts storage). So it makes developing locally a lot easier as well, I don't even remember last time I had to build the native C++ stuff myself since I don't work on it.
- anbotero 2y ago
  Do you know the criteria used to pick the nx.dev? That is, do you pay for their Cloud, or do you do some plumbing yourselves to make it work on GitHub and other things? Looks interesting. We’ve picked tools based on time saved without too much extra knowledge or overhead required, so this may prove promising.
  - DanielHB 2y ago
    To be honest, I wasn't the one who added it and have only occasionally done some small changes to the NX configuration. I don't think we pay for their Cloud, I think all our stored artifacts are stored in GHA caching system and we pull them using our github SSH keys. Although I don't know exactly how that was set up. The fact that someone set it up and I just started using it and it just works is a testament of how good it works. NX is good because it does the caching part of CI in a way that works both locally and on CI. But of course it doesn't really help at all with the other points raised by the OP. One interesting thing about NX as well is that it helps with you managing your own local build chain, like in the example I mentioned above, when I run a project that requires the C++ native dependency, that project gets built automatically (or rather my computer pulls the built binaries from the remote cache). But for all of this to work you need to set up these dependency chains explicitly in your NX configuration, but that is formalizing an actual requirement instead of leaving it implicit (or in Makefiles or in scripts that only run in CI). I do have to say that our NX configuration is quite long though, but I feel that once you start using NX it is just too tempting to split your project up in individual cacheable steps even if said steps are very fast to run and produce no artifacts. Although you don't have to. For example we have separate steps for linting, typescript type-checking, code formatting, unit testing for each unique project in our mono-repo. In practice they could be all the same step because they all get invalidated at the same time (basically on any file change).

pydry 2y ago

It never becomes unbearably messy this way though. The reason it gets unbearably messy is because most people google "how to do x in github actions" (e.g. send a slack message) and there is a way and it's almost always worse than scripting it yourself.

SOLAR_FIELDS 2y ago
The reason it gets unbearably messy is that GitHub has constructed an ecosystem that encourages developers to write Turing complete imperative behavior into YAML without providing the same language constructs/tooling that a proper adult language provides to encourage code reuse and debugging. Without tooling like this any sufficiently complex system is guaranteed to evolve into a spaghetti mess, because no sane way exists to maintain such a system at scale without proper tooling, which one would need to hand roll themselves against a giant, ever changing mostly undocumented black box proprietary system (GitHub Actions). Someone tried to do this, the project is called “act”. The results are described by the author in the article as “subpar”. The only sane way to use GitHub Actions at scale is to take the subset of its features that you leverage to perform the execution (event triggers, runs-on, etc) and only use those features, and farm out all the rest of the work in something that is actually maintainable eg Buildkit, Bazel, Gradle etc
- WorldMaker 2y ago
  Which I feel is a recurring lesson in CI in general. CI systems have to be scriptable to get their job done because they can't anticipate every build system. With even terrible scriptability comes Turing completeness, because it is hard to avoid, a natural law. Eventually someone figures out how to make it do wild things. Eventually those wild things becomes someone's requirements in their CI pipeline. Eventually someone is blog posting about how terrible that entire CI system is because of how baroque their pipeline has become and how many crazy scripts it has that are hard to test and harder to fix. It's like the circle of life for CI systems.
- Tainnor 2y ago
  It's not just Github, Gitlab uses the same mess of YAML-programming with custom extensions. So do, I believe, many other systems. Ironically, Jenkins does this one correctly, they just give you a regular programming language. It's a shame there are so many other pitfalls, though.
  - SOLAR_FIELDS 2y ago
    Jenkins has almost the opposite problem: they give you the freedom to do everything, but point the gun right at your foot in the process of doing so

bluGill 2y ago

You should generate your report with regular scripts. You need ci config to deploy them but that is the only part that should be different.

evilduck 2y ago
This doesn’t really work when you start sharding tests.
- bluGill 2y ago
  It works just fine - you have ci scripts for tests-group-1, test-group-2, and so on. You data collection will need to aggregate data from them all, but that is something most data collection systems have (and at least the ones I know of also allow individuals to upload their local test data thus meaning you can test that upload locally). If you break those test groups up right most developers will know which they should run as a result of their changes (if your tests are not so long that developers wouldn't run them all before pushing then you shouldn't shard anyway, though it may be reasonable to say your CI shards are still longer than what a developer would run locally) I have had many tests which manipulate the global environment (integration tests should do this, though I'm not convinced the distinction between integration and unit tests is valuable) so the ability run the same tests as CI is very helpful in finding and verifying you fixed these.

DanielHB 2y ago

Caching and sharing artifacts is usually the main culprit. My company has been using https://nx.dev/ https://nx.dev/ for that. It works locally as well and CI and it just works. Our NX is pointed to store artifacts in GHA, but our GHA scripts don't do any caching directly, it is all handled by NX. It works so well I would even consider pulling a nodejs environment to run it in non-nodejs projects (although I haven't tried, probably would run into some problems). It is somewhat heavy on configuration, but it just moves the complexity from CI configuration to NX configuration (which is nicer IMO). Our CI pipelines are super fast if you don't hit one of one of our slow compilling parts of the codebase. With NX your local dev environment can pull cached items that were built from previous CI ran-jobs or other devs. We have some native C++ dependencies that are kind of a pain to build locally, our dev machines can pull the built binaries built by other devs (since all devs and CI also share the same cache-artifacts storage). So it makes developing locally a lot easier as well, I don't even remember last time I had to build the native C++ stuff myself since I don't work on it.

anbotero 2y ago
Do you know the criteria used to pick the nx.dev? That is, do you pay for their Cloud, or do you do some plumbing yourselves to make it work on GitHub and other things? Looks interesting. We’ve picked tools based on time saved without too much extra knowledge or overhead required, so this may prove promising.
- DanielHB 2y ago
  To be honest, I wasn't the one who added it and have only occasionally done some small changes to the NX configuration. I don't think we pay for their Cloud, I think all our stored artifacts are stored in GHA caching system and we pull them using our github SSH keys. Although I don't know exactly how that was set up. The fact that someone set it up and I just started using it and it just works is a testament of how good it works. NX is good because it does the caching part of CI in a way that works both locally and on CI. But of course it doesn't really help at all with the other points raised by the OP. One interesting thing about NX as well is that it helps with you managing your own local build chain, like in the example I mentioned above, when I run a project that requires the C++ native dependency, that project gets built automatically (or rather my computer pulls the built binaries from the remote cache). But for all of this to work you need to set up these dependency chains explicitly in your NX configuration, but that is formalizing an actual requirement instead of leaving it implicit (or in Makefiles or in scripts that only run in CI). I do have to say that our NX configuration is quite long though, but I feel that once you start using NX it is just too tempting to split your project up in individual cacheable steps even if said steps are very fast to run and produce no artifacts. Although you don't have to. For example we have separate steps for linting, typescript type-checking, code formatting, unit testing for each unique project in our mono-repo. In practice they could be all the same step because they all get invalidated at the same time (basically on any file change).