4 ms·

I'm in the apparently small group of people who ignores 90% of the made & shared features, and push back against making internal ops to depend too much to any p

by j1elo 3y ago

I'm in the apparently small group of people who ignores 90% of the made & shared features, and push back against making internal ops to depend too much to any particular CI system. We know how to build, package, and release our software, and at work I firmly support that we write those operations as standalone scripts that in principle could even run in the developer's machine itself. Moving them to CI is just changing the machine that will run the scripts, and that's where I draw the line of how much coupling to accept with any given runner: run scripts, inject secrets, and maybe accumulate artifacts between jobs. There are a myriad of things done with Actions, such as preparing NPM, or running Docker containers. I see no point in them, you should be able to manually run the same commands, one by one, locally on a VM or empty Docker container, so at that point why not write them as a script. Use CI as a dumb shell runner. Getting to depend on all those small Actions saves 5 minutes today, only to make migrations immensely painful tomorrow.

wombatpm 3y ago
Especially if you have to run and deploy on your own servers. Not everything can be in the cloud.
gonzo41 3y ago
I agree with this, I recently had a pretty long network issue with our CI system and for the most part we just reran scripts or converted ci yml into scripts. I like CI for secrets management, scrpted pipelines and deploying. And having one button do do all of that. But all the extra junk people do with it seems like another thing to have break.
hardwaresofton 3y ago
I definitely agree with and do this — it only falls apart when CI environment stuff interferes with file locations and other external state. That said it is nice (if the scripts are simple) to do things like machine set up very easily.
physicsguy 3y ago
Same. I've also had to do things like migrate CI when a company was acquired, from Travis to Circle and Circle to GitLab in the past. It's very painful to do this if you've leaned into everything the CI service offers.
DrScientist 3y ago
Indeed - a whole generation seem to be doomed to learn the hard way the painful lessons of lock-in.
seadan83 3y ago
Agree, I like to have a top level "verify" script that does all checks and have CI run exclusively and minimally that one script. Ideally the verify script is just a wrapper around a build and test command
growse 3y ago
This is fine if you treat your CI provider as a "dumb shell runner". But good CI platforms have actually useful features and APIs (e.g. caching) and if you want to use them, a simple Makefile isn't going to work. For projects where the difference between a cold and warm cache build is tens of minutes, those features have meaningful quality of life improvements. This may be a tradeoff you're ok with, but for a lot of people, it's not.
- j1elo 3y ago
  C++ with templating can take a while to compile, so for caching my build system would include mounting a Docker volume with contents of ~/.ccache/. This was well integrated with the rest of the scripts, so it would work equally fine if I ran them in my laptop or in the CI runner. My point is that I really don't believe that CI systems provide anything so unique that it couldn't be also provided by local software in a developer's laptop. If the question of "how do I build this in my laptop" is "you cannot, must use CI because we truly require some of its features", I'd consider that an ops failure. However, I fully admit my experience comes from small to middle size projects. I cannot talk about big or huge scale projects. Maybe that's where needs grow so complex that CI features become truly necessary.
  - growse 3y ago
    > My point is that I really don't believe that CI systems provide anything so unique that it couldn't be also provided by local software in a developer's laptop. If the question of "how do I build this in my laptop" is "you cannot, must use CI because we truly require some of its features", I'd consider that an ops failure. So I do agree with this. But I think there's nuance here. The job of producing a build artifact involves steps that broadly break down into two categories: setting up the context, and doing the build. I totally agree that the "doing the build" bit should be a series of simple steps that are agnostic of their environment. A shell script, or a Makefile - something you can just invoke anywhere. But the context bit is also super-important. On my laptop, I've already got the (e.g.) right JDK installed so that when I run `make`, the build succeeds. But I'm also not wiping my laptop before every build. On a CI platform, you're effectively starting from scratch every time, so whilst you can go and write your own code to set up the caching, download the tooling etc. etc. there's an enormous amount of value to be gained by re-using the CI-platform's software and features to do that as easily as possible. No-one wants to be writing code that works out how to go set up the right JDK in the right place, when `actions/setup-java` will basically do all that for you. In theory, yes, you could go and curate a container image that has everything you need and just run your build inside that, but now you've got two bits of software to manage. If you can't run the build locally, then yes, you're in a pickle.

by j1elo 3y ago

wombatpm 3y ago

Especially if you have to run and deploy on your own servers. Not everything can be in the cloud.

gonzo41 3y ago

I agree with this, I recently had a pretty long network issue with our CI system and for the most part we just reran scripts or converted ci yml into scripts. I like CI for secrets management, scrpted pipelines and deploying. And having one button do do all of that. But all the extra junk people do with it seems like another thing to have break.

hardwaresofton 3y ago

I definitely agree with and do this — it only falls apart when CI environment stuff interferes with file locations and other external state. That said it is nice (if the scripts are simple) to do things like machine set up very easily.

physicsguy 3y ago

Same. I've also had to do things like migrate CI when a company was acquired, from Travis to Circle and Circle to GitLab in the past. It's very painful to do this if you've leaned into everything the CI service offers.

DrScientist 3y ago

Indeed - a whole generation seem to be doomed to learn the hard way the painful lessons of lock-in.

seadan83 3y ago

Agree, I like to have a top level "verify" script that does all checks and have CI run exclusively and minimally that one script. Ideally the verify script is just a wrapper around a build and test command

growse 3y ago

This is fine if you treat your CI provider as a "dumb shell runner". But good CI platforms have actually useful features and APIs (e.g. caching) and if you want to use them, a simple Makefile isn't going to work. For projects where the difference between a cold and warm cache build is tens of minutes, those features have meaningful quality of life improvements. This may be a tradeoff you're ok with, but for a lot of people, it's not.

j1elo 3y ago
C++ with templating can take a while to compile, so for caching my build system would include mounting a Docker volume with contents of ~/.ccache/. This was well integrated with the rest of the scripts, so it would work equally fine if I ran them in my laptop or in the CI runner. My point is that I really don't believe that CI systems provide anything so unique that it couldn't be also provided by local software in a developer's laptop. If the question of "how do I build this in my laptop" is "you cannot, must use CI because we truly require some of its features", I'd consider that an ops failure. However, I fully admit my experience comes from small to middle size projects. I cannot talk about big or huge scale projects. Maybe that's where needs grow so complex that CI features become truly necessary.
- growse 3y ago
  > My point is that I really don't believe that CI systems provide anything so unique that it couldn't be also provided by local software in a developer's laptop. If the question of "how do I build this in my laptop" is "you cannot, must use CI because we truly require some of its features", I'd consider that an ops failure. So I do agree with this. But I think there's nuance here. The job of producing a build artifact involves steps that broadly break down into two categories: setting up the context, and doing the build. I totally agree that the "doing the build" bit should be a series of simple steps that are agnostic of their environment. A shell script, or a Makefile - something you can just invoke anywhere. But the context bit is also super-important. On my laptop, I've already got the (e.g.) right JDK installed so that when I run `make`, the build succeeds. But I'm also not wiping my laptop before every build. On a CI platform, you're effectively starting from scratch every time, so whilst you can go and write your own code to set up the caching, download the tooling etc. etc. there's an enormous amount of value to be gained by re-using the CI-platform's software and features to do that as easily as possible. No-one wants to be writing code that works out how to go set up the right JDK in the right place, when `actions/setup-java` will basically do all that for you. In theory, yes, you could go and curate a container image that has everything you need and just run your build inside that, but now you've got two bits of software to manage. If you can't run the build locally, then yes, you're in a pickle.