4 ms·

Shameless self-plug: I very much live by the "fix all warnings before you commit" (or at least before you merge), so I have Shellcheck and a bunch of other lin

Shameless self-plug: I very much live by the "fix all warnings before you commit" (or at least before you merge), so I have Shellcheck and a bunch of other linters set up in my pre-commit configurations. But the majority of the shell in most of my projects ends up embedded in .gitlab-ci.yml files, where it's hard to check. So I made a wrapper that does that automatically: https://pypi.org/project/glscpc/ https://pypi.org/project/glscpc/. It uses the Shellcheck project and some magic to give you the Shellcheck remarks with (mostly) accurate line-numbers.

brirec 3y ago
I’d love to see a project that would do this, but more generally. I don’t use GitLab CI, but I do use a good handful of other file types that essentially inline shell scripts. Dockerfiles, GitHub Actions, and Justfiles, just to name a couple. Usually, and almost exclusively for the sake of ShellCheck, I make a point of putting anything more complex than a couple of commands into their own shell scripts that I call from the inlined script in my Dockerfile. (This pattern also helps me keep my CI from being too locked into GitHub Actions)
- stouset 3y ago
  This is the way
- iamjackg 3y ago
  Absolutely agree. The main downside of that pattern is that it doesn't work with jobs included from other projects in GitLab CI, since the job runs in the context of the project that imported it and therefore can't find the script in its original repo. Huge bummer.
  - vinnymac 3y ago
    Bundle up configuration and scripts, and now we have containerized CI infrastructure.
- vinnymac 3y ago
  I’d rather see CI services add a mode that would enforce all scripts must live in separate files, rather than inline. It’s really not necessary to support inline except for single lines that are very short (under 30 chars).
  - Piraty 3y ago
    so much this. it will help CI not become a holy, super-hard-to-debug, unreproducible mammoth. write scripts, call them from CI
  - Dries007 3y ago
    I would say: stick to straight command sequences with variables, avoid if/for/while/functions etc. Subshells and pipes only for trivial things. Setting something like a 30 char limit will inevitably lead to code-golfing to fit a mess into the 30 char limit. But that becomes harder to automatically check, that's why you should still have good peer reviews based on written standards somewhere.
  - plorkyeran 3y ago
    I am not generally a fan of Xcode Cloud's design, but this is one thing I think it gets very right. Rather than let you specify the build actions in the CI settings, it invokes scripts with fixed names in the `ci_scripts` directory of your repo if they exist. There just isn't a mechanism for creating jobs which aren't something that you can build locally and test independently of Xcode Cloud.
- verdverm 3y ago
  You might like Dagger, building images with code, uses the same buildkit engine under the hood No more linear Dockerfile, use the powers of your preferred language
- marcosnils 3y ago
  Hi there, Dagger contributor here. We're solving this exact same problem by allowing you to encode your CI/CD pipelines in your favorite programming language and run them the same way locally and/or any CI provider (Gitlab, Github Actions, Jenkins, etc). We're very active in our Discord server if you have any questions! https://discord.gg/invite/dagger-io https://discord.gg/invite/dagger-io
  - Dries007 3y ago
    We've been experimenting with Dagger here, as part of the alternative to writing glscpc actually, but I'm not convinced it's ready to replace Gitlab CI. Dagger has one very big downside IMO: It does not have native integration with Gitlab, so you end up having to use Docker-in-Docker and just running dagger as a job in your pipeline. This causes a number of issues: It clumps all your previously separated steps into a single step in the Gitlab pipeline. That doesn't matter too much for console output (although it does when your different steps should run on different runners), but is very annoying if you use Gitlab CI's built in parsing of junit/coverage/... files, since you now have extra layers of context to dig trough when tests fail etc. Plus not all of these allow for multiple input files, so now you have to add extra merging steps. If your job already uses Docker-in-Docker for something, you have to be careful not to end up with Docker-in-Docker-in-Docker situations, or container name conflicts if you just pass through the DOCKER_HOST variable. The one thing that would make this worth it is being able to run the pipelines locally to debug, but I've just written quick-and-dirty scripts to do that every time I've needed it. For example, running the test job in our pipeline on every Python version: https://gitlab.com/Qteal/oss/gitlabci-shellcheck-precommit/-/blob/main/versiontest.sh?ref_type=heads https://gitlab.com/Qteal/oss/gitlabci-shellcheck-precommit/-...
    - marcosnils 3y ago
      > but I'm not convinced it's ready to replace Gitlab CI. The purpose of Dagger it's not to replace your entire CI (Gitlab in your case). As you can see from our website (https://dagger.io/engine https://dagger.io/engine), it works and integrates with all the current CI providers. Where Dagger really shines is to help you and your teams move all the artisanal scripts encoded in YAML into actual code and run them in containers through a fluent SDK which can be written in your language of choice. This unlocks a lot of benefits which are detailed in our docs (https://docs.dagger.io/ https://docs.dagger.io/). > Dagger has one very big downside IMO: It does not have native integration with Gitlab, so you end up having to use Docker-in-Docker and just running dagger as a job in your pipeline. Dagger doesn't depend on Docker. We're just conveniently using Docker (and other container runtimes) as it's generally available pretty much everywhere by default as a way to bootstrap the Dagger Engine. You can read more about the Dagger architecture here: https://github.com/dagger/dagger/blob/main/core/docs/d7yxc-operator_manual.md https://github.com/dagger/dagger/blob/main/core/docs/d7yxc-o... As you can see from our docs (https://docs.dagger.io/759201/gitlab-google-cloud/#step-5-create-a-gitlab-cicd-pipeline https://docs.dagger.io/759201/gitlab-google-cloud/#step-5-cr...), we're leveraging the *default* Gitlab CI `docker` service to bootstrap the engine. There's no `docker-in-docker` happening there. > It clumps all your previously separated steps into a single step in the Gitlab pipeline. It's not generally how we recommend to start, we should definitely improve our docs to reflect that. You can organize your dagger pipelines in multiple functions and call them in separate Gitlab jobs as you're currently doing. For example, you can do the following: ```.gitlab-ci.yml build: script: # no funky untestable shellscripts here, but calls to \*real\* code - dagger run go run ci/main.go build test: script: # no funky untestable shellscripts here, but calls to \*real\* code - dagger run go run ci/main.go test ``` This way, if your pipeline currently has a `build` and `test` job, you still keep using the same structure. > but is very annoying if you use Gitlab CI's built in parsing of junit/coverage/... files, since you now have extra layers of context to dig trough when tests fail etc You can also still keep using these. The only thing you need to be aware is to export the required test / coverage output files from your Dagger pipelines so Gitlab can use them to do what it needs. > but I've just written quick-and-dirty scripts to do that every time I've needed it. This is what we're trying to improve. Those quick-and-dirty generally start very simple but they become brittle and very difficult to test by other engineers. Yes, you could use docker or any container-like thing to enable portability, but you'll probably have to write more scripts to glue all that together. Quoting one of our founders: "Our mission is to help your teams to keep the CI configuration as light and "dumb" as possible, by moving as much logic as possible into portable scripts. This minimizes "push and pray", where any pipeline change requires committing, pushing, and waiting for the proprietary CI black box to give you a green or red light. Ideally those scripts use containers for maximum reproduceability. Our goal at Dagger is to help democratize this way of creating pipelines, and making it a standard, so that an actual software ecosystem can appear, where devops engineers can actually reuse each other's code, the same way application developers can."
- Dries007 3y ago
  Generalizing this is non-trivial (I tried initially) but I'm sure others can build in the same principles. I think this comes close for Dockerfiles: https://hadolint.github.io/hadolint/ https://hadolint.github.io/hadolint/ Just have to write a pre-commit hook for it.
iamjackg 3y ago
Oh, that's really cool. I was trying to solve this from a different perspective a while ago: I wanted to add some pre-processing that would take a "normal" shell script and render it to the `script` part of the corresponding job at build time, the advantage being that you still have everything self-contained in the gitlab-ci job. I stopped working on it because dealing with shell shenanigans in the GitLab CI runner environment is such a miserable experience that we're in the process of moving all our jobs to python scripts.
- Dries007 3y ago
  Yea, Pythonifying the scripts is also generally my preferences the moment they become somewhat complex. But even then it's nice that you can be reasonably sure you're not forgetting quotes around variables or using bash constructs where you only have sh.
deleted 3y ago
[deleted]
l0b0 3y ago
High five - I did something similar very recently[1]; usage[2]: - repo: https://gitlab.com/engmark/shellcheck-gitlab-ci-scripts-hook rev: 174fda8f384db229aca5b40380e713d0c25a1cb9 # frozen: v1 hooks: - id: shellcheck-gitlab-ci-scripts files: ^\.gitlab-ci\.yml$ [1] https://gitlab.com/engmark/shellcheck-gitlab-ci-scripts-hook https://gitlab.com/engmark/shellcheck-gitlab-ci-scripts-hook [2] https://gitlab.com/engmark/root/-/blob/9f7d9b93c2297d0b170e5dd03d756eeedf801fc0/.pre-commit-config.yaml#L104-108 https://gitlab.com/engmark/root/-/blob/9f7d9b93c2297d0b170e5...

brirec 3y ago

I’d love to see a project that would do this, but more generally. I don’t use GitLab CI, but I do use a good handful of other file types that essentially inline shell scripts. Dockerfiles, GitHub Actions, and Justfiles, just to name a couple. Usually, and almost exclusively for the sake of ShellCheck, I make a point of putting anything more complex than a couple of commands into their own shell scripts that I call from the inlined script in my Dockerfile. (This pattern also helps me keep my CI from being too locked into GitHub Actions)

stouset 3y ago
This is the way
iamjackg 3y ago
Absolutely agree. The main downside of that pattern is that it doesn't work with jobs included from other projects in GitLab CI, since the job runs in the context of the project that imported it and therefore can't find the script in its original repo. Huge bummer.
- vinnymac 3y ago
  Bundle up configuration and scripts, and now we have containerized CI infrastructure.
vinnymac 3y ago
I’d rather see CI services add a mode that would enforce all scripts must live in separate files, rather than inline. It’s really not necessary to support inline except for single lines that are very short (under 30 chars).
- Piraty 3y ago
  so much this. it will help CI not become a holy, super-hard-to-debug, unreproducible mammoth. write scripts, call them from CI
- Dries007 3y ago
  I would say: stick to straight command sequences with variables, avoid if/for/while/functions etc. Subshells and pipes only for trivial things. Setting something like a 30 char limit will inevitably lead to code-golfing to fit a mess into the 30 char limit. But that becomes harder to automatically check, that's why you should still have good peer reviews based on written standards somewhere.
- plorkyeran 3y ago
  I am not generally a fan of Xcode Cloud's design, but this is one thing I think it gets very right. Rather than let you specify the build actions in the CI settings, it invokes scripts with fixed names in the `ci_scripts` directory of your repo if they exist. There just isn't a mechanism for creating jobs which aren't something that you can build locally and test independently of Xcode Cloud.
verdverm 3y ago
You might like Dagger, building images with code, uses the same buildkit engine under the hood No more linear Dockerfile, use the powers of your preferred language
marcosnils 3y ago
Hi there, Dagger contributor here. We're solving this exact same problem by allowing you to encode your CI/CD pipelines in your favorite programming language and run them the same way locally and/or any CI provider (Gitlab, Github Actions, Jenkins, etc). We're very active in our Discord server if you have any questions! https://discord.gg/invite/dagger-io https://discord.gg/invite/dagger-io
- Dries007 3y ago
  We've been experimenting with Dagger here, as part of the alternative to writing glscpc actually, but I'm not convinced it's ready to replace Gitlab CI. Dagger has one very big downside IMO: It does not have native integration with Gitlab, so you end up having to use Docker-in-Docker and just running dagger as a job in your pipeline. This causes a number of issues: It clumps all your previously separated steps into a single step in the Gitlab pipeline. That doesn't matter too much for console output (although it does when your different steps should run on different runners), but is very annoying if you use Gitlab CI's built in parsing of junit/coverage/... files, since you now have extra layers of context to dig trough when tests fail etc. Plus not all of these allow for multiple input files, so now you have to add extra merging steps. If your job already uses Docker-in-Docker for something, you have to be careful not to end up with Docker-in-Docker-in-Docker situations, or container name conflicts if you just pass through the DOCKER_HOST variable. The one thing that would make this worth it is being able to run the pipelines locally to debug, but I've just written quick-and-dirty scripts to do that every time I've needed it. For example, running the test job in our pipeline on every Python version: https://gitlab.com/Qteal/oss/gitlabci-shellcheck-precommit/-/blob/main/versiontest.sh?ref_type=heads https://gitlab.com/Qteal/oss/gitlabci-shellcheck-precommit/-...
  - marcosnils 3y ago
    > but I'm not convinced it's ready to replace Gitlab CI. The purpose of Dagger it's not to replace your entire CI (Gitlab in your case). As you can see from our website (https://dagger.io/engine https://dagger.io/engine), it works and integrates with all the current CI providers. Where Dagger really shines is to help you and your teams move all the artisanal scripts encoded in YAML into actual code and run them in containers through a fluent SDK which can be written in your language of choice. This unlocks a lot of benefits which are detailed in our docs (https://docs.dagger.io/ https://docs.dagger.io/). > Dagger has one very big downside IMO: It does not have native integration with Gitlab, so you end up having to use Docker-in-Docker and just running dagger as a job in your pipeline. Dagger doesn't depend on Docker. We're just conveniently using Docker (and other container runtimes) as it's generally available pretty much everywhere by default as a way to bootstrap the Dagger Engine. You can read more about the Dagger architecture here: https://github.com/dagger/dagger/blob/main/core/docs/d7yxc-operator_manual.md https://github.com/dagger/dagger/blob/main/core/docs/d7yxc-o... As you can see from our docs (https://docs.dagger.io/759201/gitlab-google-cloud/#step-5-create-a-gitlab-cicd-pipeline https://docs.dagger.io/759201/gitlab-google-cloud/#step-5-cr...), we're leveraging the *default* Gitlab CI `docker` service to bootstrap the engine. There's no `docker-in-docker` happening there. > It clumps all your previously separated steps into a single step in the Gitlab pipeline. It's not generally how we recommend to start, we should definitely improve our docs to reflect that. You can organize your dagger pipelines in multiple functions and call them in separate Gitlab jobs as you're currently doing. For example, you can do the following: ```.gitlab-ci.yml build: script: # no funky untestable shellscripts here, but calls to \*real\* code - dagger run go run ci/main.go build test: script: # no funky untestable shellscripts here, but calls to \*real\* code - dagger run go run ci/main.go test ``` This way, if your pipeline currently has a `build` and `test` job, you still keep using the same structure. > but is very annoying if you use Gitlab CI's built in parsing of junit/coverage/... files, since you now have extra layers of context to dig trough when tests fail etc You can also still keep using these. The only thing you need to be aware is to export the required test / coverage output files from your Dagger pipelines so Gitlab can use them to do what it needs. > but I've just written quick-and-dirty scripts to do that every time I've needed it. This is what we're trying to improve. Those quick-and-dirty generally start very simple but they become brittle and very difficult to test by other engineers. Yes, you could use docker or any container-like thing to enable portability, but you'll probably have to write more scripts to glue all that together. Quoting one of our founders: "Our mission is to help your teams to keep the CI configuration as light and "dumb" as possible, by moving as much logic as possible into portable scripts. This minimizes "push and pray", where any pipeline change requires committing, pushing, and waiting for the proprietary CI black box to give you a green or red light. Ideally those scripts use containers for maximum reproduceability. Our goal at Dagger is to help democratize this way of creating pipelines, and making it a standard, so that an actual software ecosystem can appear, where devops engineers can actually reuse each other's code, the same way application developers can."
Dries007 3y ago
Generalizing this is non-trivial (I tried initially) but I'm sure others can build in the same principles. I think this comes close for Dockerfiles: https://hadolint.github.io/hadolint/ https://hadolint.github.io/hadolint/ Just have to write a pre-commit hook for it.

iamjackg 3y ago

Oh, that's really cool. I was trying to solve this from a different perspective a while ago: I wanted to add some pre-processing that would take a "normal" shell script and render it to the `script` part of the corresponding job at build time, the advantage being that you still have everything self-contained in the gitlab-ci job. I stopped working on it because dealing with shell shenanigans in the GitLab CI runner environment is such a miserable experience that we're in the process of moving all our jobs to python scripts.

Dries007 3y ago
Yea, Pythonifying the scripts is also generally my preferences the moment they become somewhat complex. But even then it's nice that you can be reasonably sure you're not forgetting quotes around variables or using bash constructs where you only have sh.

deleted 3y ago

[deleted]

l0b0 3y ago

High five - I did something similar very recently[1]; usage[2]: - repo: https://gitlab.com/engmark/shellcheck-gitlab-ci-scripts-hook rev: 174fda8f384db229aca5b40380e713d0c25a1cb9 # frozen: v1 hooks: - id: shellcheck-gitlab-ci-scripts files: ^\.gitlab-ci\.yml$ [1] https://gitlab.com/engmark/shellcheck-gitlab-ci-scripts-hook https://gitlab.com/engmark/shellcheck-gitlab-ci-scripts-hook [2] https://gitlab.com/engmark/root/-/blob/9f7d9b93c2297d0b170e5dd03d756eeedf801fc0/.pre-commit-config.yaml#L104-108 https://gitlab.com/engmark/root/-/blob/9f7d9b93c2297d0b170e5...