3 ms·
Builds should just be done with shell scripts. I recently crossed swords with .gitlab-ci.yml. An obfuscation layer that fights back. Ditto jenkinsfile and wha
by specialist 8y ago
Builds should just be done with shell scripts.
I recently crossed swords with .gitlab-ci.yml. An obfuscation layer that fights back.
Ditto jenkinsfile and whatever you wanna call GoCD's config.xml.
This silliness reminds me most of all those ETL workflow obfuscation frameworks, like BizTalk and TalenD.
PS- If anyone has the bad judgement to use YAML, it'd be nice if the linter reported something more specific than "syntax error line 1 char 1" for deeply buried typos. I thought XML sucked until I met JSON. I thought JSON sucked until I met YAML. What's next? The resurrection of ASN.1?
PS2- It's not programming if you can't set a break point. The edit, eval, fail, profanity loop for troubleshooting .gitlab-ci.yml is pretty intense.
- drdaeman 8y ago> Builds should just be done with shell scripts You still need something to describe the order and dependencies of those shell scripts as well as when to run them (branches, tags) and how to handle the artifacts (expiry time, etc). Which should be, much preferably, something not Turing-complete, so you can make an UI for this. Draw the graph, etc etc. Take those parts out, and almost all what's left in .gitlab-ci.yml is just the `script` key. You can debug most things locally (without having to do that ugly commit-try-fail-repeat dance) with https://docs.gitlab.com/runner/commands/#gitlab-runner-exec https://docs.gitlab.com/runner/commands/#gitlab-runner-exec
- specialist 8y agoTuring complete? What? If your visual programming fuzzy whatzzit drops down to metal (shell commands in this case), just use the metal. I did get the git-runner to run locally. That was fun. Why do I have to commit my changes to my local branch so the git-runner can see them? MY CODE IS RIGHT THERE! I reiterate: Just use shell scripts.
- drdaeman 8y ago> Turing complete? What? You'll need to run the CI pipeline to figure out the execution graph. In a restricted language, you don't have to - it can just describe that graph. > If your visual programming fuzzy whatzzit drops down to metal I don't get you. Everything programming eventually drops down to metal one way or another. It is not about visual programming, it's about the appropriate languages. Shell scripts are not exactly appropriate to describe a graph. Especially a graph meant to be ran on a distributed system - so, really, *sh may be not the best fit here. Maybe you have some better idea, but I'm just afraid you'll end up with some shell-based DSL that would be no better than YAML, but suffer from all possible issues (for example, there's no concept of pure functions in shell scripting, so every CI run plan can take completely different shape) Oh. If you're a person who likes all their configuration files being programs - then I won't argue as it'll be a matter of taste. > Why do I have to commit my changes to my local branch so the git-runner can see them? Yeah, I'm also disappointed in this. It was dismissed as "wontfix" a while ago https://gitlab.com/gitlab-org/gitlab-runner/issues/1545 https://gitlab.com/gitlab-org/gitlab-runner/issues/1545 and seems that it stays this way even now (they've deprecated `exec` at some point but then reconsidered). Had to write myself an alias for `git commit --amend && gitlab-runner exec`
- specialist 8y agoI'm a simple cave man. I don't understand why there needs to be an execution graph. Conditional logic? "Everything programming eventually drops down to metal one way or another." Keeping with the trog theme... I need things to be as explicit as possible. I just couldn't figure out why my script commands were not doing variable substitution. I expected my line ' - echo id: $CI_BUILD_ID' to cut and paste 'echo id: $CI_BUILD_ID' to the command line and run that bad boy as-is. And I'd see the expected output in the log. Just like if I run 'echo id: $CI_BUILD_ID' locally. What could be more simple? Nope. gitlab-ci does some ninja code escape mangling thing (turn on debugging output to see it in action). So in your log you continue to see 'echo id: $CI_BUILD_ID' instead of 'id: 123456'. I have no idea why anyone would do that. I want to run commands. Why not just run them as-is? Why the obfuscation layers? -- Inevitably, abstraction layers introduce an impedance mismatch, where some behavior will surprise you, where you have to do a workaround. Then you're fighting the tool instead of solving your problem directly. Then you say "fuck it" and just have the layer call your loose code (eg shell script) directly. Then what's the point of having the layer at all?
- drdaeman 8y ago> I don't understand why there needs to be an execution graph. Conditional logic? Of course. You may have different requirements, but here's an example flow of the one project I had worked on: Lint -------------------- \ Build -+-- Tests (fast) --+-- Release --- Deploy \ / - Tests (slow) - There are some extra rules like deploy and slow tests only running for release tag builds (vX.Y.Z) and lint being allowed to fail for branches with "wip-" prefix in their name. Lint runs in parallel with build + test (multiple runners, on different machines, each having some concurrency). Different tests run in parallel so you have to wait a little less. Tests only run after build is ready. Release only runs if both lint and tests succeed. Deploy is after the release. You can write this graph as a shell script, but this is not without downsides: 1. You need a DSL (a bunch of pre-defined shell functions or external commands) that control the individual jobs in a sane manner. 2. You can't just look at the script and draw the pipeline graph (so before the pipeline's ran CI/CD can't tell you what's going to happen next). 3. You can't guarantee that the plan will be the same if pipeline is re-ran again. With shell scripts, program output may change, even though all inputs are the same. If all you need is to run a simple shell script, with no multi-machine parallelism and no fancy conditional logic - all of the YAML stuff is an overkill you don't need. Although I guess all you need is approximately this: run-ci: script: - ./run-ci.sh Which doesn't look like any large sacrifice to me. > gitlab-ci does some ninja code escape mangling thing (turn on debugging output to see it in action). So in your log you continue to see 'echo id: $CI_BUILD_ID' instead of 'id: 123456'. Not sure I've encountered this much, but yes, this probably makes things complex. I agree that this is can be a pain point. > I have no idea why anyone would do that. I'm not sure, but the probable rationale is likely that they didn't want secrets to leak. Variables are the place where you can store various secret information, like CI signing keys or remote server credentials. They probably didn't want to leak what happens with e.g. `curl --user "ci:$CI_DEPLOY_PASSWORD" "$CI_DEPLOY_URL"`.
- Matt3o12_ 8y agoBut how do you tell a spell script to be a multi staged build? You can't just write `set OS=macos` and suddenly your OS changes from debian to macos. The best you could do is to ssh into that machine but how do you tell your build system to provision a machine with such parameters? You would essential have to build a new build system just for that. Or you could just use a config file -- whether you call it config.ini, .gitlab-ci.yaml, or build.json should not matter. Furthermore, how would you execute a bash script on Windows? You would have to at least tell it to enable Linux Subsystem and make sure it is a Pro version. I personally don't like to have a whole shell script inside .gitlab-ci.yaml file either -- that's why I have a seperate script called setup.bash/build.bash inside a CI folder and juse write `script: ./ci/build.bash` inside gitlab-ci.yaml. But nobody ever told you, you cannot do that. The gitlab-ci.yml is just a configuration file for the build system to tell it what you want your environment(s) to be. I agree that error reporting for YAML files suck -- but at least they are more consistent and easier to parse then XML files. And they are also way more readable then JSON files -- they have essential features such as long strings, and comments. The only annoying thing with Gitlab CI (and any other CI system) is to make sure it works. It often takes me 10 tries to make sure everything works correctly.
- specialist 8y agoMulti-stage? I dunno. Simplify your workflow? "just write `script: ./ci/build.bash` inside gitlab-ci.yaml" Yup. Then why have the YAML? I said shell script, not bash. The first "CI/CD" (as the kids now call it) I ever did was on Windows. Worked great.
- Matt3o12_ 8y ago> Multi-stage? I dunno. Simplify your workflow? How would you simplify testing on multiple systems/oses? Or have one project written be tested for multiple versions of that language (because you need backwards compatibility)? It's nice if your build system is as simple as make build test publish and only need to have it work on a Linux distribution of your choice, but some many projects, that's just not enough. Your .gitlab-ci file is just a file to tell your build system: for debian-python3.{3,4,5,6} execute that, for windows: do that. Btw: debian-python:devel is allowed to fail because it is not stable. It's just nice to have an idea if it might work for further versions.