5 ms·

> Builds should just be done with shell scripts You still need something to describe the order and dependencies of those shell scripts as well as when to run t

> Builds should just be done with shell scripts You still need something to describe the order and dependencies of those shell scripts as well as when to run them (branches, tags) and how to handle the artifacts (expiry time, etc). Which should be, much preferably, something not Turing-complete, so you can make an UI for this. Draw the graph, etc etc. Take those parts out, and almost all what's left in .gitlab-ci.yml is just the `script` key. You can debug most things locally (without having to do that ugly commit-try-fail-repeat dance) with https://docs.gitlab.com/runner/commands/#gitlab-runner-exec https://docs.gitlab.com/runner/commands/#gitlab-runner-exec

specialist 8y ago
Turing complete? What? If your visual programming fuzzy whatzzit drops down to metal (shell commands in this case), just use the metal. I did get the git-runner to run locally. That was fun. Why do I have to commit my changes to my local branch so the git-runner can see them? MY CODE IS RIGHT THERE! I reiterate: Just use shell scripts.
- drdaeman 8y ago
  > Turing complete? What? You'll need to run the CI pipeline to figure out the execution graph. In a restricted language, you don't have to - it can just describe that graph. > If your visual programming fuzzy whatzzit drops down to metal I don't get you. Everything programming eventually drops down to metal one way or another. It is not about visual programming, it's about the appropriate languages. Shell scripts are not exactly appropriate to describe a graph. Especially a graph meant to be ran on a distributed system - so, really, *sh may be not the best fit here. Maybe you have some better idea, but I'm just afraid you'll end up with some shell-based DSL that would be no better than YAML, but suffer from all possible issues (for example, there's no concept of pure functions in shell scripting, so every CI run plan can take completely different shape) Oh. If you're a person who likes all their configuration files being programs - then I won't argue as it'll be a matter of taste. > Why do I have to commit my changes to my local branch so the git-runner can see them? Yeah, I'm also disappointed in this. It was dismissed as "wontfix" a while ago https://gitlab.com/gitlab-org/gitlab-runner/issues/1545 https://gitlab.com/gitlab-org/gitlab-runner/issues/1545 and seems that it stays this way even now (they've deprecated `exec` at some point but then reconsidered). Had to write myself an alias for `git commit --amend && gitlab-runner exec`
  - specialist 8y ago
    I'm a simple cave man. I don't understand why there needs to be an execution graph. Conditional logic? "Everything programming eventually drops down to metal one way or another." Keeping with the trog theme... I need things to be as explicit as possible. I just couldn't figure out why my script commands were not doing variable substitution. I expected my line ' - echo id: $CI_BUILD_ID' to cut and paste 'echo id: $CI_BUILD_ID' to the command line and run that bad boy as-is. And I'd see the expected output in the log. Just like if I run 'echo id: $CI_BUILD_ID' locally. What could be more simple? Nope. gitlab-ci does some ninja code escape mangling thing (turn on debugging output to see it in action). So in your log you continue to see 'echo id: $CI_BUILD_ID' instead of 'id: 123456'. I have no idea why anyone would do that. I want to run commands. Why not just run them as-is? Why the obfuscation layers? -- Inevitably, abstraction layers introduce an impedance mismatch, where some behavior will surprise you, where you have to do a workaround. Then you're fighting the tool instead of solving your problem directly. Then you say "fuck it" and just have the layer call your loose code (eg shell script) directly. Then what's the point of having the layer at all?
    - drdaeman 8y ago
      > I don't understand why there needs to be an execution graph. Conditional logic? Of course. You may have different requirements, but here's an example flow of the one project I had worked on: Lint -------------------- \ Build -+-- Tests (fast) --+-- Release --- Deploy \ / - Tests (slow) - There are some extra rules like deploy and slow tests only running for release tag builds (vX.Y.Z) and lint being allowed to fail for branches with "wip-" prefix in their name. Lint runs in parallel with build + test (multiple runners, on different machines, each having some concurrency). Different tests run in parallel so you have to wait a little less. Tests only run after build is ready. Release only runs if both lint and tests succeed. Deploy is after the release. You can write this graph as a shell script, but this is not without downsides: 1. You need a DSL (a bunch of pre-defined shell functions or external commands) that control the individual jobs in a sane manner. 2. You can't just look at the script and draw the pipeline graph (so before the pipeline's ran CI/CD can't tell you what's going to happen next). 3. You can't guarantee that the plan will be the same if pipeline is re-ran again. With shell scripts, program output may change, even though all inputs are the same. If all you need is to run a simple shell script, with no multi-machine parallelism and no fancy conditional logic - all of the YAML stuff is an overkill you don't need. Although I guess all you need is approximately this: run-ci: script: - ./run-ci.sh Which doesn't look like any large sacrifice to me. > gitlab-ci does some ninja code escape mangling thing (turn on debugging output to see it in action). So in your log you continue to see 'echo id: $CI_BUILD_ID' instead of 'id: 123456'. Not sure I've encountered this much, but yes, this probably makes things complex. I agree that this is can be a pain point. > I have no idea why anyone would do that. I'm not sure, but the probable rationale is likely that they didn't want secrets to leak. Variables are the place where you can store various secret information, like CI signing keys or remote server credentials. They probably didn't want to leak what happens with e.g. `curl --user "ci:$CI_DEPLOY_PASSWORD" "$CI_DEPLOY_URL"`.

specialist 8y ago

Turing complete? What? If your visual programming fuzzy whatzzit drops down to metal (shell commands in this case), just use the metal. I did get the git-runner to run locally. That was fun. Why do I have to commit my changes to my local branch so the git-runner can see them? MY CODE IS RIGHT THERE! I reiterate: Just use shell scripts.

drdaeman 8y ago
> Turing complete? What? You'll need to run the CI pipeline to figure out the execution graph. In a restricted language, you don't have to - it can just describe that graph. > If your visual programming fuzzy whatzzit drops down to metal I don't get you. Everything programming eventually drops down to metal one way or another. It is not about visual programming, it's about the appropriate languages. Shell scripts are not exactly appropriate to describe a graph. Especially a graph meant to be ran on a distributed system - so, really, *sh may be not the best fit here. Maybe you have some better idea, but I'm just afraid you'll end up with some shell-based DSL that would be no better than YAML, but suffer from all possible issues (for example, there's no concept of pure functions in shell scripting, so every CI run plan can take completely different shape) Oh. If you're a person who likes all their configuration files being programs - then I won't argue as it'll be a matter of taste. > Why do I have to commit my changes to my local branch so the git-runner can see them? Yeah, I'm also disappointed in this. It was dismissed as "wontfix" a while ago https://gitlab.com/gitlab-org/gitlab-runner/issues/1545 https://gitlab.com/gitlab-org/gitlab-runner/issues/1545 and seems that it stays this way even now (they've deprecated `exec` at some point but then reconsidered). Had to write myself an alias for `git commit --amend && gitlab-runner exec`
- specialist 8y ago
  I'm a simple cave man. I don't understand why there needs to be an execution graph. Conditional logic? "Everything programming eventually drops down to metal one way or another." Keeping with the trog theme... I need things to be as explicit as possible. I just couldn't figure out why my script commands were not doing variable substitution. I expected my line ' - echo id: $CI_BUILD_ID' to cut and paste 'echo id: $CI_BUILD_ID' to the command line and run that bad boy as-is. And I'd see the expected output in the log. Just like if I run 'echo id: $CI_BUILD_ID' locally. What could be more simple? Nope. gitlab-ci does some ninja code escape mangling thing (turn on debugging output to see it in action). So in your log you continue to see 'echo id: $CI_BUILD_ID' instead of 'id: 123456'. I have no idea why anyone would do that. I want to run commands. Why not just run them as-is? Why the obfuscation layers? -- Inevitably, abstraction layers introduce an impedance mismatch, where some behavior will surprise you, where you have to do a workaround. Then you're fighting the tool instead of solving your problem directly. Then you say "fuck it" and just have the layer call your loose code (eg shell script) directly. Then what's the point of having the layer at all?
  - drdaeman 8y ago
    > I don't understand why there needs to be an execution graph. Conditional logic? Of course. You may have different requirements, but here's an example flow of the one project I had worked on: Lint -------------------- \ Build -+-- Tests (fast) --+-- Release --- Deploy \ / - Tests (slow) - There are some extra rules like deploy and slow tests only running for release tag builds (vX.Y.Z) and lint being allowed to fail for branches with "wip-" prefix in their name. Lint runs in parallel with build + test (multiple runners, on different machines, each having some concurrency). Different tests run in parallel so you have to wait a little less. Tests only run after build is ready. Release only runs if both lint and tests succeed. Deploy is after the release. You can write this graph as a shell script, but this is not without downsides: 1. You need a DSL (a bunch of pre-defined shell functions or external commands) that control the individual jobs in a sane manner. 2. You can't just look at the script and draw the pipeline graph (so before the pipeline's ran CI/CD can't tell you what's going to happen next). 3. You can't guarantee that the plan will be the same if pipeline is re-ran again. With shell scripts, program output may change, even though all inputs are the same. If all you need is to run a simple shell script, with no multi-machine parallelism and no fancy conditional logic - all of the YAML stuff is an overkill you don't need. Although I guess all you need is approximately this: run-ci: script: - ./run-ci.sh Which doesn't look like any large sacrifice to me. > gitlab-ci does some ninja code escape mangling thing (turn on debugging output to see it in action). So in your log you continue to see 'echo id: $CI_BUILD_ID' instead of 'id: 123456'. Not sure I've encountered this much, but yes, this probably makes things complex. I agree that this is can be a pain point. > I have no idea why anyone would do that. I'm not sure, but the probable rationale is likely that they didn't want secrets to leak. Variables are the place where you can store various secret information, like CI signing keys or remote server credentials. They probably didn't want to leak what happens with e.g. `curl --user "ci:$CI_DEPLOY_PASSWORD" "$CI_DEPLOY_URL"`.