6 ms·

1. Don't use bash, use a scripting language that is more CI friendly. I strongly prefer pwsh. 2. Don't have logic in your workflows. Workflows should be dumb a

1. Don't use bash, use a scripting language that is more CI friendly. I strongly prefer pwsh. 2. Don't have logic in your workflows. Workflows should be dumb and simple (KISS) and they should call your scripts. 3. Having standalone scripts will allow you to develop/modify and test locally without having to get caught in a loop of hell. 4. Design your entire CI pipeline for easier debugging, put that print state in, echo out the version of whatever. You don't need it _now_, but your future self will thank you when you do it need it. 5. Consider using third party runners that have better debugging capabilities

Storment33 9mo ago
I would disagree with 1. if you need anything more than shell that starts to become a smell to me. The build/testing process etc should be simple enough to not need anything more.
- embedding-shape 9mo ago
  That's literally point #2, but I had the same reaction as you when I first read point #1 :)
  - Storment33 9mo ago
    I agree with #2, I meant more if you are calling out to something that is not a task runner(Make, Taskfile, Just etc) or a shell script thats a bit of a smell to me. E.g. I have seen people call out to Python scripts etc and it concerns me.
    - embedding-shape 9mo ago
      Huh? Who cares if the script is .sh, .bash, Makefile, Justfile, .py, .js or even .php? If it works it works, as long as you can run it locally, it'll be good enough, and sometimes it's an even better idea to keep it in the same language the rest of the project is. It all depends and what language a script is made in shouldn't be considered a "smell".
      - Storment33 9mo ago
        > Huh? Who cares if the script is .sh, .bash, Makefile, Justfile, .py, .js or even .php? Me, typically I have found it to be a sign of over-engineering and found no benefits over just using shell script/task runner, as all it should be is plumbing that should be simple enough that a task runner can handle it. > If it works it works, as long as you can run it locally, it'll be good enough, Maybe when it is your own personal project "If it works it works" is fine. But when you come to corporate environment there starts to be issues of readability, maintainability, proprietary tooling, additional dependencies etc I have found when people start to over-engineer and use programming languages(like Python). E.g. > never_inline 30 minutes ago | parent | prev | next [–] > Build a CLI in python or whatever which does the same thing as CI, every CI stage should just call its subcommands. However, > and sometimes it's an even better idea to keep it in the same language the rest of the project is I'll agree. Depending on the project's language etc other options might make sense. But personally so far everytime I have come across something not using a task runner it has just been the wrong decision.
        pjc50 9mo ago
        Using shell becomes deeply miserable as soon as you encounter its kryptonite, the space character. Especially but not limited to filenames.
        embedding-shape 9mo ago
        > But personally so far everytime I have come across something not using a task runner it has just been the wrong decision. Yeah, tends to happen a lot when you hold strong opinions with strong conviction :) Not that it's wrong or anything, but it's highly subjective in the end. Typically I see larger issues being created from "under-engineering" and just rushing with the first idea people can think of when they implement things, rather than "over-engineering" causing similarly sized future issues. But then I also know everyone's history is vastly different, my views are surely shaped by the specific issues I've witnessed (and sometimes contributed to :| ), than anything else.
        Storment33 9mo ago
    - masfuerte 9mo ago
      My software runs on Windows, Linux and MacOS. The same Python testing code runs on all three platforms. I mostly dislike Python but I can't think of anything better for this use case.
      - tracker1 9mo ago
        You might consider Deno with Typescript... it's a single exe runtime, with a self-update mechanism (deno upgrade) and can run typescript/javascript files that directly reference the repository/http/modules that it needs and doesn't require a separate install step for dependency management. I've been using it for most of my local and environment scripting since relatively early on.
      - Storment33 9mo ago
        I don't touch Windows so I would not know. > The same Python testing code runs on all three platforms. I have no objections to Python being used for testing, I use it myself for the end to end tests in my projects. I just don't think Python as a build script/task runner is a good idea, see below where I got Claude to convert one of my open source projects for an example.
    - WorldMaker 9mo ago
      It's interesting because #1 is still suggesting a shell script, it's just suggesting a better shell to script.
      - Storment33 9mo ago
        I had no idea 'pwsh' was PowerShell. Personally not interested, maybe if your a Microsoft shop or something then yeah.
        WorldMaker 9mo ago
        "pwsh" is often used as the short-hand for modern cross-platform PowerShell to better differentiate it from the old Windows-only PowerShell. I think pwsh is worth exploring. It is cross-platform. It is post-Python and the Python mantra that "~~code~~ scripts are read more often than they are written". It provides a lot of nice tools out of the box. It's built in an "object-oriented" way, resembling Python and owing much to C#. When done well the "object-oriented" way provides a number of benefits over "dumb text pipes" that shells like bash were built on. It is easy to extend with C# and a few other languages, should you need to extend it. I would consider not dismissing it off hand without trying it just because Microsoft built it and/or that it was for a while Windows-only.
        ptx 9mo ago
        It's also both a larger download and slower to start than Java, which is not known for being light and nimble. In fact, PowerShell is so slow that you can both compile and run the equivalent C# program before PowerShell finishes launching. Not ideal for a shell or a scripting language. Also, the newer versions aren't included with Windows, which would have been useful – instead Windows includes an incompatible older version that admonishes you to download the new version. But why would you download several hundred megabytes of pwsh when you can equally well download any other language runtime? Also, it sends "telemetry" to Microsoft by default. Also, the error handling is just awful, silencing errors by default, requiring several different incantations to fix. Also, the documentation is vague and useless. And the syntax is ugly.
- dijit 9mo ago
  I mean, at some point you are bash calling some other language anyway. I'm a huge fan of "train as you fight", whatever build tools you have locally should be what's used in CI. If your CI can do things that you can't do locally: that is a problem.
  - embedding-shape 9mo ago
    > If your CI can do things that you can't do locally: that is a problem. Probably most of the times when this is an actual problem, is building across many platforms. I'm running Linux x86_64 locally, but some of my deliverables are for macOS and Windows and ARM, and while I could cross-compile for all of them on Linux (macOS was a bitch to get working though), it always felt better to compile on the hardware I'm targeting. Sometimes there are Windows/macOS-specific failures, and if I couldn't just ssh in and correct/investigate, and instead had to "change > commit > push" in an endless loop, it's possible I'd quite literally would lose my mind.
    - ethin 9mo ago
      I literally had to do this push > commit > test loop yesterday because apparently building universal Python wheels on MacOS is a pain in the ass. And I don't have a mac, so if I want to somewhat reliably reproduce how the runner might behave, I have to either test it on GH actions or rent one from something like Scaleway. Mainly because I don't currently knwo how else to do it. It's so, so frustrating and if anyone has ideas on making my life a bit better that would be nice lol.
      - Imustaskforhelp 9mo ago
        there is quickemu which can install mac vm on linux (or any other host) rather quickly, what are your thoughts on it (I am an absolute quickemu shill because I love that software) https://github.com/quickemu-project/quickemu https://github.com/quickemu-project/quickemu [ Quickly create and run optimised Windows, macOS and Linux virtual machines ]
        9mo ago
embedding-shape 9mo ago
Step 0. Stop using CI services that purposefully waste your time, and use CI services that have "Rebuild with SSH" or similar. From previous discussions (https://news.ycombinator.com/item?id=46592643 https://news.ycombinator.com/item?id=46592643), seems like Semaphore CI still offers that.
ufo 9mo ago
How do you handle persistent state in your actions? For my actions, the part that takes the longest to run is installing all the dependencies from scratch. I'd like to speed that up but I could never figure it out. All the options I could find for caching deps sounded so complicated.
- embedding-shape 9mo ago
  > How do you handle persistent state in your actions? You shouldn't. Besides caching that is. > All the options I could find for caching deps sounded so complicated. In reality, it's fairly simple, as long as you leverage content-hashing. First, take your lock file, compute the sha256sum. Then check if the cache has an artifact with that hash as the ID. If it's found, download and extract, those are your dependencies. If not, you run the installation of the dependencies, then archive the results, with the ID set to the hash. It really isn't more to it. I'm sure there are helpers/sub-actions/whatever Microsoft calls it, for doing all of this with 1-3 lines or something.
  - ufo 9mo ago
    The tricky bit for me was figuring out which cache to use, and how to use and test it locally. Do you use the proprietary github actions stuff? If the installation process inside the actions runner is different from what we use in the developer machines, now we have two sets of scripts and it's harder to test and debug...
    - embedding-shape 9mo ago
      > Do you use the proprietary github actions stuff? If I can avoid it, no. Almost everything I can control is outside of the Microsoft ecosystem. But as a freelancer, I have to deal a bunch with GitHub and Microsoft anyways, so in many of those cases, yes. Many times, I end up using https://github.com/actions/cache https://github.com/actions/cache for the clients who already use Actions, and none of that runs in the local machines at all. Typically I use a single Makefile/Justfile, that sometimes have most of the logic inside of it for running tests and what not, sometimes shell out to "proper" scripts. But that's disconnected from the required "setup", so Make/Just doesn't actually download dependencies, that's outside of the responsibilities of whatever runs the test. And also, with a lot of languages, it doesn't matter if you run an extra "npm install" over already existing node_modules/, it'll figure out what's missing/there already, so you could in theory still have "make test" do absolute everything locally, including installing dependencies (if you now wish this), and still do the whole "hash > find cache > extract > continue" thing before running "make test", and it'll skip the dependencies part if it's there already.
never_inline 9mo ago
Build a CLI in python or whatever which does the same thing as CI, every CI stage should just call its subcommands.
- Storment33 9mo ago
  Just use a task runner(Make, Just, Taskfile) this is what they were designed for.
  - jonhohle 9mo ago
    I typically use make for this and feel like I’m constantly clawing back scripts written in workflows that are hard to debug if they’re even runnable locally. This isn’t only a problem with GitHub Actions though. I’ve run into it with every CI runner I’ve come across.
  - never_inline 9mo ago
    In many enterprise environments, deployment logic would be quite large for bash.
    - Storment33 9mo ago
      Personally, I have never found the Python as a task runners to be less code, more readable or maintainable.
linuxftw 9mo ago
1. Just no. Unless you are some sort of Windows shop.
- rerdavies 9mo ago
  If you're building for Windows, then bash is "just no", so it's either cmd/.bat, or pwsh/.ps. <shrugs>
  - pixl97 9mo ago
    I mean, if you're a Windows shop you really should be using powershell.
  - import 9mo ago
    That’s the only reason for sure.
  - zabzonk 9mo ago
    I develop on Windows. And I use bash and (gnu) make - combination that cannot be beat, in my experience.
  - c-hendricks 9mo ago
    All my windows work / ci runs still use bash.
- jayd16 9mo ago
  Pwsh scripts are portable across mac, linux and windows with arguably less headache than bash. Its actually really nice. You should try it. If you don't like it, you can get bash to work on windows anyway.
- NSPG911 9mo ago
  you should try it, powershell isnt just 'type insanely long phrases', there are aliases for it
- latentsea 9mo ago
  I've never been a fan of PowerShell. I work across Windows and Linux on .NET and Nuke has become my go to tool for this.
zelphirkalt 9mo ago
I don't agree with (1), but agree with (2). I recommend just putting a Makefile in the repo and have that have CI targets, which you can then easily call from CI via a simple `make ci-test` or similar. And don't make the Makefiles overcomplicated. Of course, if you use something else as a task runner, that works as well.
- reactordev 9mo ago
  Makefile or scripts/do_thing either way this is correct. CI workflows should only do 1 thing each step. That one thing should be a command. What that command does is up to you in the Makefile or scripts. This keeps workflows/actions readable and mostly reusable.
- elSidCampeador 9mo ago
  makefile commands are the way
- pydry 9mo ago
  >I don't agree with (1) Neither do most people, probably but it's kinda neat how they suggested fix for github actions' ploy to maintain vendor lock-in is to swap it with a language invented by that very same vendor.
- Wilder7977 9mo ago
  For certain things, makefiles are great options. For others though they are a nightmare. From a security perspective, especially if you are trying to reach SLSA level 2+, you want all the build execution to be isolated and executed in a trusted, attestable and disposable environment, following predefined steps. Having makefiles (or scripts) with logical steps within them, makes it much, much harder to have properly attested outputs. Using makefiles mixes execution contexts between the CI pipeline and the code within the repository (that ends up containing the logic for the build), instead of using - centrally stored - external workflows that contains all the business logic for the build steps (e.g., compiler options, docker build steps etc.). For example, how can you attest in the CI that your code is tested if the workflow only contains "make test"? You need to double check at runtime what the makefile did, but the makefile might have been modified by that time, so you need to build a chain of trust etc. Instead, in a standardized workflow, you just need to establish the ground truth (e.g., tools are installed and are at this path), and the execution cannot be modified by in-repo resources.
newsoftheday 9mo ago
> Don't use bash What? Bash is the best scripting language available for CI flows.
kstrauser 9mo ago
I was once hired to manage a build farm. All of the build jobs were huge pipelines of Jenkins plugins that did various things in various orders. It was a freaking nightmare. Never again. Since then, every CI setup I’ve touched is a wrapper around “make build” or similar, with all the smarts living in Git next to the code it was building. I’ll die on this hill.
- TeeMassive 9mo ago
  Pathological organizations for some reason all prefer to never version the CI in the same repository it is testing.
tracker1 9mo ago
Minor variance on #1, I've come to use Deno typescripts for anything more complex than what can be easily done in bash or powershell. While I recognize that pwsh can do a LOT in the box, I absolutely hate the ergonomics and a lot of the interactions are awkward for people not used to it, while IMO more developers will be more closely aligned to TypeScript/JavaScript. Not to mention, Deno can run TS directly and can reference repository/http modules directly without a separate install step, which is useful for shell scripting beyond what pwsh can do. ex: pulling a dbms client and interacting directly for testing, setup or configuration. For the above reasons, I'll also use Deno for e2e testing over other languages that may be used for the actual project/library/app.
arwhatever 9mo ago
Do you (or does anyone) see possible value in a CI tool that just launches your script directly? It seems like if you > 2. Don't have logic in your workflows. Workflows should be dumb and simple (KISS) and they should call your scripts. then you’re basically working against or despite the CI tool, and at that point maybe someone should build a better or more suitable CI tool.
- zelphirkalt 9mo ago
  Can we have a CI tool, that simply takes a Makefile as input? Perhaps takes all targets, that start with "ci" or something.
jayd16 9mo ago
#2 is not a slam dunk because the CI system loses insight into your build process if you just use one big script. Does anyone have a way to mark script sections as separate build steps with defined artifacts? Would be nice to just have scripts with something like. BeginStep("Step Name") ... EndStep("Step Name", artifacts) They could noop on local runs but be reflected in the github/gitlab as separate steps/stages and allow resumes and retries and such. As it stands there's no way to really have CI/CD run the exact same scripts locally and get all the insights and functionality. I haven't seen anything like that but it would be nice to know.
latentsea 9mo ago
We use Nuke for this purpose and I really like it. We're a .NET shop, and Nuke is C#, so works pretty well. Almost all our Nuke targets are shared between local and CI except some docker stuff that needs caching in CI.

Storment33 9mo ago

I would disagree with 1. if you need anything more than shell that starts to become a smell to me. The build/testing process etc should be simple enough to not need anything more.

embedding-shape 9mo ago
That's literally point #2, but I had the same reaction as you when I first read point #1 :)
- Storment33 9mo ago
  I agree with #2, I meant more if you are calling out to something that is not a task runner(Make, Taskfile, Just etc) or a shell script thats a bit of a smell to me. E.g. I have seen people call out to Python scripts etc and it concerns me.
  - embedding-shape 9mo ago
    Huh? Who cares if the script is .sh, .bash, Makefile, Justfile, .py, .js or even .php? If it works it works, as long as you can run it locally, it'll be good enough, and sometimes it's an even better idea to keep it in the same language the rest of the project is. It all depends and what language a script is made in shouldn't be considered a "smell".
    - Storment33 9mo ago
      > Huh? Who cares if the script is .sh, .bash, Makefile, Justfile, .py, .js or even .php? Me, typically I have found it to be a sign of over-engineering and found no benefits over just using shell script/task runner, as all it should be is plumbing that should be simple enough that a task runner can handle it. > If it works it works, as long as you can run it locally, it'll be good enough, Maybe when it is your own personal project "If it works it works" is fine. But when you come to corporate environment there starts to be issues of readability, maintainability, proprietary tooling, additional dependencies etc I have found when people start to over-engineer and use programming languages(like Python). E.g. > never_inline 30 minutes ago | parent | prev | next [–] > Build a CLI in python or whatever which does the same thing as CI, every CI stage should just call its subcommands. However, > and sometimes it's an even better idea to keep it in the same language the rest of the project is I'll agree. Depending on the project's language etc other options might make sense. But personally so far everytime I have come across something not using a task runner it has just been the wrong decision.
      - pjc50 9mo ago
        Using shell becomes deeply miserable as soon as you encounter its kryptonite, the space character. Especially but not limited to filenames.
      - embedding-shape 9mo ago
        > But personally so far everytime I have come across something not using a task runner it has just been the wrong decision. Yeah, tends to happen a lot when you hold strong opinions with strong conviction :) Not that it's wrong or anything, but it's highly subjective in the end. Typically I see larger issues being created from "under-engineering" and just rushing with the first idea people can think of when they implement things, rather than "over-engineering" causing similarly sized future issues. But then I also know everyone's history is vastly different, my views are surely shaped by the specific issues I've witnessed (and sometimes contributed to :| ), than anything else.
        Storment33 9mo ago
  - masfuerte 9mo ago
    My software runs on Windows, Linux and MacOS. The same Python testing code runs on all three platforms. I mostly dislike Python but I can't think of anything better for this use case.
    - tracker1 9mo ago
      You might consider Deno with Typescript... it's a single exe runtime, with a self-update mechanism (deno upgrade) and can run typescript/javascript files that directly reference the repository/http/modules that it needs and doesn't require a separate install step for dependency management. I've been using it for most of my local and environment scripting since relatively early on.
    - Storment33 9mo ago
      I don't touch Windows so I would not know. > The same Python testing code runs on all three platforms. I have no objections to Python being used for testing, I use it myself for the end to end tests in my projects. I just don't think Python as a build script/task runner is a good idea, see below where I got Claude to convert one of my open source projects for an example.
  - WorldMaker 9mo ago
    It's interesting because #1 is still suggesting a shell script, it's just suggesting a better shell to script.
    - Storment33 9mo ago
      I had no idea 'pwsh' was PowerShell. Personally not interested, maybe if your a Microsoft shop or something then yeah.
      - WorldMaker 9mo ago
        "pwsh" is often used as the short-hand for modern cross-platform PowerShell to better differentiate it from the old Windows-only PowerShell. I think pwsh is worth exploring. It is cross-platform. It is post-Python and the Python mantra that "~~code~~ scripts are read more often than they are written". It provides a lot of nice tools out of the box. It's built in an "object-oriented" way, resembling Python and owing much to C#. When done well the "object-oriented" way provides a number of benefits over "dumb text pipes" that shells like bash were built on. It is easy to extend with C# and a few other languages, should you need to extend it. I would consider not dismissing it off hand without trying it just because Microsoft built it and/or that it was for a while Windows-only.
        ptx 9mo ago
        It's also both a larger download and slower to start than Java, which is not known for being light and nimble. In fact, PowerShell is so slow that you can both compile and run the equivalent C# program before PowerShell finishes launching. Not ideal for a shell or a scripting language. Also, the newer versions aren't included with Windows, which would have been useful – instead Windows includes an incompatible older version that admonishes you to download the new version. But why would you download several hundred megabytes of pwsh when you can equally well download any other language runtime? Also, it sends "telemetry" to Microsoft by default. Also, the error handling is just awful, silencing errors by default, requiring several different incantations to fix. Also, the documentation is vague and useless. And the syntax is ugly.
dijit 9mo ago
I mean, at some point you are bash calling some other language anyway. I'm a huge fan of "train as you fight", whatever build tools you have locally should be what's used in CI. If your CI can do things that you can't do locally: that is a problem.
- embedding-shape 9mo ago
  > If your CI can do things that you can't do locally: that is a problem. Probably most of the times when this is an actual problem, is building across many platforms. I'm running Linux x86_64 locally, but some of my deliverables are for macOS and Windows and ARM, and while I could cross-compile for all of them on Linux (macOS was a bitch to get working though), it always felt better to compile on the hardware I'm targeting. Sometimes there are Windows/macOS-specific failures, and if I couldn't just ssh in and correct/investigate, and instead had to "change > commit > push" in an endless loop, it's possible I'd quite literally would lose my mind.
  - ethin 9mo ago
    I literally had to do this push > commit > test loop yesterday because apparently building universal Python wheels on MacOS is a pain in the ass. And I don't have a mac, so if I want to somewhat reliably reproduce how the runner might behave, I have to either test it on GH actions or rent one from something like Scaleway. Mainly because I don't currently knwo how else to do it. It's so, so frustrating and if anyone has ideas on making my life a bit better that would be nice lol.
    - Imustaskforhelp 9mo ago
      there is quickemu which can install mac vm on linux (or any other host) rather quickly, what are your thoughts on it (I am an absolute quickemu shill because I love that software) https://github.com/quickemu-project/quickemu https://github.com/quickemu-project/quickemu [ Quickly create and run optimised Windows, macOS and Linux virtual machines ]
      - 9mo ago

embedding-shape 9mo ago

Step 0. Stop using CI services that purposefully waste your time, and use CI services that have "Rebuild with SSH" or similar. From previous discussions (https://news.ycombinator.com/item?id=46592643 https://news.ycombinator.com/item?id=46592643), seems like Semaphore CI still offers that.

ufo 9mo ago

How do you handle persistent state in your actions? For my actions, the part that takes the longest to run is installing all the dependencies from scratch. I'd like to speed that up but I could never figure it out. All the options I could find for caching deps sounded so complicated.

embedding-shape 9mo ago
> How do you handle persistent state in your actions? You shouldn't. Besides caching that is. > All the options I could find for caching deps sounded so complicated. In reality, it's fairly simple, as long as you leverage content-hashing. First, take your lock file, compute the sha256sum. Then check if the cache has an artifact with that hash as the ID. If it's found, download and extract, those are your dependencies. If not, you run the installation of the dependencies, then archive the results, with the ID set to the hash. It really isn't more to it. I'm sure there are helpers/sub-actions/whatever Microsoft calls it, for doing all of this with 1-3 lines or something.
- ufo 9mo ago
  The tricky bit for me was figuring out which cache to use, and how to use and test it locally. Do you use the proprietary github actions stuff? If the installation process inside the actions runner is different from what we use in the developer machines, now we have two sets of scripts and it's harder to test and debug...
  - embedding-shape 9mo ago
    > Do you use the proprietary github actions stuff? If I can avoid it, no. Almost everything I can control is outside of the Microsoft ecosystem. But as a freelancer, I have to deal a bunch with GitHub and Microsoft anyways, so in many of those cases, yes. Many times, I end up using https://github.com/actions/cache https://github.com/actions/cache for the clients who already use Actions, and none of that runs in the local machines at all. Typically I use a single Makefile/Justfile, that sometimes have most of the logic inside of it for running tests and what not, sometimes shell out to "proper" scripts. But that's disconnected from the required "setup", so Make/Just doesn't actually download dependencies, that's outside of the responsibilities of whatever runs the test. And also, with a lot of languages, it doesn't matter if you run an extra "npm install" over already existing node_modules/, it'll figure out what's missing/there already, so you could in theory still have "make test" do absolute everything locally, including installing dependencies (if you now wish this), and still do the whole "hash > find cache > extract > continue" thing before running "make test", and it'll skip the dependencies part if it's there already.

never_inline 9mo ago

Build a CLI in python or whatever which does the same thing as CI, every CI stage should just call its subcommands.

Storment33 9mo ago
Just use a task runner(Make, Just, Taskfile) this is what they were designed for.
- jonhohle 9mo ago
  I typically use make for this and feel like I’m constantly clawing back scripts written in workflows that are hard to debug if they’re even runnable locally. This isn’t only a problem with GitHub Actions though. I’ve run into it with every CI runner I’ve come across.
- never_inline 9mo ago
  In many enterprise environments, deployment logic would be quite large for bash.
  - Storment33 9mo ago
    Personally, I have never found the Python as a task runners to be less code, more readable or maintainable.

linuxftw 9mo ago

1. Just no. Unless you are some sort of Windows shop.

rerdavies 9mo ago
If you're building for Windows, then bash is "just no", so it's either cmd/.bat, or pwsh/.ps. <shrugs>
- pixl97 9mo ago
  I mean, if you're a Windows shop you really should be using powershell.
- import 9mo ago
  That’s the only reason for sure.
- zabzonk 9mo ago
  I develop on Windows. And I use bash and (gnu) make - combination that cannot be beat, in my experience.
- c-hendricks 9mo ago
  All my windows work / ci runs still use bash.
jayd16 9mo ago
Pwsh scripts are portable across mac, linux and windows with arguably less headache than bash. Its actually really nice. You should try it. If you don't like it, you can get bash to work on windows anyway.
NSPG911 9mo ago
you should try it, powershell isnt just 'type insanely long phrases', there are aliases for it
latentsea 9mo ago
I've never been a fan of PowerShell. I work across Windows and Linux on .NET and Nuke has become my go to tool for this.

zelphirkalt 9mo ago

I don't agree with (1), but agree with (2). I recommend just putting a Makefile in the repo and have that have CI targets, which you can then easily call from CI via a simple `make ci-test` or similar. And don't make the Makefiles overcomplicated. Of course, if you use something else as a task runner, that works as well.

reactordev 9mo ago
Makefile or scripts/do_thing either way this is correct. CI workflows should only do 1 thing each step. That one thing should be a command. What that command does is up to you in the Makefile or scripts. This keeps workflows/actions readable and mostly reusable.
elSidCampeador 9mo ago
makefile commands are the way
pydry 9mo ago
>I don't agree with (1) Neither do most people, probably but it's kinda neat how they suggested fix for github actions' ploy to maintain vendor lock-in is to swap it with a language invented by that very same vendor.
Wilder7977 9mo ago
For certain things, makefiles are great options. For others though they are a nightmare. From a security perspective, especially if you are trying to reach SLSA level 2+, you want all the build execution to be isolated and executed in a trusted, attestable and disposable environment, following predefined steps. Having makefiles (or scripts) with logical steps within them, makes it much, much harder to have properly attested outputs. Using makefiles mixes execution contexts between the CI pipeline and the code within the repository (that ends up containing the logic for the build), instead of using - centrally stored - external workflows that contains all the business logic for the build steps (e.g., compiler options, docker build steps etc.). For example, how can you attest in the CI that your code is tested if the workflow only contains "make test"? You need to double check at runtime what the makefile did, but the makefile might have been modified by that time, so you need to build a chain of trust etc. Instead, in a standardized workflow, you just need to establish the ground truth (e.g., tools are installed and are at this path), and the execution cannot be modified by in-repo resources.

newsoftheday 9mo ago

> Don't use bash What? Bash is the best scripting language available for CI flows.

kstrauser 9mo ago

I was once hired to manage a build farm. All of the build jobs were huge pipelines of Jenkins plugins that did various things in various orders. It was a freaking nightmare. Never again. Since then, every CI setup I’ve touched is a wrapper around “make build” or similar, with all the smarts living in Git next to the code it was building. I’ll die on this hill.

TeeMassive 9mo ago
Pathological organizations for some reason all prefer to never version the CI in the same repository it is testing.

tracker1 9mo ago

Minor variance on #1, I've come to use Deno typescripts for anything more complex than what can be easily done in bash or powershell. While I recognize that pwsh can do a LOT in the box, I absolutely hate the ergonomics and a lot of the interactions are awkward for people not used to it, while IMO more developers will be more closely aligned to TypeScript/JavaScript. Not to mention, Deno can run TS directly and can reference repository/http modules directly without a separate install step, which is useful for shell scripting beyond what pwsh can do. ex: pulling a dbms client and interacting directly for testing, setup or configuration. For the above reasons, I'll also use Deno for e2e testing over other languages that may be used for the actual project/library/app.

arwhatever 9mo ago

Do you (or does anyone) see possible value in a CI tool that just launches your script directly? It seems like if you > 2. Don't have logic in your workflows. Workflows should be dumb and simple (KISS) and they should call your scripts. then you’re basically working against or despite the CI tool, and at that point maybe someone should build a better or more suitable CI tool.

zelphirkalt 9mo ago
Can we have a CI tool, that simply takes a Makefile as input? Perhaps takes all targets, that start with "ci" or something.

jayd16 9mo ago

#2 is not a slam dunk because the CI system loses insight into your build process if you just use one big script. Does anyone have a way to mark script sections as separate build steps with defined artifacts? Would be nice to just have scripts with something like. BeginStep("Step Name") ... EndStep("Step Name", artifacts) They could noop on local runs but be reflected in the github/gitlab as separate steps/stages and allow resumes and retries and such. As it stands there's no way to really have CI/CD run the exact same scripts locally and get all the insights and functionality. I haven't seen anything like that but it would be nice to know.

latentsea 9mo ago

We use Nuke for this purpose and I really like it. We're a .NET shop, and Nuke is C#, so works pretty well. Almost all our Nuke targets are shared between local and CI except some docker stuff that needs caching in CI.