10 ms·

Already see people saying GitLab is better: yes it is, but it also sucks in different ways. After years of dealing with this (first Jenkins, then GitLab, then

by deng 2y ago

Already see people saying GitLab is better: yes it is, but it also sucks in different ways. After years of dealing with this (first Jenkins, then GitLab, then GitHub), my takeaway is: * Write as much CI logic as possible in your own code. Does not really matter what you use (shell scripts, make, just, doit, mage, whatever) as long as it is proper, maintainable code. * Invest time that your pipelines can run locally on a developer machine as well (as much as possible at least), otherwise testing/debugging pipelines becomes a nightmare. * Avoid YAML as much as possible, period. * Don't bind yourself to some fancy new VC-financed thing that will solve CI once and for all but needs to get monetized eventually (see: earthly, dagger, etc.) * Always use your own runners, on-premise if possible

Aeolun 2y ago
This is where I was going to say something about dagger, but it seems it turned into AI crud. Let me at least recommend depot.dev for having absurdly fast runners.
- oulipo 2y ago
  can you give more feedback about dagger? what is good/not good about it? I was going to start looking into it
  - toastal 2y ago
    For starts looking at their website, it looks like all collaboration is locked behind proprietary platforms… Discord, Twitter, LinkedIn, Microsoft GitHub.
  - Aeolun 2y ago
    I liked their setup before, though I never got around to actually using it, but the tagline on the website has changed to “AI powered workflow orchestration”, which is quite different from the original “Write pipeline once, run everywhere”
    - oulipo 2y ago
      yes but I went on their Discord, and the AI thing is more an "extension" of the CI/CD thing, it's just marketing-speak for the investors, they are still building the CI/CD tool
  - esafak 2y ago
    Look into it; it works, more or less.
- shykes 2y ago
  Hello! Dagger CEO here. We are indeed getting an influx of AI workloads (AI agents to be specific, which is the fancy industry term for "software with LLMs inside"), and are of course trying to capitalize on that in our marketing material. We're still looking for the right balance of CI and AI on our website. Crucially, it's the same engine running both. Because, as it turns out, AI agents are mostly workflows under the hood, and Dagger is great at running those. I shared more context in this thread: https://x.com/solomonstre/status/1895671390176747682 https://x.com/solomonstre/status/1895671390176747682
alex_suzuki 2y ago
> * Always use your own runners, on-premise if possible Why? I understand it in cases where security is critical or intellectual property is at stake. Are you talking about "snowflake runners" or just dumb executors of container images?
- deleted 2y ago
  [deleted]
- deng 2y ago
  It obviously depends on your load. Fast pipelines matter, so don't run them on some weak cloud runner with the speed of a C64. Fast cloud runners are expensive. Just invest some money and buy or at least rent some beefy servers with lots of cores, RAM and storage and never look back. Use caches for everything to speed up things. Security is another thing where this can come in handy, but properly firewalling CI runners and having mirrors of all your dependencies is a lot of work and might very well be overkill for most people.
  - zoobab 2y ago
    "Fast cloud runners are expensive." Buy a cheap Ryzen, and put it on your desk, that's a cheap runner.
    - withinboredom 2y ago
      30 bucks a month on hetzner for a dedicated machine with 12-16 cores and 64 gb of ram and unlimited 1gbps bandwidth.
- saidinesh5 2y ago
  Caching is nicer on own runners. No need to redownload 10+GB of "development container images" just to build your 10 lines of changed code. With self hosted Gitlab runners it was almost as fast as doing incremental builds. When your build process can take like 15-20 minutes (medium sized C++ code base), this brought down the total time to 30 seconds or so.
ehansdais 2y ago
After years of trial and error our team has come to the same conclusion. I know some people might consider this insanity, but we actually run all of our scripts as a separate C# CLI application (The main application is a C# web server). Effectively no bash scripts, except as the entry point here and there. The build step and passing the executable around is a small price to pay for the gain in static type checking, being able to pull in libraries as needed, and knowing that our CI is not going to down because someone made a dumb typo somewhere. The other thing I would add is consider passing in all environment variables as args. This makes it easy to see what dependencies the script actually needs, and has the bonus of being even more portable.
- baq 2y ago
  > I know some people might consider this insanity Some people here still can’t believe YAML is used for not only configuration, but complex code like optimized CI pipelines. This is insane. You’re actually introducing much needed sanity into the process by admitting that a real programming language is the tool to use here. I can’t imagine the cognitive dissonance Lisp folks have when dealing with this madness, not being one myself.
  - mschuster91 2y ago
    > Some people here still can’t believe YAML is used for not only configuration, but complex code like optimized CI pipelines. This is insane. It's miles better than Jenkins and the horrors people created there. GitLab CI can at least be easily migrated to any other GitLab instance and stuff should Just Work because it is in the end not much more than self contained bash scripts, but Jenkins... is a clown show, especially for Ops people of larger instances. On one side, you got 50 plugins with CVEs but you can't update them because you need to find a slot that works for all development teams to have a week or two to fix their pipelines again, and on the other side you got a Jenkins instance for each project which lessens the coordination effort but you gotta worry about dozens of Jenkins instances. Oh and that doesn't include the fact many old pipelines aren't written in Groovy or, in fact, in any code at all but only in Jenkins's UI... Github Actions however, I'd say for someone coming from GitLab, is even worse to work with than Jenkins.
JanMa 2y ago
Whenever possible I now just use GitHub actions as a thin wrapper around a Makefile and this has improved my experience with it a lot. The Makefile takes care of installing all necessary dependencies and runs the relevant build/Test commands. This also enables me to test that stuff locally again without the long feedback loop mentioned in other comments in this thread.
- oulipo 2y ago
  mise (https://mise.jdx.dev/ https://mise.jdx.dev/) and dagger (https://github.com/dagger/dagger https://github.com/dagger/dagger) seem like nice candidates too! Mise can install all your deps, and run tasks
  - lou1306 2y ago
    > * Don't bind yourself to some fancy new VC-financed thing that will solve CI once and for all but needs to get monetized eventually (see: earthly, dagger, etc.) Literally from comment at the root of this thread.
    - triyambakam 2y ago
      But are mise and dagger VC funded? I don't see any pricing pages there.
      - deleted 2y ago
        [deleted]
      - deng 2y ago
        Dagger is even YCombinator funded. https://www.boringbusinessnerd.com/startups/dagger https://www.boringbusinessnerd.com/startups/dagger Mise indeed isn't, but its scope is quite a bit smaller than Dagger.
        internetter 2y ago
        > Mise indeed isn't, but its scope is quite a bit smaller than Dagger. A lot of us could learn... do one thing and do it well
cesnja 2y ago
You can build the first pipeline with oneliners, but as long as you want to keep optimizing the pipelines, the yaml code will keep piling up with CI vendor's specific approaches to job selection, env variable delivery, caching, output sharing between jobs and so on.
forrestthewoods 2y ago
> pipelines can run locally on a developer machine as well (as much as possible at least) Facts. However I’ll go a step further and say “only implement your logic in a tool that has a debugger”. YAML is the worse. But shell scripts are second worst. Use a real language.
- never_inline 2y ago
  Python with click and PyYAML can go a long way - then you can build it as a CLI application and use the same from CI. In a java shop, picocli + graalvm probably. I wouldn't like Go for this purpose (against the conventional wisdom - because boilerplate and pretty bad debugging capabilities). That said, if you absolutely need to use shell script for reasons, keep it all in single script, define logging functions including debug logs, rigorously check every constraint and variable, use shellcheck, factor the code well into functions - I should sometimes write a blog post about it.
- folmar 2y ago
  There is a debugger for bash: https://github.com/Trepan-Debuggers/bashdb https://github.com/Trepan-Debuggers/bashdb Not that I'm recommending 10k-line programs in bash, but a debugger is useful when you need it.
- sgarland 2y ago
  Shell is a very real language, and it has a debugger; it’s called set -x and/or strace.
  - forrestthewoods 2y ago
    Like I said, use something with a real debugger.
ozim 2y ago
Good insight, because that is just a complex issue - especially when there is team churn and everyone adds their parts in yaml or configuration. Doesn’t matter Jenkins or actions - it is just complicated. Making it simpler is on devs/ops not the tool.
qweiopqweiop 2y ago
Can you explain YAML? I've found declarative pipelines with it have been... fine?
- fergie 2y ago
  As a developer based in Norway, one fairly major drawback to YAML is the way that it processes the language code for Norwegian ("no").
  - marcusramberg 2y ago
    That particular pain was fixed in yaml 1.2 :)
    - withinboredom 2y ago
      TIL I learned that yaml has versions and now I'm wondering what version of yaml parsers are running where. I think I might be on a new level of hell. It would be better to delete your comment so nobody else has to has to ever have this crisis.
  - LeonM 2y ago
    The Norwegian ISO3166 code colliding with the English word 'no' is not a YAML problem per se, I've been bitten by that a few times in other situations as well. For example: Stripe uses constants for types of tax registration numbers (VAT/GST/TIN, etc.). So there is EU_VAT for European VAT numbers, US_TIN for US tax identification numbers, etc. But what value to use for tax-exempt organisations that don't have a tax number? Well... guess how I found out about NO_VAT... On the bright side, I did learn that way that although Norway is in the Schengen zone, apparently they are not part of the EU (hence the separation of EU_VAT and NO_VAT). I guess the 'no' name collision has taught many developers something about Norway :-)
- dharmab 2y ago
  1. The YAML spec is extremely complex with some things being ambiguous. You might not notice this if your restrict yourself to a small subset of the language. But you will notice it when different YAML libraries and programming languages interpret the same YAML file as different content. 2. Trying to encode logic and control flow in a YAML document is much more difficult than writing that flow in a "real" programming language. Debugging is especially much easier in "real" languages.
cookiengineer 2y ago
This. I heavily invested in a local runner based CI/CD workflow. First I was using gogs and drone, now the forgejo and woodpecker CI forks. It runs with multiple redundancies because it's a pretty easy setup to replicate on decentralized hardware. The only thing that's a little painful is authentication and cross-system pull requests, so we still need our single point of failure to merge feature branches and do code reviews. Due to us building everything in go, we also decided to have always a /toolchain/build.go so that we have everything in a single language, and don't need even bash in our CI/CD podman/docker images. We just use FROM scratch, with go, and that's it. The only exception being when we need to compile/rebuild our ebpf kernel modules. To me, personally, the Github Actions CVE from August 2024 was the final nail in the coffin. I blogged about it in more technical detail [1] and guess what was the reason that the TJ actions have been compromised last week? Yep, you guessed right, the same attack surface that Github refuses to fix, a year later. The only tool, as far as I know, that somehow validates against these kind of vulnerabilities, is zizmor [2]. All other tools validate schemas, not vulnerabilities and weaknesses. [1] https://cookie.engineer/weblog/articles/malware-insights-github-actions-script-injection.html https://cookie.engineer/weblog/articles/malware-insights-git... [2] https://github.com/woodruffw/zizmor https://github.com/woodruffw/zizmor
- pcthrowaway 2y ago
  My years using Concourse were a dream compared to the CI/CD pains of trying to make github actions work (which I fortunately didn't have to do a lot of). Add that to the list of options for people who want open source and their own runners
  - regularfry 2y ago
    One of the very few CI platforms that I've heard spoken well of was a big shared Concourse instance where the entire pipeline was predefined. You added some scripts named by convention to your project to do the right thing at each step, and it all just worked for you. Keeping it running was the job of a specific team.
- sleepybrett 2y ago
Kwpolska 2y ago
There are tradeoffs to that. If your CI logic is in shell scripts, you will probably get worse error reporting than the dedicated tasks from the CI tool (which hook into the build system, or which know how to parse logs).
valenterry 2y ago
Amen. To put the first 3 points into different words: you should treat the CI only as a tool that manages the interface and provides interaction with the outside world (including injecting secrets/configuration, setting triggers, storing caches etc.) and helps to visualize things. Unfortunately, to do that, it puts constraints on how you can use it. Apart from that, no logic should live in the CI.
stinos 2y ago
This. I don't know which guru came up with it but this is the 'one-click build' principle. If youcan't do that, you have a problem. So if even remotely possible we write all CI as a single 'one-click' script which can do it all by itself. Makes developing/testing the whole CI easy. Makes changing between CI implementations easy. Can solve really nasty issues (think: CI is down, need to send update to customer) easily because if you want a release you just build it locally. The only thing it won't automaticaly do out of the box is being fast, because obviously this script also needs to setup most of the build environment. So depending on the exact implementation there's variation in the split between what constitutes setting up a build environment and running the CI script. As in: for some tools our CI scripts will do 'everything' so starting from a minimal OS install. Whereas others expect an OS with build tools and possibly some dependencies already available.
- xyzal 2y ago
  I think it was mentioned as a part of the 'Joel test' https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-steps-to-better-code/ https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-s...
  - tailspin2019 2y ago
    25 years later and we’re still having to relearn some of his lessons!
    - cruffle_duffle 2y ago
      At least we generally aren’t fighting “use source control”. Maybe the VCS used by the shop is dogshit but it’s better than nothing!
  - stinos 2y ago
    Yeah spot on, this was definitley it. I now remember reading this probably right after it came out and being somewhat proud to be able to tick most stuff of the list without ever being told directly to do so. But the 'Can you make a build in one step?' was not one of them so I figured that since the rest of the list made so much sense, I'd better get started on that one as well. I also really like that most of this list is practical, low-level advice. No 'use tech X' or 'agile ftw', just basic stuff which automatically happens anyway if you'd opt to use tech X or agile - should those be the right tools for the job, but which would cause more friction if not.
crabbone 2y ago
First of all, I cannot agree more, given what we have today. Unfortunately, this isn't a good plan going forward... :( Going forward I'd wish for a tool that's as ubiquitous as Git, has good integration with editors like language servers, can be sold as a service or run completely in-house. And it would allow defining the actions of the automated builds and tests, have a way of dealing with releases, expose interface for collecting statistics, integrate with bug tracing software for the purpose of excluding / including tests in test runs, allowed organizing tests in groups (eg. sanity / nightly / rc). The problem is that tools today don't come anywhere close to being what I want for CI, neither free nor commercial tools aren't even going in the desired direction. So, the best option is simply to minimize their use.
speleding 2y ago
I would like to add one point: * Consider whether it's not easier to do away with CI in the cloud and just build locally on the dev's laptop With fast laptops and Docker you can get perfectly reproducible builds and tests locally that are infinitely easier to debug. It works for us.
- claytonjy 2y ago
  How do you ensure what a dev builds and tags and pushes is coherent, meaning the tag matches the code commit it’s expected to? I think builds must be possible locally, but i’d never rely on devs for the source of truth artifacts running in production, past a super early startup.
  - speleding 2y ago
    You can add all kind of verification scripts to git hooks, that trigger before and after someone pushes, like you do with GitHub actions. Whether you trust you devs less than your build pipeline is an organizational issue, but in our org only a few senior devs can merge to master.
doix 2y ago
I came from the semiconductor industry, where everything was locally hosted Jenkins + bash scripts. The Jenkins job would just launch the bash script that was stored in perforce(vcs), so all you had to do to run things locally was run the same bash script. When I joined my first web SaaS startup I had a bit of a culture shock. Everything was running on 3rd party services with their own proprietary config/language/etc. The base knowledge of POSIX/Linux/whatever was almost completely useless. I'm kinda used to it now, but I'm not convinced it's any better. There are so many layers of abstraction now that I'm not sure anybody truly understands it all.
- nsonha 2y ago
  it's just common sense, which is unfortunately lost with sloppy devs. People go straight from junior dev to SRE without learning engineering principles through building products first.
- Xcelerate 2y ago
  Haha, I had the same experience going from scientific work in grad school to big tech. The phrase “a solution in search of a problem” comes to mind. The additional complexity does create new problems however, which is fine for devops, because now we have a recursive system of ensuring job security. It blows my mind what is involved in creating a simple web app nowadays compared to when I was a kid in the mid-2000s. Do kids even do that nowadays? I’m not sure I’d even want to get started with all the complexity involved.
  - deleted 2y ago
    [deleted]
  - DrFalkyn 2y ago
    Creating a simple web app isn’t that hard. If you want to use a framework The React tutorials from Traversy media are pretty good. You can even do cross platform into mobile app with frameworks like React Native or Flutter if you want iOS/Android native apps. Vite has been a godsend for React/Vue. It’s no longer the circus it was in the mid 2010s. Google’s monopoly has made things easier for web devs. No more babel or polyfill or createReactApp. People do still avoid frameworks and use raw HTML/CSS/Javascript. HTMX has made sever fetches a lot easier. You probably want a decent CSS framework for reponsive design. Everyone used to use minimalist ones like Tailwimd have become more popular. If you need a backend and want to do something simple you can use BaaS (Backend as a service) platforms like Firebase. Otherwise setting up a NodeJS server with some SQL or KV store like SQLLite or MongoDB isn’t too difficult CI/CD systems exist to streamline testing and deployment for large complex apps. But for individual hobbyist projects it’s not worth it.
makeitdouble 2y ago
> * Invest time that your pipelines can run locally on a developer machine as well (as much as possible at least), otherwise testing/debugging pipelines becomes a nightmare. Yes, a thousand time. Deploy scripts are tougher to deal with, as they'll naturally rely on a flurry of environment variables, protected credentials etc. But for everything else writing the script for local execution first, and generalizating them for CI one they run well enough is the absolute best approach. It doesn't even need to run in the local shell, having all the CI stuff in a dedicated docker image is fine if it requires specific libraries or env.
jpgvm 2y ago
This is the way. My personal favourite solution is Bazel specifically because it can be so isolated from those layers. No need for Docker (or Docker in Docker as many of these solutions end up requiring) or other exotic stuff, can produce OCI image artifacts with `rules_oci` directly. By requiring so little of the runner you really don't care for runner features, you can then restrict your CI/CD runner selection to just reliability, cost, performance and ease of integration.
20thr 2y ago
I spend a lot of time in CI (building https://namespace.so https://namespace.so) and I agree with most of this: - Treat pipelines as code. - Make pipelines parts composable, as code. - Be mindful of vendor lock-in and/or lack of portability (it is a trade-off). For on-promise: if you're already deeply invested in running your own infrastructure, that seems like a good fit. When thinking about how we build Namespace -- there are parts that are so important that we just build and run internally; and there are others where we find that the products in the market just bring a tremendous amount of value beyond self-hosting (Honeycomb is a prime example). Use the tools that work best for you.
julienEar 2y ago
Completely agree. Keeping CI logic in actual code instead of YAML is a lifesaver. The GitHub Actions security issues just reinforce why self-hosted runners are the way to go.
cnotv 2y ago
Once, a reliable and wise colleague told me "Use in CI what you use locally" and that has been the best devop advice that never failed me to save my time. The second one has been, from someone else: if you can use anything else than bash, do that.
- amtamt 2y ago
  Try brainfuck... Jokes aside... it's so trendy to bash bash that it's not funny anymore. Bash is still quite reliable for work that usually gets done in CI, and nearly maintenance free if used well.
  - balls187 2y ago
    These days, AI Copilots are quite good at helping write and maintain bash (and shell) scripts, that it's not much of a prolem.
  - badmintonbaseba 2y ago
    I prefer python there, although we do test/deploy on Windows too, so it's nice to have a common python script for Windows and Linux CI. Not interested in making bash work on Windows or scripting in powershell. And although it's a lot more awkward to use python than bash for invoking subprocesses, it's nicer in most other ways.
agumonkey 2y ago
seconded, it was great to leverage hosted cicd at work, until we realized that local testing would now be handled differently.. as always, enough decoupling is useful
hi_hi 2y ago
I came to the exact same conclusion accidentally in my first role as a Tech Lead a few years back. It was a large enterprise CMS project. The client had previously told everyone they couldn't automate deployments due to the hosted platform security, so deployments of code and configs were all done manually by a specific support engineer following a complex multistep run sheet. That was going about as well as you'd expect. I first solved my own headaches by creating a bunch of bash scripts to package and deploy to my local server. Then I shared that with the squads to solve their headaches. Once the bugs were ironed out, the scripts were updated to deploy from local to the dev instance. Jenkins was then brought in an quickly setup to use the same bash scripts, so now we had full CI/CD working to dev and test. Then the platform support guy got bored manually following the run sheet approach and started using our (now mature) scripts to automate deployments to stage and prod. By the time the client found out I'd completely ignored their direction they were over the moon because we had repeatable and error free automated deployments from local all the way up to prod. I was quite proud of that piece of gorilla consulting :-)
- badloginagain 2y ago
  I hate the fact that CI peaked with Jenkins. I hate Jenkins, I hate Groovy, but for every company I've worked for there's been a 6-year-uptime Jenkins instance casually holding up the entire company. There's probably a lesson in there.
  - mike_hearn 2y ago
    It peaked with Jenkins? I'm curious which CI platforms you've used. I swear by TeamCity. It doesn't seem to have any of these problems other people are facing with GitHub Actions. You can configure it with a GUI, or in XML, or using a type safe Kotlin DSL. These all actually interact so you can 'patch' a config via the GUI even if the system is configured via code, and TeamCity knows how to store config in a git repository and make commits when changes are made, which is great for quick things where it's not worth looking up the DSL docs or for experimentation. The UI is clean and intuitive. It has all the features you'd need. It scales. It isn't riddled with insecure patterns like GH Actions is.
gchamonlive 2y ago
The way that gitlab shines is just fundamentally better than GitHub actions. It's really easy to extend and compose jobs, so it's simple to unit test your pipeline: https://gitlab.com/nunet/test-suite/-/tree/main/cicd/tests?ref_type=heads https://gitlab.com/nunet/test-suite/-/tree/main/cicd/tests?r... This way I can code my pipeline and use the same infrastructure to isolate groups of jobs that compose a relevant functionality and test it in isolation to the rest of the pipeline. I just wish components didn't have such a rigid opinion on folder structure, because they are really powerful, but you have to adopt gitlab prescription
DanielHB 2y ago
Man I tried this approach by making my builds dockerized, turns out docker layer caching is pretty slow on CI and adds a lot of overhead locally. Do not recommend this approach (of using docker for building).
- adra 2y ago
  Make builds in docker by mounting volumes and have your sources, intermediate files, caches, etc. in these volume mounts. Building a bunch of intermediate or incremental data IN the container every time you execute a new partial compile is insanity. It's very satisfying just compile an application with a super esoteric tool chain in docker vs the nightmares of setting it up locally (and keeping it working over time).
  - DanielHB 2y ago
    I had a project that had to build for macos, linux and windows on armv7, armv8 and x64 (and there were some talks about mips too). Just setting up all the stuff required to compile for all these target archs was a nightmare. We used a single huge docker image with all the dependencies we needed to cross compile to all architectures. The image was around 1GB, it did its job but it was super slow on CI to pull it.
no_wizard 2y ago
Am I an outlier in that not only do I find GitHub actions pleasant to use, but that most folks over complicate their CI/CD pipelines? I've had to re-write alot of actions configurations over the last few years, and in every case, the issue was simply not thinking through the limits of the platform, or when things would be better to run as custom docker images (which you can do via GitHub Actions) etc. It tends to be that folks want to shoehorn some technology into the pipeline that doesn't really fit, or they make these giant one shot configurations instead of running multiple small parallel jobs by setting up different configurations for different concerns etc.
- davidham 2y ago
  I'm with you! I kind of love GitHub Actions, and as long as I keep it to tools and actions I understand, I think it works great. It's super flexible and has many event hooks. It's reasonably easy to get it to do the things I want. And my current company has a pretty robust CI suite that catches most problems before they get merged in. It's my favorite of the CI platforms I have used.
ed_elliott_asc 2y ago
* print out the working directory and a directory listing every time
- 12_throw_away 2y ago
  And the environment! (Also, don't put secrets in environment vars)
otikik 2y ago
> * Avoid YAML as much as possible, period. That's also a very valid takeaway for life in general
LukaD 2y ago
> [...] use (shell scripts, make, just, doit, mage, whatever) as long as it is proper, maintainable code I fully agree with the recommendation to use maintainable code. But that effectively rules out shell scripts in my oppinion. CI shell scripts tend to become big ball of mud rather quickly as you run into the limitations of bash. I think most devs only have superficial knowledge of shell scripts, so do yourself a favor and skip them and go straight to whatever language your team is comfortable with.
- sgarland 2y ago
  Maybe people should get better at shell, instead. Read the bash / zsh manual. Use ShellCheck.
  - int_19h 2y ago
    Shellcode is just a horrible PL, period. Not only it's weird and unlike anything else out there, there's way too many footguns. One can learn to use it to the point where it's usable to do advanced automation... but why, when there are so many better options available?
    - sgarland 2y ago
      Because it’s never going away, and it’s always going to be there. It is the lowest common denominator. Also, a shell script generally doesn’t have any other dependencies (modulo writing one that calls jq or something). No risk of solver hell.
jordanbeiber 2y ago
We’ve gone full-on full-code. Although we’re using temporal to schedule the workflows, we have a full-code typescript CI/CD setup. We’ve been through them all starting with Jenkins ending with drone, until we realized that full-code makes it so much easier to maintain and share the work over the whole dev org. No more yaml, code generating yaml, product quirk, groovy or DSLs!
deleted 2y ago
[deleted]
philistine 2y ago
> * Avoid YAML as much as possible, period. Why does YAML have any traction when JSON is right there? I'm an idiot amateur and even I learned this lesson; my 1 MB YAML file full of data took 15 seconds to parse each time. I quickly learned to use JSON instead, takes half a second.
- sofixa 2y ago
  Because YAML, as much as it sucks, is relatively straightforward to write by humans. It sucks to read and parse, you can make tons of small mistakes that screw it up entirely, but it's still less cruft than tons of needless "": { } . For short configs, YAML is acceptable-ish. For anything longer I'd take TOML or something else.
- 12_throw_away 2y ago
  > Why does YAML have any traction when JSON is right there? Because it has comments, which are utterly essential for anything used as a human readable/writable configuration file format (your use case, with 1 MB of data, needs a data interchange format, for which yes JSON is at least much better than YAML).
- fireflash38 2y ago
  JSON is valid YAML. YAML has comments. YAML is easily & trivially written by humans. JSON is easily & trivially written by code. My lesson learned here? When generating YAML, instead generate JSON. If it's meant to be read and updated by humans, use something that can communicate to the humans (comments). And don't use YAML as a data interchange format.
djha-skin 2y ago
I couldn't agree more, really. My whole career points to this as the absolute correct advice in CI.
amedvednikov 2y ago
We recently migrated from YAML CI to VSH as well: https://github.com/vlang/v/blob/master/ci/linux_ci.vsh https://github.com/vlang/v/blob/master/ci/linux_ci.vsh
amadio 2y ago
I think this is good advice overall. I wrote a CMake script that does most of the heavy lifting for XRootD (see https://news.ycombinator.com/item?id=39657703 https://news.ycombinator.com/item?id=39657703). The CI is then a couple of lines, one to install the dependencies using the packaging tools, and another one calling that script. So don't underestimate the convenience that packaging can give you when installing dependencies.
bob1029 2y ago
> Write as much CI logic as possible in your own code This has been my entire strategy since I've been able to do this: https://learn.microsoft.com/en-us/dotnet/core/deploying/#publish-self-contained https://learn.microsoft.com/en-us/dotnet/core/deploying/#pub... Pulling the latest from git, running "dotnet build" and sending the artifacts to zip/S3 is now much easier than setting up and managing Jenkins, et. al. You also get the benefit of having 100% of your CI/CD pipeline under source control alongside the product. In my last professional application of this (B2B/SaaS; customer hosts on-prem), we didn't even have to write the deployment piece. All we needed to do was email the S3 zip link to the customer and they learned a quick procedure to extract it on the server each time.
- ptx 2y ago
  > All we needed to do was email the S3 zip link to the customer and they learned a quick procedure to extract it on the server each time. My concern with this kind of deployment solution, where the customer is instructed to install software from links received in e-mails, is that someone else could very easily send them a link to a malicious installer and they would be hosed. E-mail is not authenticated (usually) and the sender can be forged. I suppose you could use a shared OneDrive folder or something, which would be safer, as long as the customer doesn't rely on receiving the link to OneDrive by e-mail.
Tainnor 2y ago
> Write as much CI logic as possible in your own code. Does not really matter what you use (shell scripts, make, just, doit, mage, whatever) as long as it is proper, maintainable code. To an extent, yes. There should be one command to build, one to run tests, etc. But in many cases, you do actually want the pipeline functionality that something like Gitlab CI offers - having multiple jobs instead of a single one has many benefits (better/shorter retry behaviour, parallelisation, manual triggers, caching, reacting to specific repository hooks, running subsets of tests depending on the changed files, secrets in env vars, artifact publishing, etc.). It's at this point that it becomes almost unavoidable to use many of the configuration features including branching statements, job dependencies etc. and that's where it gets messy. The problem is really that you're forced to do all of that in YAML instead of an actual programming language.
okayishdefaults 2y ago
[dead]
neves 2y ago
How AWS Code Builder compares? I'm delving into AWS world now.
mikepurvis 2y ago
Strongly isolated systems like Nix and Bazel are amazing for giving no-fuss local reproducibility. Every CI "platform" is trying to seduce you into breaking things out into steps so that you can see their little visualizations of what's running in parallel or write special logic in groovy or JS to talk to an API and generate notifications or badges or whatever on the build page. All of that is cute, but it's ultimately the tail wagging the dog— the underlying build tool should be what is managing and ordering the build, not the GUI. What I'd really like for next gen CI is a system that can get deep hooks into local-first tools. Don't make me define a bunch of "steps" for you to run, instead talk to my build tool and just display for me what the build tool is doing. Show me the order of things it built, show me the individual logs of everything it did. Same thing with test runners. How are we still stuck in a world where the test runner has its own totally opaque parallelism regime and our only insight is whatever it chooses to dump into XML at the end, which will be probably be nothing if the test executable crashes? Why can't the test runner tell the CI system what all the processes are that it forked off and where each one's respective log file and exit status is expected to be?
- nand_gate 2y ago
  Why would you need extra visualisation anyway, tooling like Nix is already what you see is what you get!
  - jkarni 2y ago
    It’s still helpful to eg fold different phases in Nix, and different derivation output. I work on garnix.io, which is exactly a Nix-based CI alternative for GitHub, and we had to build a lot of these small things to make the experience better.
- steeleduncan 2y ago
  > Write as much CI logic as possible in your own code Nix really helps with this. Its not just that you do everything via a single script invocation, local or ci, you do it in an identical environment, local or ci. You are not trying to debug the difference between Ubuntu as setup in GHA or Arch as it is on your laptop. Setting up a nix build cache also means that any artefact built by your CI is instantly available locally which can speed up some workflows a lot.
rbongers 2y ago
In my opinion, unless if you need its ability to figure out when something should rebuild or potentially if you already use it, Make is not the right tool for the job. You should capture your pipeline jobs in scripts or similar, but Make just adds another language for developers to learn on top of everything. Make is not a simple script runner. I maintained a Javascript project that used Make and it just turned into a mess. We simply changed all of our `make some-job` jobs into `./scripts/some-job.sh` and not only was the code much nicer, less experienced developers were suddenly more comfortable making changes to scripts. We didn't really need Make to figure out when to rebuild anything, all of our tools already had caching.
- DanHulton 2y ago
  This is why I've become a huge fan of Just, which is just a command runner, not a build caching system or anything. It allows you to define a central interface into your project (largely what I find people justify using Make for), but smoothes out so many of the weird little bumps you run into from "using Make wrong." Plus, you can an any point just drop into running a script in a different language as your command, so it basically "supports bash scripts" too. https://github.com/casey/just https://github.com/casey/just
- JanMa 2y ago
  Make is definitely just my personal preference. If using bash scripts, Just, Taskfile or something similar works better for you then by all means use it. The main argument I wanted to make is that it works very well to just use GitHub actions to execute your tool of choice.
psyclobe 2y ago
'* Write as much CI logic as possible in your own code. Does not really matter what you use (shell scripts, make, just, doit, mage, whatever) as long as it is proper, maintainable code.' THIS 10000% percent.
outofpaper 2y ago
Agree with everything except for the avoidance of YAML. What is your rationale for this?
WhyNotHugo 2y ago
If CI just installs some packages and runs `make check` (or something close), then it's going to be much much easier for others to run checks locally.
Ancalagon 2y ago
+1 for avoiding YAML at all costs Also lol @deng
fahhem 2y ago
Why use your own runners? If it's about cost, why not use a cheaper cloud like SonicInfra.com?
jiehong 2y ago
I always thought it could be cool to use systemd as a CI agent replacement someday: Each systemd service could represent a step built by running a script, and each service can say what it depends on, thus helping parallelize any step that can be. I have not found anyone trying that so far. Is anybody aware of something similar and more POSIX/cross platform that allows writing a DAG of scripts to execute?
iloveitaly 2y ago
I very much agree here. I've had the best luck when there is as little as possible config in CI as possible: - mise for lang config - direnv for environment loading - op for secret injection - justfile for lint, build, etc Here's a template repo that I've been working on that has all of this implemented: https://github.com/iloveitaly/python-starter-template https://github.com/iloveitaly/python-starter-template It's more complex than I would like it to be, but it's consistent and avoids having to deal with GHA too much. I've also found having a GHA playground is helpful: https://github.com/iloveitaly/github-action-playground https://github.com/iloveitaly/github-action-playground
totaldude87 2y ago
Gitlab's search just sucks..
julik 2y ago
Seconded. Moreover... > as long as it is proper, maintainable code ...in an imperative language you know well and which has a sufficient amount of type/null checking you can tolerate.
carlmr 2y ago
>Invest time that your pipelines can run locally on a developer machine as well (as much as possible at least), otherwise testing/debugging pipelines becomes a nightmare. This so much. This ties into the previous point about using as much shell as possible. Additionally I'd say environment control via Docker/Nix, as well as modularizing the pipeline so you can restart it just before the point of failure instead of rerunning the whole business just to replay one little failure.
deleted 2y ago
[deleted]
wvh 2y ago
I like the premise of something like Dagger, being an Actions CI that can run locally and uses Docker. I don't know if there's an up-and-coming "safe" open-source alternative that does not have that threat of a VC time bomb hanging over it. Docker and to some extent, however unwieldy, Kubernetes at least allow you to run anywhere, anytime without locking you into a third party. A "pipeline language" that can run anywhere, even locally, sets up dependency services and initial data, runs tests and avoids YAML overload would be a much needed addition to software engineering.
gabyx 2y ago
Ohh, @deng, my exact words and you are 100% right. Same experience, same conclusion: - I would go even further: Do not use bash/python or any duck-typed lang. (only for simple projects, but better just dont get started). - Leverage Nix (!! no its not a joke ecosystem) : devshells or/and build devcontainers out of it. - Treat tooling code, ci code, the exact same as your other code. - Maybe generate the pipeline for your YAML based CI system in code. - If you use a CI system, gitlab, circle etc, use one which does not do stupid things with your containers (like Github: 4 years! old f** up: https://github.com/actions/runner/issues/863#issuecomment-2591932159 https://github.com/actions/runner/issues/863#issuecomment-25...) Thats why we built our own build tool which does that, or at least helps us doing the above things: https://github.com/sdsc-ordes/quitsh https://github.com/sdsc-ordes/quitsh
gabyxgabyx 2y ago
Ohh, @deng, my exact words and you are 100% right. Same experience, same conclusion: - I would go even further: Do not use bash/python or any duck-typed lang. (only for simple projects, but better just dont get started). - Leverage Nix (!! no its not a joke ecosystem) : devshells or/and build devcontainers out of it. - Treat tooling code, ci code, the exact same as your other code. - Maybe generate the pipeline for your YAML based CI system in code. - If you use a CI system, gitlab, circle etc, use one which does not do stupid things with your containers (like Github: 4 years! old f** up: https://github.com/actions/runner/issues/863#issuecomment-2591932159 https://github.com/actions/runner/issues/863#issuecomment-25...). Also one which lets you run dynamically generated pipelines. Thats why we built our own build tool which does that, or at least helps us doing the above things: https://github.com/sdsc-ordes/quitsh https://github.com/sdsc-ordes/quitsh

by deng 2y ago

Aeolun 2y ago

This is where I was going to say something about dagger, but it seems it turned into AI crud. Let me at least recommend depot.dev for having absurdly fast runners.

oulipo 2y ago
can you give more feedback about dagger? what is good/not good about it? I was going to start looking into it
- toastal 2y ago
  For starts looking at their website, it looks like all collaboration is locked behind proprietary platforms… Discord, Twitter, LinkedIn, Microsoft GitHub.
- Aeolun 2y ago
  I liked their setup before, though I never got around to actually using it, but the tagline on the website has changed to “AI powered workflow orchestration”, which is quite different from the original “Write pipeline once, run everywhere”
  - oulipo 2y ago
    yes but I went on their Discord, and the AI thing is more an "extension" of the CI/CD thing, it's just marketing-speak for the investors, they are still building the CI/CD tool
- esafak 2y ago
  Look into it; it works, more or less.
shykes 2y ago
Hello! Dagger CEO here. We are indeed getting an influx of AI workloads (AI agents to be specific, which is the fancy industry term for "software with LLMs inside"), and are of course trying to capitalize on that in our marketing material. We're still looking for the right balance of CI and AI on our website. Crucially, it's the same engine running both. Because, as it turns out, AI agents are mostly workflows under the hood, and Dagger is great at running those. I shared more context in this thread: https://x.com/solomonstre/status/1895671390176747682 https://x.com/solomonstre/status/1895671390176747682

alex_suzuki 2y ago

> * Always use your own runners, on-premise if possible Why? I understand it in cases where security is critical or intellectual property is at stake. Are you talking about "snowflake runners" or just dumb executors of container images?

deleted 2y ago
[deleted]
deng 2y ago
It obviously depends on your load. Fast pipelines matter, so don't run them on some weak cloud runner with the speed of a C64. Fast cloud runners are expensive. Just invest some money and buy or at least rent some beefy servers with lots of cores, RAM and storage and never look back. Use caches for everything to speed up things. Security is another thing where this can come in handy, but properly firewalling CI runners and having mirrors of all your dependencies is a lot of work and might very well be overkill for most people.
- zoobab 2y ago
  "Fast cloud runners are expensive." Buy a cheap Ryzen, and put it on your desk, that's a cheap runner.
  - withinboredom 2y ago
    30 bucks a month on hetzner for a dedicated machine with 12-16 cores and 64 gb of ram and unlimited 1gbps bandwidth.
saidinesh5 2y ago
Caching is nicer on own runners. No need to redownload 10+GB of "development container images" just to build your 10 lines of changed code. With self hosted Gitlab runners it was almost as fast as doing incremental builds. When your build process can take like 15-20 minutes (medium sized C++ code base), this brought down the total time to 30 seconds or so.

ehansdais 2y ago

After years of trial and error our team has come to the same conclusion. I know some people might consider this insanity, but we actually run all of our scripts as a separate C# CLI application (The main application is a C# web server). Effectively no bash scripts, except as the entry point here and there. The build step and passing the executable around is a small price to pay for the gain in static type checking, being able to pull in libraries as needed, and knowing that our CI is not going to down because someone made a dumb typo somewhere. The other thing I would add is consider passing in all environment variables as args. This makes it easy to see what dependencies the script actually needs, and has the bonus of being even more portable.

baq 2y ago
> I know some people might consider this insanity Some people here still can’t believe YAML is used for not only configuration, but complex code like optimized CI pipelines. This is insane. You’re actually introducing much needed sanity into the process by admitting that a real programming language is the tool to use here. I can’t imagine the cognitive dissonance Lisp folks have when dealing with this madness, not being one myself.
- mschuster91 2y ago
  > Some people here still can’t believe YAML is used for not only configuration, but complex code like optimized CI pipelines. This is insane. It's miles better than Jenkins and the horrors people created there. GitLab CI can at least be easily migrated to any other GitLab instance and stuff should Just Work because it is in the end not much more than self contained bash scripts, but Jenkins... is a clown show, especially for Ops people of larger instances. On one side, you got 50 plugins with CVEs but you can't update them because you need to find a slot that works for all development teams to have a week or two to fix their pipelines again, and on the other side you got a Jenkins instance for each project which lessens the coordination effort but you gotta worry about dozens of Jenkins instances. Oh and that doesn't include the fact many old pipelines aren't written in Groovy or, in fact, in any code at all but only in Jenkins's UI... Github Actions however, I'd say for someone coming from GitLab, is even worse to work with than Jenkins.

JanMa 2y ago

Whenever possible I now just use GitHub actions as a thin wrapper around a Makefile and this has improved my experience with it a lot. The Makefile takes care of installing all necessary dependencies and runs the relevant build/Test commands. This also enables me to test that stuff locally again without the long feedback loop mentioned in other comments in this thread.

oulipo 2y ago
mise (https://mise.jdx.dev/ https://mise.jdx.dev/) and dagger (https://github.com/dagger/dagger https://github.com/dagger/dagger) seem like nice candidates too! Mise can install all your deps, and run tasks
- lou1306 2y ago
  > * Don't bind yourself to some fancy new VC-financed thing that will solve CI once and for all but needs to get monetized eventually (see: earthly, dagger, etc.) Literally from comment at the root of this thread.
  - triyambakam 2y ago
    But are mise and dagger VC funded? I don't see any pricing pages there.
    - deleted 2y ago
      [deleted]
    - deng 2y ago
      Dagger is even YCombinator funded. https://www.boringbusinessnerd.com/startups/dagger https://www.boringbusinessnerd.com/startups/dagger Mise indeed isn't, but its scope is quite a bit smaller than Dagger.
      - internetter 2y ago
        > Mise indeed isn't, but its scope is quite a bit smaller than Dagger. A lot of us could learn... do one thing and do it well

cesnja 2y ago

You can build the first pipeline with oneliners, but as long as you want to keep optimizing the pipelines, the yaml code will keep piling up with CI vendor's specific approaches to job selection, env variable delivery, caching, output sharing between jobs and so on.

forrestthewoods 2y ago

> pipelines can run locally on a developer machine as well (as much as possible at least) Facts. However I’ll go a step further and say “only implement your logic in a tool that has a debugger”. YAML is the worse. But shell scripts are second worst. Use a real language.

never_inline 2y ago
Python with click and PyYAML can go a long way - then you can build it as a CLI application and use the same from CI. In a java shop, picocli + graalvm probably. I wouldn't like Go for this purpose (against the conventional wisdom - because boilerplate and pretty bad debugging capabilities). That said, if you absolutely need to use shell script for reasons, keep it all in single script, define logging functions including debug logs, rigorously check every constraint and variable, use shellcheck, factor the code well into functions - I should sometimes write a blog post about it.
folmar 2y ago
There is a debugger for bash: https://github.com/Trepan-Debuggers/bashdb https://github.com/Trepan-Debuggers/bashdb Not that I'm recommending 10k-line programs in bash, but a debugger is useful when you need it.
sgarland 2y ago
Shell is a very real language, and it has a debugger; it’s called set -x and/or strace.
- forrestthewoods 2y ago
  Like I said, use something with a real debugger.

ozim 2y ago

Good insight, because that is just a complex issue - especially when there is team churn and everyone adds their parts in yaml or configuration. Doesn’t matter Jenkins or actions - it is just complicated. Making it simpler is on devs/ops not the tool.

qweiopqweiop 2y ago

Can you explain YAML? I've found declarative pipelines with it have been... fine?

fergie 2y ago
As a developer based in Norway, one fairly major drawback to YAML is the way that it processes the language code for Norwegian ("no").
- marcusramberg 2y ago
  That particular pain was fixed in yaml 1.2 :)
  - withinboredom 2y ago
    TIL I learned that yaml has versions and now I'm wondering what version of yaml parsers are running where. I think I might be on a new level of hell. It would be better to delete your comment so nobody else has to has to ever have this crisis.
- LeonM 2y ago
  The Norwegian ISO3166 code colliding with the English word 'no' is not a YAML problem per se, I've been bitten by that a few times in other situations as well. For example: Stripe uses constants for types of tax registration numbers (VAT/GST/TIN, etc.). So there is EU_VAT for European VAT numbers, US_TIN for US tax identification numbers, etc. But what value to use for tax-exempt organisations that don't have a tax number? Well... guess how I found out about NO_VAT... On the bright side, I did learn that way that although Norway is in the Schengen zone, apparently they are not part of the EU (hence the separation of EU_VAT and NO_VAT). I guess the 'no' name collision has taught many developers something about Norway :-)
dharmab 2y ago
1. The YAML spec is extremely complex with some things being ambiguous. You might not notice this if your restrict yourself to a small subset of the language. But you will notice it when different YAML libraries and programming languages interpret the same YAML file as different content. 2. Trying to encode logic and control flow in a YAML document is much more difficult than writing that flow in a "real" programming language. Debugging is especially much easier in "real" languages.

cookiengineer 2y ago

This. I heavily invested in a local runner based CI/CD workflow. First I was using gogs and drone, now the forgejo and woodpecker CI forks. It runs with multiple redundancies because it's a pretty easy setup to replicate on decentralized hardware. The only thing that's a little painful is authentication and cross-system pull requests, so we still need our single point of failure to merge feature branches and do code reviews. Due to us building everything in go, we also decided to have always a /toolchain/build.go so that we have everything in a single language, and don't need even bash in our CI/CD podman/docker images. We just use FROM scratch, with go, and that's it. The only exception being when we need to compile/rebuild our ebpf kernel modules. To me, personally, the Github Actions CVE from August 2024 was the final nail in the coffin. I blogged about it in more technical detail [1] and guess what was the reason that the TJ actions have been compromised last week? Yep, you guessed right, the same attack surface that Github refuses to fix, a year later. The only tool, as far as I know, that somehow validates against these kind of vulnerabilities, is zizmor [2]. All other tools validate schemas, not vulnerabilities and weaknesses. [1] https://cookie.engineer/weblog/articles/malware-insights-github-actions-script-injection.html https://cookie.engineer/weblog/articles/malware-insights-git... [2] https://github.com/woodruffw/zizmor https://github.com/woodruffw/zizmor

pcthrowaway 2y ago
My years using Concourse were a dream compared to the CI/CD pains of trying to make github actions work (which I fortunately didn't have to do a lot of). Add that to the list of options for people who want open source and their own runners
- regularfry 2y ago
  One of the very few CI platforms that I've heard spoken well of was a big shared Concourse instance where the entire pipeline was predefined. You added some scripts named by convention to your project to do the right thing at each step, and it all just worked for you. Keeping it running was the job of a specific team.
sleepybrett 2y ago

Kwpolska 2y ago

There are tradeoffs to that. If your CI logic is in shell scripts, you will probably get worse error reporting than the dedicated tasks from the CI tool (which hook into the build system, or which know how to parse logs).

valenterry 2y ago

Amen. To put the first 3 points into different words: you should treat the CI only as a tool that manages the interface and provides interaction with the outside world (including injecting secrets/configuration, setting triggers, storing caches etc.) and helps to visualize things. Unfortunately, to do that, it puts constraints on how you can use it. Apart from that, no logic should live in the CI.

stinos 2y ago

This. I don't know which guru came up with it but this is the 'one-click build' principle. If youcan't do that, you have a problem. So if even remotely possible we write all CI as a single 'one-click' script which can do it all by itself. Makes developing/testing the whole CI easy. Makes changing between CI implementations easy. Can solve really nasty issues (think: CI is down, need to send update to customer) easily because if you want a release you just build it locally. The only thing it won't automaticaly do out of the box is being fast, because obviously this script also needs to setup most of the build environment. So depending on the exact implementation there's variation in the split between what constitutes setting up a build environment and running the CI script. As in: for some tools our CI scripts will do 'everything' so starting from a minimal OS install. Whereas others expect an OS with build tools and possibly some dependencies already available.

xyzal 2y ago
I think it was mentioned as a part of the 'Joel test' https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-steps-to-better-code/ https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-s...
- tailspin2019 2y ago
  25 years later and we’re still having to relearn some of his lessons!
  - cruffle_duffle 2y ago
    At least we generally aren’t fighting “use source control”. Maybe the VCS used by the shop is dogshit but it’s better than nothing!
- stinos 2y ago
  Yeah spot on, this was definitley it. I now remember reading this probably right after it came out and being somewhat proud to be able to tick most stuff of the list without ever being told directly to do so. But the 'Can you make a build in one step?' was not one of them so I figured that since the rest of the list made so much sense, I'd better get started on that one as well. I also really like that most of this list is practical, low-level advice. No 'use tech X' or 'agile ftw', just basic stuff which automatically happens anyway if you'd opt to use tech X or agile - should those be the right tools for the job, but which would cause more friction if not.

crabbone 2y ago

First of all, I cannot agree more, given what we have today. Unfortunately, this isn't a good plan going forward... :( Going forward I'd wish for a tool that's as ubiquitous as Git, has good integration with editors like language servers, can be sold as a service or run completely in-house. And it would allow defining the actions of the automated builds and tests, have a way of dealing with releases, expose interface for collecting statistics, integrate with bug tracing software for the purpose of excluding / including tests in test runs, allowed organizing tests in groups (eg. sanity / nightly / rc). The problem is that tools today don't come anywhere close to being what I want for CI, neither free nor commercial tools aren't even going in the desired direction. So, the best option is simply to minimize their use.

speleding 2y ago

I would like to add one point: * Consider whether it's not easier to do away with CI in the cloud and just build locally on the dev's laptop With fast laptops and Docker you can get perfectly reproducible builds and tests locally that are infinitely easier to debug. It works for us.

claytonjy 2y ago
How do you ensure what a dev builds and tags and pushes is coherent, meaning the tag matches the code commit it’s expected to? I think builds must be possible locally, but i’d never rely on devs for the source of truth artifacts running in production, past a super early startup.
- speleding 2y ago
  You can add all kind of verification scripts to git hooks, that trigger before and after someone pushes, like you do with GitHub actions. Whether you trust you devs less than your build pipeline is an organizational issue, but in our org only a few senior devs can merge to master.

doix 2y ago

I came from the semiconductor industry, where everything was locally hosted Jenkins + bash scripts. The Jenkins job would just launch the bash script that was stored in perforce(vcs), so all you had to do to run things locally was run the same bash script. When I joined my first web SaaS startup I had a bit of a culture shock. Everything was running on 3rd party services with their own proprietary config/language/etc. The base knowledge of POSIX/Linux/whatever was almost completely useless. I'm kinda used to it now, but I'm not convinced it's any better. There are so many layers of abstraction now that I'm not sure anybody truly understands it all.

nsonha 2y ago
it's just common sense, which is unfortunately lost with sloppy devs. People go straight from junior dev to SRE without learning engineering principles through building products first.
Xcelerate 2y ago
Haha, I had the same experience going from scientific work in grad school to big tech. The phrase “a solution in search of a problem” comes to mind. The additional complexity does create new problems however, which is fine for devops, because now we have a recursive system of ensuring job security. It blows my mind what is involved in creating a simple web app nowadays compared to when I was a kid in the mid-2000s. Do kids even do that nowadays? I’m not sure I’d even want to get started with all the complexity involved.
- deleted 2y ago
  [deleted]
- DrFalkyn 2y ago
  Creating a simple web app isn’t that hard. If you want to use a framework The React tutorials from Traversy media are pretty good. You can even do cross platform into mobile app with frameworks like React Native or Flutter if you want iOS/Android native apps. Vite has been a godsend for React/Vue. It’s no longer the circus it was in the mid 2010s. Google’s monopoly has made things easier for web devs. No more babel or polyfill or createReactApp. People do still avoid frameworks and use raw HTML/CSS/Javascript. HTMX has made sever fetches a lot easier. You probably want a decent CSS framework for reponsive design. Everyone used to use minimalist ones like Tailwimd have become more popular. If you need a backend and want to do something simple you can use BaaS (Backend as a service) platforms like Firebase. Otherwise setting up a NodeJS server with some SQL or KV store like SQLLite or MongoDB isn’t too difficult CI/CD systems exist to streamline testing and deployment for large complex apps. But for individual hobbyist projects it’s not worth it.

makeitdouble 2y ago

> * Invest time that your pipelines can run locally on a developer machine as well (as much as possible at least), otherwise testing/debugging pipelines becomes a nightmare. Yes, a thousand time. Deploy scripts are tougher to deal with, as they'll naturally rely on a flurry of environment variables, protected credentials etc. But for everything else writing the script for local execution first, and generalizating them for CI one they run well enough is the absolute best approach. It doesn't even need to run in the local shell, having all the CI stuff in a dedicated docker image is fine if it requires specific libraries or env.

jpgvm 2y ago

This is the way. My personal favourite solution is Bazel specifically because it can be so isolated from those layers. No need for Docker (or Docker in Docker as many of these solutions end up requiring) or other exotic stuff, can produce OCI image artifacts with `rules_oci` directly. By requiring so little of the runner you really don't care for runner features, you can then restrict your CI/CD runner selection to just reliability, cost, performance and ease of integration.

20thr 2y ago

I spend a lot of time in CI (building https://namespace.so https://namespace.so) and I agree with most of this: - Treat pipelines as code. - Make pipelines parts composable, as code. - Be mindful of vendor lock-in and/or lack of portability (it is a trade-off). For on-promise: if you're already deeply invested in running your own infrastructure, that seems like a good fit. When thinking about how we build Namespace -- there are parts that are so important that we just build and run internally; and there are others where we find that the products in the market just bring a tremendous amount of value beyond self-hosting (Honeycomb is a prime example). Use the tools that work best for you.

julienEar 2y ago

Completely agree. Keeping CI logic in actual code instead of YAML is a lifesaver. The GitHub Actions security issues just reinforce why self-hosted runners are the way to go.

cnotv 2y ago

Once, a reliable and wise colleague told me "Use in CI what you use locally" and that has been the best devop advice that never failed me to save my time. The second one has been, from someone else: if you can use anything else than bash, do that.

amtamt 2y ago
Try brainfuck... Jokes aside... it's so trendy to bash bash that it's not funny anymore. Bash is still quite reliable for work that usually gets done in CI, and nearly maintenance free if used well.
- balls187 2y ago
  These days, AI Copilots are quite good at helping write and maintain bash (and shell) scripts, that it's not much of a prolem.
- badmintonbaseba 2y ago
  I prefer python there, although we do test/deploy on Windows too, so it's nice to have a common python script for Windows and Linux CI. Not interested in making bash work on Windows or scripting in powershell. And although it's a lot more awkward to use python than bash for invoking subprocesses, it's nicer in most other ways.

agumonkey 2y ago

seconded, it was great to leverage hosted cicd at work, until we realized that local testing would now be handled differently.. as always, enough decoupling is useful

hi_hi 2y ago

I came to the exact same conclusion accidentally in my first role as a Tech Lead a few years back. It was a large enterprise CMS project. The client had previously told everyone they couldn't automate deployments due to the hosted platform security, so deployments of code and configs were all done manually by a specific support engineer following a complex multistep run sheet. That was going about as well as you'd expect. I first solved my own headaches by creating a bunch of bash scripts to package and deploy to my local server. Then I shared that with the squads to solve their headaches. Once the bugs were ironed out, the scripts were updated to deploy from local to the dev instance. Jenkins was then brought in an quickly setup to use the same bash scripts, so now we had full CI/CD working to dev and test. Then the platform support guy got bored manually following the run sheet approach and started using our (now mature) scripts to automate deployments to stage and prod. By the time the client found out I'd completely ignored their direction they were over the moon because we had repeatable and error free automated deployments from local all the way up to prod. I was quite proud of that piece of gorilla consulting :-)

badloginagain 2y ago
I hate the fact that CI peaked with Jenkins. I hate Jenkins, I hate Groovy, but for every company I've worked for there's been a 6-year-uptime Jenkins instance casually holding up the entire company. There's probably a lesson in there.
- mike_hearn 2y ago
  It peaked with Jenkins? I'm curious which CI platforms you've used. I swear by TeamCity. It doesn't seem to have any of these problems other people are facing with GitHub Actions. You can configure it with a GUI, or in XML, or using a type safe Kotlin DSL. These all actually interact so you can 'patch' a config via the GUI even if the system is configured via code, and TeamCity knows how to store config in a git repository and make commits when changes are made, which is great for quick things where it's not worth looking up the DSL docs or for experimentation. The UI is clean and intuitive. It has all the features you'd need. It scales. It isn't riddled with insecure patterns like GH Actions is.

gchamonlive 2y ago

The way that gitlab shines is just fundamentally better than GitHub actions. It's really easy to extend and compose jobs, so it's simple to unit test your pipeline: https://gitlab.com/nunet/test-suite/-/tree/main/cicd/tests?ref_type=heads https://gitlab.com/nunet/test-suite/-/tree/main/cicd/tests?r... This way I can code my pipeline and use the same infrastructure to isolate groups of jobs that compose a relevant functionality and test it in isolation to the rest of the pipeline. I just wish components didn't have such a rigid opinion on folder structure, because they are really powerful, but you have to adopt gitlab prescription

DanielHB 2y ago

Man I tried this approach by making my builds dockerized, turns out docker layer caching is pretty slow on CI and adds a lot of overhead locally. Do not recommend this approach (of using docker for building).

adra 2y ago
Make builds in docker by mounting volumes and have your sources, intermediate files, caches, etc. in these volume mounts. Building a bunch of intermediate or incremental data IN the container every time you execute a new partial compile is insanity. It's very satisfying just compile an application with a super esoteric tool chain in docker vs the nightmares of setting it up locally (and keeping it working over time).
- DanielHB 2y ago
  I had a project that had to build for macos, linux and windows on armv7, armv8 and x64 (and there were some talks about mips too). Just setting up all the stuff required to compile for all these target archs was a nightmare. We used a single huge docker image with all the dependencies we needed to cross compile to all architectures. The image was around 1GB, it did its job but it was super slow on CI to pull it.

no_wizard 2y ago

Am I an outlier in that not only do I find GitHub actions pleasant to use, but that most folks over complicate their CI/CD pipelines? I've had to re-write alot of actions configurations over the last few years, and in every case, the issue was simply not thinking through the limits of the platform, or when things would be better to run as custom docker images (which you can do via GitHub Actions) etc. It tends to be that folks want to shoehorn some technology into the pipeline that doesn't really fit, or they make these giant one shot configurations instead of running multiple small parallel jobs by setting up different configurations for different concerns etc.

davidham 2y ago
I'm with you! I kind of love GitHub Actions, and as long as I keep it to tools and actions I understand, I think it works great. It's super flexible and has many event hooks. It's reasonably easy to get it to do the things I want. And my current company has a pretty robust CI suite that catches most problems before they get merged in. It's my favorite of the CI platforms I have used.

ed_elliott_asc 2y ago

* print out the working directory and a directory listing every time

12_throw_away 2y ago
And the environment! (Also, don't put secrets in environment vars)

otikik 2y ago

> * Avoid YAML as much as possible, period. That's also a very valid takeaway for life in general

LukaD 2y ago

> [...] use (shell scripts, make, just, doit, mage, whatever) as long as it is proper, maintainable code I fully agree with the recommendation to use maintainable code. But that effectively rules out shell scripts in my oppinion. CI shell scripts tend to become big ball of mud rather quickly as you run into the limitations of bash. I think most devs only have superficial knowledge of shell scripts, so do yourself a favor and skip them and go straight to whatever language your team is comfortable with.

sgarland 2y ago
Maybe people should get better at shell, instead. Read the bash / zsh manual. Use ShellCheck.
- int_19h 2y ago
  Shellcode is just a horrible PL, period. Not only it's weird and unlike anything else out there, there's way too many footguns. One can learn to use it to the point where it's usable to do advanced automation... but why, when there are so many better options available?
  - sgarland 2y ago
    Because it’s never going away, and it’s always going to be there. It is the lowest common denominator. Also, a shell script generally doesn’t have any other dependencies (modulo writing one that calls jq or something). No risk of solver hell.

jordanbeiber 2y ago

We’ve gone full-on full-code. Although we’re using temporal to schedule the workflows, we have a full-code typescript CI/CD setup. We’ve been through them all starting with Jenkins ending with drone, until we realized that full-code makes it so much easier to maintain and share the work over the whole dev org. No more yaml, code generating yaml, product quirk, groovy or DSLs!

deleted 2y ago

[deleted]

philistine 2y ago

> * Avoid YAML as much as possible, period. Why does YAML have any traction when JSON is right there? I'm an idiot amateur and even I learned this lesson; my 1 MB YAML file full of data took 15 seconds to parse each time. I quickly learned to use JSON instead, takes half a second.

sofixa 2y ago
Because YAML, as much as it sucks, is relatively straightforward to write by humans. It sucks to read and parse, you can make tons of small mistakes that screw it up entirely, but it's still less cruft than tons of needless "": { } . For short configs, YAML is acceptable-ish. For anything longer I'd take TOML or something else.
12_throw_away 2y ago
> Why does YAML have any traction when JSON is right there? Because it has comments, which are utterly essential for anything used as a human readable/writable configuration file format (your use case, with 1 MB of data, needs a data interchange format, for which yes JSON is at least much better than YAML).
fireflash38 2y ago
JSON is valid YAML. YAML has comments. YAML is easily & trivially written by humans. JSON is easily & trivially written by code. My lesson learned here? When generating YAML, instead generate JSON. If it's meant to be read and updated by humans, use something that can communicate to the humans (comments). And don't use YAML as a data interchange format.

djha-skin 2y ago

I couldn't agree more, really. My whole career points to this as the absolute correct advice in CI.

amedvednikov 2y ago

We recently migrated from YAML CI to VSH as well: https://github.com/vlang/v/blob/master/ci/linux_ci.vsh https://github.com/vlang/v/blob/master/ci/linux_ci.vsh

amadio 2y ago

I think this is good advice overall. I wrote a CMake script that does most of the heavy lifting for XRootD (see https://news.ycombinator.com/item?id=39657703 https://news.ycombinator.com/item?id=39657703). The CI is then a couple of lines, one to install the dependencies using the packaging tools, and another one calling that script. So don't underestimate the convenience that packaging can give you when installing dependencies.

bob1029 2y ago

> Write as much CI logic as possible in your own code This has been my entire strategy since I've been able to do this: https://learn.microsoft.com/en-us/dotnet/core/deploying/#publish-self-contained https://learn.microsoft.com/en-us/dotnet/core/deploying/#pub... Pulling the latest from git, running "dotnet build" and sending the artifacts to zip/S3 is now much easier than setting up and managing Jenkins, et. al. You also get the benefit of having 100% of your CI/CD pipeline under source control alongside the product. In my last professional application of this (B2B/SaaS; customer hosts on-prem), we didn't even have to write the deployment piece. All we needed to do was email the S3 zip link to the customer and they learned a quick procedure to extract it on the server each time.

ptx 2y ago
> All we needed to do was email the S3 zip link to the customer and they learned a quick procedure to extract it on the server each time. My concern with this kind of deployment solution, where the customer is instructed to install software from links received in e-mails, is that someone else could very easily send them a link to a malicious installer and they would be hosed. E-mail is not authenticated (usually) and the sender can be forged. I suppose you could use a shared OneDrive folder or something, which would be safer, as long as the customer doesn't rely on receiving the link to OneDrive by e-mail.

Tainnor 2y ago

> Write as much CI logic as possible in your own code. Does not really matter what you use (shell scripts, make, just, doit, mage, whatever) as long as it is proper, maintainable code. To an extent, yes. There should be one command to build, one to run tests, etc. But in many cases, you do actually want the pipeline functionality that something like Gitlab CI offers - having multiple jobs instead of a single one has many benefits (better/shorter retry behaviour, parallelisation, manual triggers, caching, reacting to specific repository hooks, running subsets of tests depending on the changed files, secrets in env vars, artifact publishing, etc.). It's at this point that it becomes almost unavoidable to use many of the configuration features including branching statements, job dependencies etc. and that's where it gets messy. The problem is really that you're forced to do all of that in YAML instead of an actual programming language.

okayishdefaults 2y ago

[dead]

neves 2y ago

How AWS Code Builder compares? I'm delving into AWS world now.

mikepurvis 2y ago

Strongly isolated systems like Nix and Bazel are amazing for giving no-fuss local reproducibility. Every CI "platform" is trying to seduce you into breaking things out into steps so that you can see their little visualizations of what's running in parallel or write special logic in groovy or JS to talk to an API and generate notifications or badges or whatever on the build page. All of that is cute, but it's ultimately the tail wagging the dog— the underlying build tool should be what is managing and ordering the build, not the GUI. What I'd really like for next gen CI is a system that can get deep hooks into local-first tools. Don't make me define a bunch of "steps" for you to run, instead talk to my build tool and just display for me what the build tool is doing. Show me the order of things it built, show me the individual logs of everything it did. Same thing with test runners. How are we still stuck in a world where the test runner has its own totally opaque parallelism regime and our only insight is whatever it chooses to dump into XML at the end, which will be probably be nothing if the test executable crashes? Why can't the test runner tell the CI system what all the processes are that it forked off and where each one's respective log file and exit status is expected to be?

nand_gate 2y ago
Why would you need extra visualisation anyway, tooling like Nix is already what you see is what you get!
- jkarni 2y ago
  It’s still helpful to eg fold different phases in Nix, and different derivation output. I work on garnix.io, which is exactly a Nix-based CI alternative for GitHub, and we had to build a lot of these small things to make the experience better.
steeleduncan 2y ago
> Write as much CI logic as possible in your own code Nix really helps with this. Its not just that you do everything via a single script invocation, local or ci, you do it in an identical environment, local or ci. You are not trying to debug the difference between Ubuntu as setup in GHA or Arch as it is on your laptop. Setting up a nix build cache also means that any artefact built by your CI is instantly available locally which can speed up some workflows a lot.

rbongers 2y ago

In my opinion, unless if you need its ability to figure out when something should rebuild or potentially if you already use it, Make is not the right tool for the job. You should capture your pipeline jobs in scripts or similar, but Make just adds another language for developers to learn on top of everything. Make is not a simple script runner. I maintained a Javascript project that used Make and it just turned into a mess. We simply changed all of our `make some-job` jobs into `./scripts/some-job.sh` and not only was the code much nicer, less experienced developers were suddenly more comfortable making changes to scripts. We didn't really need Make to figure out when to rebuild anything, all of our tools already had caching.

DanHulton 2y ago
This is why I've become a huge fan of Just, which is just a command runner, not a build caching system or anything. It allows you to define a central interface into your project (largely what I find people justify using Make for), but smoothes out so many of the weird little bumps you run into from "using Make wrong." Plus, you can an any point just drop into running a script in a different language as your command, so it basically "supports bash scripts" too. https://github.com/casey/just https://github.com/casey/just
JanMa 2y ago
Make is definitely just my personal preference. If using bash scripts, Just, Taskfile or something similar works better for you then by all means use it. The main argument I wanted to make is that it works very well to just use GitHub actions to execute your tool of choice.

psyclobe 2y ago

'* Write as much CI logic as possible in your own code. Does not really matter what you use (shell scripts, make, just, doit, mage, whatever) as long as it is proper, maintainable code.' THIS 10000% percent.

outofpaper 2y ago

Agree with everything except for the avoidance of YAML. What is your rationale for this?

WhyNotHugo 2y ago

If CI just installs some packages and runs `make check` (or something close), then it's going to be much much easier for others to run checks locally.

Ancalagon 2y ago

+1 for avoiding YAML at all costs Also lol @deng

fahhem 2y ago

Why use your own runners? If it's about cost, why not use a cheaper cloud like SonicInfra.com?

jiehong 2y ago

I always thought it could be cool to use systemd as a CI agent replacement someday: Each systemd service could represent a step built by running a script, and each service can say what it depends on, thus helping parallelize any step that can be. I have not found anyone trying that so far. Is anybody aware of something similar and more POSIX/cross platform that allows writing a DAG of scripts to execute?

iloveitaly 2y ago

I very much agree here. I've had the best luck when there is as little as possible config in CI as possible: - mise for lang config - direnv for environment loading - op for secret injection - justfile for lint, build, etc Here's a template repo that I've been working on that has all of this implemented: https://github.com/iloveitaly/python-starter-template https://github.com/iloveitaly/python-starter-template It's more complex than I would like it to be, but it's consistent and avoids having to deal with GHA too much. I've also found having a GHA playground is helpful: https://github.com/iloveitaly/github-action-playground https://github.com/iloveitaly/github-action-playground

totaldude87 2y ago

Gitlab's search just sucks..

julik 2y ago

Seconded. Moreover... > as long as it is proper, maintainable code ...in an imperative language you know well and which has a sufficient amount of type/null checking you can tolerate.

carlmr 2y ago

>Invest time that your pipelines can run locally on a developer machine as well (as much as possible at least), otherwise testing/debugging pipelines becomes a nightmare. This so much. This ties into the previous point about using as much shell as possible. Additionally I'd say environment control via Docker/Nix, as well as modularizing the pipeline so you can restart it just before the point of failure instead of rerunning the whole business just to replay one little failure.

deleted 2y ago

[deleted]

wvh 2y ago

I like the premise of something like Dagger, being an Actions CI that can run locally and uses Docker. I don't know if there's an up-and-coming "safe" open-source alternative that does not have that threat of a VC time bomb hanging over it. Docker and to some extent, however unwieldy, Kubernetes at least allow you to run anywhere, anytime without locking you into a third party. A "pipeline language" that can run anywhere, even locally, sets up dependency services and initial data, runs tests and avoids YAML overload would be a much needed addition to software engineering.

gabyx 2y ago

Ohh, @deng, my exact words and you are 100% right. Same experience, same conclusion: - I would go even further: Do not use bash/python or any duck-typed lang. (only for simple projects, but better just dont get started). - Leverage Nix (!! no its not a joke ecosystem) : devshells or/and build devcontainers out of it. - Treat tooling code, ci code, the exact same as your other code. - Maybe generate the pipeline for your YAML based CI system in code. - If you use a CI system, gitlab, circle etc, use one which does not do stupid things with your containers (like Github: 4 years! old f** up: https://github.com/actions/runner/issues/863#issuecomment-2591932159 https://github.com/actions/runner/issues/863#issuecomment-25...) Thats why we built our own build tool which does that, or at least helps us doing the above things: https://github.com/sdsc-ordes/quitsh https://github.com/sdsc-ordes/quitsh

gabyxgabyx 2y ago

Ohh, @deng, my exact words and you are 100% right. Same experience, same conclusion: - I would go even further: Do not use bash/python or any duck-typed lang. (only for simple projects, but better just dont get started). - Leverage Nix (!! no its not a joke ecosystem) : devshells or/and build devcontainers out of it. - Treat tooling code, ci code, the exact same as your other code. - Maybe generate the pipeline for your YAML based CI system in code. - If you use a CI system, gitlab, circle etc, use one which does not do stupid things with your containers (like Github: 4 years! old f** up: https://github.com/actions/runner/issues/863#issuecomment-2591932159 https://github.com/actions/runner/issues/863#issuecomment-25...). Also one which lets you run dynamically generated pipelines. Thats why we built our own build tool which does that, or at least helps us doing the above things: https://github.com/sdsc-ordes/quitsh https://github.com/sdsc-ordes/quitsh