4 ms·

I came to the exact same conclusion accidentally in my first role as a Tech Lead a few years back. It was a large enterprise CMS project. The client had previo

by hi_hi 2y ago

I came to the exact same conclusion accidentally in my first role as a Tech Lead a few years back. It was a large enterprise CMS project. The client had previously told everyone they couldn't automate deployments due to the hosted platform security, so deployments of code and configs were all done manually by a specific support engineer following a complex multistep run sheet. That was going about as well as you'd expect. I first solved my own headaches by creating a bunch of bash scripts to package and deploy to my local server. Then I shared that with the squads to solve their headaches. Once the bugs were ironed out, the scripts were updated to deploy from local to the dev instance. Jenkins was then brought in an quickly setup to use the same bash scripts, so now we had full CI/CD working to dev and test. Then the platform support guy got bored manually following the run sheet approach and started using our (now mature) scripts to automate deployments to stage and prod. By the time the client found out I'd completely ignored their direction they were over the moon because we had repeatable and error free automated deployments from local all the way up to prod. I was quite proud of that piece of gorilla consulting :-)

badloginagain 2y ago
I hate the fact that CI peaked with Jenkins. I hate Jenkins, I hate Groovy, but for every company I've worked for there's been a 6-year-uptime Jenkins instance casually holding up the entire company. There's probably a lesson in there.
- mike_hearn 2y ago
  It peaked with Jenkins? I'm curious which CI platforms you've used. I swear by TeamCity. It doesn't seem to have any of these problems other people are facing with GitHub Actions. You can configure it with a GUI, or in XML, or using a type safe Kotlin DSL. These all actually interact so you can 'patch' a config via the GUI even if the system is configured via code, and TeamCity knows how to store config in a git repository and make commits when changes are made, which is great for quick things where it's not worth looking up the DSL docs or for experimentation. The UI is clean and intuitive. It has all the features you'd need. It scales. It isn't riddled with insecure patterns like GH Actions is.
  - DanielHB 2y ago
    I think people just hate CI set up by other people. I used TeamCity in a job a few years back and I absolutely hated it, however I imagine a lot of my hatred was the way it was set up. CI is just the thing no one wants to deal with, yet everyone wants to just work. And like any code or process, you need engineering to make it good. And like any project, you can't just blame bad tools for crappy results.
    - bluGill 2y ago
      I would feel that way but I've had the misfortune to work with a wide open ci system where any developer could make thanges and one guy did. The locked down system prevents me form some changes I want but in return my builds don't suddenly start failing because some ci option was turned on for everyone.
      - HeavyStorm 2y ago
        I totally prefer to have the ci break from time to time and be able to fix it than having the risk of it being broken and having no way of fixing it
        bluGill 2y ago
        The people who admin our CI system do a good job so it doesn't break, (well it does all the time, but onnetwork type errors not configuration - that is IT's fault not their fault.) The thing I want to change are things that I do in the build system so that it is checked in and previous versions when we need to build them (we are embedded where field failure is expensive so there are typically branches for the current release, next release, and head). This also means anything that can fail on CI can fail on my local system (unless it depends on something like the number of cores on the machine running the build). While the details can be slightly different, how we have CI is how it should be. most developers should have better things to do than worry about how to configure CI.
        2y ago
- skor 2y ago
  well, I got tired of Groovy and found out that using Jenkins with plain bash under source control is just right for us. Runs everywhere, very fast to test/develop and its all easy to change and improve. We build Docker images mostly so ymmv. I have a "port to github actions" ticket in the backlog but I think we're not going to go down that road now.
  - __float 2y ago
    Yeah, I've come back around to this: you do not want "end users" writing Groovy, because the tooling around it is horrible. You'll have to explain the weird CPS transformations, you'll probably end up reading the Jenkins plugins' code, and there's nothing fun down this path.
- k4rli 2y ago
  It's feature complete. Anything more will just be bloat, probably 25% of it could be reduced at least.
- finnthehuman 2y ago
  Jenkins is cron with bells and whistles. The result is a pile of plugins to capture all the dimensions of complexity you are likely to otherwise bury in the shell script but want them easier to point and click at. I'll hate on jenkins with the rest of them, but entropy is gonna grow and Jenkins isn't gonna say "no, you can't do that here". I deal with multiple tools where if tried to make fun about how low the jenkins plugin install starts are, you'd know exactly where I work. Once I've calmed down from working on CI I can appreciate Jenkins' attempts to manage all of it. Any CI product play has to differentiate in a way that makes you dependent on them. Sure it can be superficially nicer when staying inside the guard rails, but in the age of docker why has the number of ways I configure running boring shell scripts gone UP? Because they need me unable to use a lunch break to say "fuck you I don't need the integrations you reserve exclusively for your CI" and port all the jobs back to cron. And that's why jenkins is king.
- marcosdumay 2y ago
  And the lesson is that you want a simple UI to launch shell scripts, maybe with complex triggers but probably not. If you make anything more than that, your CI will fail. And you can do that with Jenkins, so the people that did it saw it work. (But Jenkins can do so much more, what is the entire reason so many people have nightmares just by hearing that name.)
rrr_oh_man 2y ago
> gorilla consulting Probably 'guerilla', but I like your version more.
- DonHopkins 2y ago
  That's when the devs all wear gorilla suits in Zoo meetings. Wikipedia: Gorilla Suit: National Gorilla Suit Day: https://en.wikipedia.org/wiki/Gorilla_suit#National_Gorilla_Suit_Day https://en.wikipedia.org/wiki/Gorilla_suit#National_Gorilla_... Put the Gorilla back in National Gorilla Suit Day: https://www.instagram.com/mad.magazine/p/C2xgmVqOjL_/ https://www.instagram.com/mad.magazine/p/C2xgmVqOjL_/ Gorilla Suit Day – January 31, 2026: https://nationaltoday.com/gorilla-suit-day/ https://nationaltoday.com/gorilla-suit-day/ National Gorilla Suit Day: https://www.youtube.com/watch?v=N2n5gAN3IlI https://www.youtube.com/watch?v=N2n5gAN3IlI
- hi_hi 2y ago
  Haha, I'm gonna admit it, all these years and I thought gorilla/guerilla was one of those American/British spelling things, like cheque/check or gaol/jail. Boy do I feel stupid.
  - maest 2y ago
    ..."gaol"?
    - QuercusMax 2y ago
      It's the British spelling of "jail", as in "John Bunyan, a prominent Puritan preacher and author, spent 12 years in Bedford Gaol from 1660 to 1672." Pronounced jail, I believe.
      - sd9 2y ago
        It’s the Gaelic spelling of “jail”. It hasn’t been used in mainstream British English since the 60s, outside of specific place names. Everyone in England says “jail” or “prison” today. It might be a bit different in Ireland.
noplacelikehome 2y ago
Nix is awesome for this -- write your entire series of CI tools in she'll or Python and run them locally in the exact same environment as they will run in CI. Add SOPS to bring secrets along for the ride.
- jimbokun 2y ago
  Would Nix work well with GitHub Actions? Or is it more of a replacement? How do you automate running tests and deploying to dev on every push, for example?
  - lewo 2y ago
    > Would Nix work well with GitHub Actions? You can use Nix with GitHub actions since there is a Nix GitHub action: https://github.com/marketplace/actions/install-nix https://github.com/marketplace/actions/install-nix. Every time the action is triggered, Nix rebuilds everything, but thanks to its caching (need to be configured), it only rebuilds targets that has changed. > How do you automate running tests and deploying to dev on every push Nix is a build tool and it's main purpose is not to deploy artifacts. There are however a lot of tools to deploy artifacts built by Nix: https://github.com/nix-community/awesome-nix?tab=readme-ov-file#deployment-tools https://github.com/nix-community/awesome-nix?tab=readme-ov-f... Note there are also several Nix CI that can do a better job than a raw GitHub actions, because they are designed for Nix (Hydra, Garnix, Hercules, ...).
  - noplacelikehome 2y ago
    One neat Nix feature is development shells, which let you define isolated shell environments that can be activated by invoking `nix develop` (or via direnv upon entering a directory): devShells.default = pkgs.mkShell { packages = with pkgs; [ opentofu terragrunt ]; }; I can then use these tools inside the devShell from my jobs like so: jobs: terragrunt-plan: runs-on: [self-hosted, Linux, X64] defaults: run: shell: nix develop --command bash -e {0} steps: - name: Checkout uses: actions/checkout@v4 - name: Plan run: terragrunt --terragrunt-non-interactive run-all plan Since I'm doing this within a Nix flake all of the dependencies for this environment are recorded in a lock file. Provided my clone of the repo is up to date I should have the same versions.

by hi_hi 2y ago

badloginagain 2y ago

I hate the fact that CI peaked with Jenkins. I hate Jenkins, I hate Groovy, but for every company I've worked for there's been a 6-year-uptime Jenkins instance casually holding up the entire company. There's probably a lesson in there.

mike_hearn 2y ago
It peaked with Jenkins? I'm curious which CI platforms you've used. I swear by TeamCity. It doesn't seem to have any of these problems other people are facing with GitHub Actions. You can configure it with a GUI, or in XML, or using a type safe Kotlin DSL. These all actually interact so you can 'patch' a config via the GUI even if the system is configured via code, and TeamCity knows how to store config in a git repository and make commits when changes are made, which is great for quick things where it's not worth looking up the DSL docs or for experimentation. The UI is clean and intuitive. It has all the features you'd need. It scales. It isn't riddled with insecure patterns like GH Actions is.
- DanielHB 2y ago
  I think people just hate CI set up by other people. I used TeamCity in a job a few years back and I absolutely hated it, however I imagine a lot of my hatred was the way it was set up. CI is just the thing no one wants to deal with, yet everyone wants to just work. And like any code or process, you need engineering to make it good. And like any project, you can't just blame bad tools for crappy results.
  - bluGill 2y ago
    I would feel that way but I've had the misfortune to work with a wide open ci system where any developer could make thanges and one guy did. The locked down system prevents me form some changes I want but in return my builds don't suddenly start failing because some ci option was turned on for everyone.
    - HeavyStorm 2y ago
      I totally prefer to have the ci break from time to time and be able to fix it than having the risk of it being broken and having no way of fixing it
      - bluGill 2y ago
        The people who admin our CI system do a good job so it doesn't break, (well it does all the time, but onnetwork type errors not configuration - that is IT's fault not their fault.) The thing I want to change are things that I do in the build system so that it is checked in and previous versions when we need to build them (we are embedded where field failure is expensive so there are typically branches for the current release, next release, and head). This also means anything that can fail on CI can fail on my local system (unless it depends on something like the number of cores on the machine running the build). While the details can be slightly different, how we have CI is how it should be. most developers should have better things to do than worry about how to configure CI.
        2y ago
skor 2y ago
well, I got tired of Groovy and found out that using Jenkins with plain bash under source control is just right for us. Runs everywhere, very fast to test/develop and its all easy to change and improve. We build Docker images mostly so ymmv. I have a "port to github actions" ticket in the backlog but I think we're not going to go down that road now.
- __float 2y ago
  Yeah, I've come back around to this: you do not want "end users" writing Groovy, because the tooling around it is horrible. You'll have to explain the weird CPS transformations, you'll probably end up reading the Jenkins plugins' code, and there's nothing fun down this path.
k4rli 2y ago
It's feature complete. Anything more will just be bloat, probably 25% of it could be reduced at least.
finnthehuman 2y ago
Jenkins is cron with bells and whistles. The result is a pile of plugins to capture all the dimensions of complexity you are likely to otherwise bury in the shell script but want them easier to point and click at. I'll hate on jenkins with the rest of them, but entropy is gonna grow and Jenkins isn't gonna say "no, you can't do that here". I deal with multiple tools where if tried to make fun about how low the jenkins plugin install starts are, you'd know exactly where I work. Once I've calmed down from working on CI I can appreciate Jenkins' attempts to manage all of it. Any CI product play has to differentiate in a way that makes you dependent on them. Sure it can be superficially nicer when staying inside the guard rails, but in the age of docker why has the number of ways I configure running boring shell scripts gone UP? Because they need me unable to use a lunch break to say "fuck you I don't need the integrations you reserve exclusively for your CI" and port all the jobs back to cron. And that's why jenkins is king.
marcosdumay 2y ago
And the lesson is that you want a simple UI to launch shell scripts, maybe with complex triggers but probably not. If you make anything more than that, your CI will fail. And you can do that with Jenkins, so the people that did it saw it work. (But Jenkins can do so much more, what is the entire reason so many people have nightmares just by hearing that name.)

rrr_oh_man 2y ago

> gorilla consulting Probably 'guerilla', but I like your version more.

DonHopkins 2y ago
That's when the devs all wear gorilla suits in Zoo meetings. Wikipedia: Gorilla Suit: National Gorilla Suit Day: https://en.wikipedia.org/wiki/Gorilla_suit#National_Gorilla_Suit_Day https://en.wikipedia.org/wiki/Gorilla_suit#National_Gorilla_... Put the Gorilla back in National Gorilla Suit Day: https://www.instagram.com/mad.magazine/p/C2xgmVqOjL_/ https://www.instagram.com/mad.magazine/p/C2xgmVqOjL_/ Gorilla Suit Day – January 31, 2026: https://nationaltoday.com/gorilla-suit-day/ https://nationaltoday.com/gorilla-suit-day/ National Gorilla Suit Day: https://www.youtube.com/watch?v=N2n5gAN3IlI https://www.youtube.com/watch?v=N2n5gAN3IlI
hi_hi 2y ago
Haha, I'm gonna admit it, all these years and I thought gorilla/guerilla was one of those American/British spelling things, like cheque/check or gaol/jail. Boy do I feel stupid.
- maest 2y ago
  ..."gaol"?
  - QuercusMax 2y ago
    It's the British spelling of "jail", as in "John Bunyan, a prominent Puritan preacher and author, spent 12 years in Bedford Gaol from 1660 to 1672." Pronounced jail, I believe.
    - sd9 2y ago
      It’s the Gaelic spelling of “jail”. It hasn’t been used in mainstream British English since the 60s, outside of specific place names. Everyone in England says “jail” or “prison” today. It might be a bit different in Ireland.

noplacelikehome 2y ago

Nix is awesome for this -- write your entire series of CI tools in she'll or Python and run them locally in the exact same environment as they will run in CI. Add SOPS to bring secrets along for the ride.

jimbokun 2y ago
Would Nix work well with GitHub Actions? Or is it more of a replacement? How do you automate running tests and deploying to dev on every push, for example?
- lewo 2y ago
  > Would Nix work well with GitHub Actions? You can use Nix with GitHub actions since there is a Nix GitHub action: https://github.com/marketplace/actions/install-nix https://github.com/marketplace/actions/install-nix. Every time the action is triggered, Nix rebuilds everything, but thanks to its caching (need to be configured), it only rebuilds targets that has changed. > How do you automate running tests and deploying to dev on every push Nix is a build tool and it's main purpose is not to deploy artifacts. There are however a lot of tools to deploy artifacts built by Nix: https://github.com/nix-community/awesome-nix?tab=readme-ov-file#deployment-tools https://github.com/nix-community/awesome-nix?tab=readme-ov-f... Note there are also several Nix CI that can do a better job than a raw GitHub actions, because they are designed for Nix (Hydra, Garnix, Hercules, ...).
- noplacelikehome 2y ago
  One neat Nix feature is development shells, which let you define isolated shell environments that can be activated by invoking `nix develop` (or via direnv upon entering a directory): devShells.default = pkgs.mkShell { packages = with pkgs; [ opentofu terragrunt ]; }; I can then use these tools inside the devShell from my jobs like so: jobs: terragrunt-plan: runs-on: [self-hosted, Linux, X64] defaults: run: shell: nix develop --command bash -e {0} steps: - name: Checkout uses: actions/checkout@v4 - name: Plan run: terragrunt --terragrunt-non-interactive run-all plan Since I'm doing this within a Nix flake all of the dependencies for this environment are recorded in a lock file. Provided my clone of the repo is up to date I should have the same versions.