5 ms·

I don't understand why this is not the evident approach for everyone writing GitHub Actions/GitLab CI/CD yaml etc.... I've struggled in some teams to explained

• arghwhat 2y ago
  My "favorite" is when I see people go all in, writing thousands of lines of Jenkins-flavor Groovy that parses JSON build specifications of arbitrary complexity to sort out how to build that particular project. "But then we can reuse the same pipeline for all our projects!"
  • lowercased 2y ago
    I can rarely reuse the same pipeline for the same project 6 months down the road, much less reuse for anything else. The few bits that end up getting reused are the externalized bash scripts.
  • ozim 2y ago
    I think that is pitfall of software devs. For me it was an epiphany as software dev - not to write reusable extensible scripts - I am so much more productive after that.
  • baby_souffle 2y ago
    > "But then we can reuse the same pipeline for all our projects!" oh god just reading that gave me PTSD flash backs. At $priorGig there was the "omni-chart". It was a helm chart that was so complex it needed to be wrapped in terraform and used composable terraform modules w/ user var overrides as needed. Debugging anything about it meant clearing your calendar for the day and probably the following day, too.
• mrweasel 2y ago
  In a previous job we had a team tasked with designing these "modern" CI/CD pipeline solutions, mostly meant for Kubernetes, but it was suppose to work for everything. They had such a hard on for tools that would run each step as a separate isolated task and did not want pipelines to "devolve" into shell scripts. Getting anything done in such environments are just a pain. You spend more time fighting the systems than you do actually solving problems. It is my opinion that a CI/CD system needs just the following features: Triggers (source code repo, http endpoints or manually triggered), secret management and shell script execution. That's it, you can build anything using that.
  • eddd-ddde 2y ago
    I think what they really wanted was something like bazel. The only real benefit I can think right now for not "devolving" into shell scripts is distributed caching with hermetic builds. It has very real benefits but it also requires real effort to work correctly.
  • datavirtue 2y ago
    I just joined as the enterprise architect for company that has never had one. There is an existing devops team that is making everyone pull their hair out and I haven't had a single spare minute to dig in on their mess but this sounds early familiar.
    • mongol 2y ago
      Is this really the job of an enterprise architect? To dig into the details of a devops team's mess?
      • datavirtue 2y ago
        It's more about giving the team and the overall strategy a thumbs up or down, so yes.
      • eru 2y ago
        The job of senior people should mostly be to make sure the organisation runs smoothly. If no one else is doing anything about the mess, then it falls to the senior person to sort it out. As a rule of thumb: - Ideally your people do the Right Thing by themselves by the magic of 'leadership'. - Second best: you chase the people to do the Right Thing. - Third: you as the senior person do the Right Thing. - Least ideal: no one fixes the mess nor implements the proper way. I guess some people can achieve the ideal outcome with pure charisma (or fear?) alone, but I find that occasionally getting your hands dirty (option 3) helps earn the respect to make the 'leadership' work. It can also help ground you in the reality of the day to day work. However, you are right that a senior person shouldn't get bogged down with such work. You need to cut your losses at some point.
• maccard 2y ago
  To make the thing actually fast at scale, a lot of the logic ends up being specific to the provider; requiring tokens, artifacts etc that aren't available locally. You end up with something that tries to detect if you're running locally or in CI, and then you end up in exactly the same situation.
  • Lutger 2y ago
    You are right, and this is where a little bit of engineering comes in. Push as much of the logic to scripts (either shell or python or whatever) that you can run locally. Perhaps in docker, whatever. All the token, variables, artifacts etc should act as inputs or parameters to your scripts. You have several mechanisms at your disposal, command line arguments, environment variables, config files, etc. Those are all well understood, universal, language and environment agnostic, to an extent. The trick is to NOT have your script depend on the specifics of the environment, but reverse the dependency. So replace all `If CI then Run X else if Local Run Y` with the ability to configure the script to run X or Y, then let the CI configure X and local configure Y. For example. I'm not saying it is always easy and obvious. For bigger builds, you often really want caching and have shitloads of secrets and configurations going on. You want to only build what is needed, so you need something like a DAG. It can get complex fast. The trick is making it only as complex as it needs be, and only as reusable as and when it is actually re-used.
    • maccard 2y ago
      > The trick is to NOT have your script depend on the specifics of the environment, but reverse the dependency. So replace all `If CI then Run X else if Local Run Y` with the ability to configure the script to run X or Y, then let the CI configure X and local configure Y. For example. > I'm not saying it is always easy and obvious. For bigger builds, you often really want caching and have shitloads of secrets and configurations going on. Here's the thing. When you don't want secrets, or caching, or messaging, or conditional deploys, none of this matters. Your build can be make/mvn/go/cargo and it just works, and is easy. It only gets messy when you want to do "detect changes since last run and run tests on those components", or "don't build the moon and the stars, pull that dependency in CI, as local users have it built already and it won't change." And the way to handle those situations involves running different targets/scripts/whatever and not what is actually in your CI environment. I've lost count of how many deploys have been marked as failed in my career because the shell script for posting updates to slack has an error, and that's not used in the local runs of CI. What you _actually_ need is a staging branch of your CI and a dry-run flag on all of your state modifying commands. Then, none of this matters.
• arccy 2y ago
  it can be quite hard to write proper scripts that work consistently... different shells have different behaviours, availability of local tools, paths, etc and it feels like fighting against the flow when you're trying to make it reusable across many repos
  • akdev1l 2y ago
    Containerize the build environment so everything is captured (dependencies, build tools, etc)
    • carlmr 2y ago
      If you're not containerizing your CI/CD, you're really lost.
      • maccard 2y ago
        How do I containerize building desktop apps for windows with MSVC?
        • eru 2y ago
          Wine? Less snarky: https://learn.microsoft.com/en-us/virtualization/windowscontainers/about/ https://learn.microsoft.com/en-us/virtualization/windowscont... seems to be a thing? And in any case, you can use VMs instead of containers.
          • maccard 2y ago
            You know, I'd love to run MSVC in Wine on a ubuntu container. I bet it would be quicker. I've had the unfortunate pleasure of working with Windows Containers in the past. They are Containers in the sense that you can write a dockerfile for them, and they're somewhat isolated, but they're no better than a VM in my experience. > And in any case, you can use VMs instead of containers. They're not the same thing. If that's the case running linux AMI's on EC2 should be the same as containers.
• dboreham 2y ago
  The reason for this is that nobody took the time to write a proper background document on Github Actions. The kind of information that you or I might convey if asked to explain it at the whiteboard to junior hires, or senior management. This syndrome is very common these days. Things are explained differentially: it's like Circle CI but in the GH repo. Well that's no use if the audience wasn't around when Circle CI was first new and readily explained (It's like Jenkins but in the cloud...).
• 0xbadcafebee 2y ago
  I think I can summarize it in a rough, general way. CI/CD is a method to automate tasks in the background that you would otherwise run on your laptop. The output of the tasks are used as quality gates for merging commits, and for deployments. - Step 1. Your "laptop in the cloud" requires some configuration (credentials, installed software, cached artifacts) before a job can be run. - Requires logic specific to the CI/CD system - Step 2. Running many jobs in parallel, passing data from step to step, etc requires some instructions. - Requires logic specific to the CI/CD system - Step 3. The job itself is the execution of a program (or programs), with some inputs and outputs. - Works the same on any computer (assuming the same software, environment, inputs, etc) - Using a container in Step 1. makes this practical and easy - Step 4. After the job finishes, artifacts need to be saved, results collected, and notifications sent. - Some steps are specific to the CI/CD system, others can be a reusable job Step 3 does not require being hard-coded into the config format of the CI/CD system. If it is instead just executable code in the repo, it allows developers to use (and work on) the code locally without the CI/CD system being involved. It also allows moving to a different CI/CD system without ever rewriting all the jobs; the only thing that needs to be rewritten are the CI/CD-specific parts, which should be generic and apply to all jobs pretty much the same. Moving the CI/CD-specific parts to a central library of configuration allows you to write some code once and reuse it many times (making it DRY). CircleCI Orbs, GitHub Actions, Jenkins Shared Libraries/ Groovy Libraries, etc are examples of these. Write your code once, fix a bug once, reuse it everywhere.
• harrall 2y ago
  A shell script has many extremely sharp edges like dealing with stdin, stderr, stdout, subprocesses, exit codes, environmental variables, etc. Most programmers have never written a shell script and writing CI files is already frustrating because sometimes you have to deploy, run, fix, deploy, run, fix, which means nobody is going to stop in the middle of that and try to learn shell scripting. Instead, they copy commands from their terminal into the file and the CI runner takes care of all the rough edges. I ALWAYS advise writing a shell script but I know it's because I actually know how to write them. But I guess that's why some people are paid more big bux.
  • eru 2y ago
    GitHub's CI yaml also accepts eg Python. (Or anything else, actually.) That's generally a bit less convenient, ie it takes a few more lines, but it has significantly fewer sharp edges than your typical shell script. And more people have written Python scripts, I guess?
    • harrall 2y ago
      I find that Python scripts that deal with calling other programs have even more sharp edges because now you have to deal with stdin, stderr and stdout much more explicitly. Now you need to both know shell scripting AND Python. Python’s subprocess has communicate(), check_output() and other helpers which takes care of a lot but (a) you need to know what method you should actually call and (b) if you need to do something outside of that, you have to use Popen directly and it’s much more work than just writing a shell script. All doable if you understand pipes and all that but if you don’t, you’ll be just throwing stuff at the wall until something sticks.
      • eru 2y ago
        Yes, but at least that's all fairly obvious---you might not know how to solve the problem, but at least you know you have a problem that needs solving. Compare that to the hidden pitfalls of eg dealing with whitespace in filenames in shell scripts. Or misspelled variable names that accidentally refer to non-existent variables but get treated as if they are set to be "".
• eru 2y ago
  > [...] instead of writing a Frankenstein of YAML and shell commands. The 'Frankenstein' approach isn't what makes it necessarily worse. Eg Makefiles work like that, too, and while I have my reservations about Make, it's not really because they embed shell scripts.
• dpkirchner 2y ago
  This all reminds me of the systemd ini-like syntax vs shell scripts debate. Shell scripts are superior, of course, but they do require deeper knowledge of unix-like systems.
  • slt2021 2y ago
    yeah if you author CI jobs, you should know linux, otherwise a person should not even touch the CI system with 10ft pole
    • surajrmal 2y ago
      I've been working with Linux since I was 10 (I'm much older now), and I still don't think I "know Linux". The upper bound on understanding it is incredibly high. Where do you draw the line?
      • slt2021 2y ago
        just basics enough to understand typical bash commands, scripts, how env variables work, etc