4 ms·

In a previous job we had a team tasked with designing these "modern" CI/CD pipeline solutions, mostly meant for Kubernetes, but it was suppose to work for every

In a previous job we had a team tasked with designing these "modern" CI/CD pipeline solutions, mostly meant for Kubernetes, but it was suppose to work for everything. They had such a hard on for tools that would run each step as a separate isolated task and did not want pipelines to "devolve" into shell scripts. Getting anything done in such environments are just a pain. You spend more time fighting the systems than you do actually solving problems. It is my opinion that a CI/CD system needs just the following features: Triggers (source code repo, http endpoints or manually triggered), secret management and shell script execution. That's it, you can build anything using that.

eddd-ddde 2y ago
I think what they really wanted was something like bazel. The only real benefit I can think right now for not "devolving" into shell scripts is distributed caching with hermetic builds. It has very real benefits but it also requires real effort to work correctly.
datavirtue 2y ago
I just joined as the enterprise architect for company that has never had one. There is an existing devops team that is making everyone pull their hair out and I haven't had a single spare minute to dig in on their mess but this sounds early familiar.
- mongol 2y ago
  Is this really the job of an enterprise architect? To dig into the details of a devops team's mess?
  - datavirtue 2y ago
    It's more about giving the team and the overall strategy a thumbs up or down, so yes.
  - eru 2y ago
    The job of senior people should mostly be to make sure the organisation runs smoothly. If no one else is doing anything about the mess, then it falls to the senior person to sort it out. As a rule of thumb: - Ideally your people do the Right Thing by themselves by the magic of 'leadership'. - Second best: you chase the people to do the Right Thing. - Third: you as the senior person do the Right Thing. - Least ideal: no one fixes the mess nor implements the proper way. I guess some people can achieve the ideal outcome with pure charisma (or fear?) alone, but I find that occasionally getting your hands dirty (option 3) helps earn the respect to make the 'leadership' work. It can also help ground you in the reality of the day to day work. However, you are right that a senior person shouldn't get bogged down with such work. You need to cut your losses at some point.
    - mongol 2y ago
      Where I work, which granted is a very large company, the enterprise architects focus on ERP processes, logistic flows, how prices flow from the system where they are managed to the places that need them, and so on. They are several levels removed from devops teams. DevOps concerns are maybe handled by tech leads, system architects or technical product managers.
      - eru 2y ago
        Makes sense. From datavirtue's comment is sounded like they joined a much smaller outfit without much in terms of established _working_ procedures here.
deleted 2y ago
[deleted]
chrisweekly 2y ago
Mostly agreed, but (maybe orthogonal) IME, popular CI/CD vendors like TeamCity* can make even basic things like shell script execution problematic. * TC offers sh, full stop. If you want to script something that depends on bash, it's a PITA and you end up with a kludge to run bash in sh in docker in docker.
- mdaniel 2y ago
  > If you need a specific interpreter to be used, specify shebang (for example, #!/bin/bash) as the first line of the script. https://www.jetbrains.com/help/teamcity/command-line.html#:~:text=for%20example%2C%20%23!%2Fbin%2Fbash https://www.jetbrains.com/help/teamcity/command-line.html#:~... Your "docker in docker" comment makes me wonder if you're conflating the image that you are using, that just happens to be run by TeamCity, versus some inherent limitation of TC. I believe a boatload of the Hashicorp images ship with dash, or busybox or worse, and practically anything named "slim" or "alpine" similarly

eddd-ddde 2y ago

I think what they really wanted was something like bazel. The only real benefit I can think right now for not "devolving" into shell scripts is distributed caching with hermetic builds. It has very real benefits but it also requires real effort to work correctly.

datavirtue 2y ago

I just joined as the enterprise architect for company that has never had one. There is an existing devops team that is making everyone pull their hair out and I haven't had a single spare minute to dig in on their mess but this sounds early familiar.

mongol 2y ago
Is this really the job of an enterprise architect? To dig into the details of a devops team's mess?
- datavirtue 2y ago
  It's more about giving the team and the overall strategy a thumbs up or down, so yes.
- eru 2y ago
  The job of senior people should mostly be to make sure the organisation runs smoothly. If no one else is doing anything about the mess, then it falls to the senior person to sort it out. As a rule of thumb: - Ideally your people do the Right Thing by themselves by the magic of 'leadership'. - Second best: you chase the people to do the Right Thing. - Third: you as the senior person do the Right Thing. - Least ideal: no one fixes the mess nor implements the proper way. I guess some people can achieve the ideal outcome with pure charisma (or fear?) alone, but I find that occasionally getting your hands dirty (option 3) helps earn the respect to make the 'leadership' work. It can also help ground you in the reality of the day to day work. However, you are right that a senior person shouldn't get bogged down with such work. You need to cut your losses at some point.
  - mongol 2y ago
    Where I work, which granted is a very large company, the enterprise architects focus on ERP processes, logistic flows, how prices flow from the system where they are managed to the places that need them, and so on. They are several levels removed from devops teams. DevOps concerns are maybe handled by tech leads, system architects or technical product managers.
    - eru 2y ago
      Makes sense. From datavirtue's comment is sounded like they joined a much smaller outfit without much in terms of established _working_ procedures here.

deleted 2y ago

[deleted]

chrisweekly 2y ago

Mostly agreed, but (maybe orthogonal) IME, popular CI/CD vendors like TeamCity* can make even basic things like shell script execution problematic. * TC offers sh, full stop. If you want to script something that depends on bash, it's a PITA and you end up with a kludge to run bash in sh in docker in docker.

mdaniel 2y ago
> If you need a specific interpreter to be used, specify shebang (for example, #!/bin/bash) as the first line of the script. https://www.jetbrains.com/help/teamcity/command-line.html#:~:text=for%20example%2C%20%23!%2Fbin%2Fbash https://www.jetbrains.com/help/teamcity/command-line.html#:~... Your "docker in docker" comment makes me wonder if you're conflating the image that you are using, that just happens to be run by TeamCity, versus some inherent limitation of TC. I believe a boatload of the Hashicorp images ship with dash, or busybox or worse, and practically anything named "slim" or "alpine" similarly