3 ms·

This is exactly why I both love and hate CI/CD. Ultimately most CI/CD setups are basically systems administrators with privileged access to everything, network

by movedx 5y ago

This is exactly why I both love and hate CI/CD. Ultimately most CI/CD setups are basically systems administrators with privileged access to everything, network connected and running 24/7. It's pretty dangerous stuff. I don't have an answer though, expect maybe to keep the CI and CD in separate, isolated instances that require manual intervention to bridge the gap on a case by case basis. That doesn't scale very well though.

hinkley 5y ago
I think in general we put too much logic into our CI/CD configurations. There is an argument to be made for a minimalist CI/CD implementation that can handle task scheduling and dependencies, understands how to fetch and tag version control, count version numbers and not much else. Even extracting test result summaries, while handy, maybe should be handled another way. For many of us, if CI is down you can't deploy anything to production, not even roll back to a previous build. Everything but the credentials should be under version control, and the right people should be able to fire off a one-liner from a runbook that has two to four sanity checked arguments in order to trigger a deployment.
- jeffjeff2 5y ago
  A place I was at recently used makefiles in each project containing entries for "build", "deploy", "test" etc. All the CI did was use shared generic jobs to call the relevant makefile command. These could just as easily be run from a developers machine with the right credentials.
  - john-tells-all 5y ago
    I strongly recommend this. Put all the CICD machinery inside the dev repos, letting all the devs see, understand, and modify the pipeline. With carefully chosen creds, Devs can run the CICD things directly, thus making feedback loops much faster in certain circumstances. Maybe don't let them delete databases, or run expensive VMs, but other than that go for it. In my experience if the pipeline is even slightly different Devs will treat it as a foreign object and will whine about it not working the way they want... they don't feel like they can change the pipeline, which is odd. Dude, it's code, just change it, I'm happy to approve your PR and/or help you make a change.
    - movedx 5y ago
      > I strongly recommend this. Put all the CICD machinery inside the dev repos, letting all the devs see, understand, and modify the pipeline. I disagree with this 1,000%. This is a massive security risk. From an InfoSec perspective it's a massive security risk because it implies the developers have access to AWS API's directly, otherwise how could they modify the pipeline or even engage it, when it manipulates infrastructure? And that's not even to mention all the other security implications that come with allowing developers to edit a pipeline... holy s-... I don't even want to imagine the chaos brought about by a quick edit to the `.gitlab-ci.yml` file that runs `terraform destroy -auto-approve` an hour before handing in one's notice. And no, code reviews aren't a sufficient barrier on their own. This is just such a bad idea on so many levels. > In my experience if the pipeline is even slightly different Devs will treat it as a foreign object and will whine about it not working the way they want... Who cares? That's not how DevOps works. It's not how a business works. Everyone has a part to play and it's not always going to be comfortable. Operations specialists are just that: specialists at handling the operations. Let them do it, just as they let developers use the tools they want or handle things the way they see best. Comfort doesn't apply.
      - hinkley 5y ago
        We're talking about putting all the code except credentials (or with revocable credentials) where developers can see it. IMO, the worst CI/CD tool on the market is Bamboo, and I've been using CI since you had to edit an XML file and restart. And the reason for that is information hiding. The Bamboo UI is completely undiscoverable. All of the things you can do with it are buried in the docs. Anything you don't have permission to do is eliminated from the UI, so unless you RTFM you don't even know what tools you have available to solve problems.
contingencies 5y ago
A distributed system is one where the failure of a machine you've never heard of stops you from getting any work done. - Leslie Lamport ... via https://github.com/globalcitizen/taoup https://github.com/globalcitizen/taoup

by movedx 5y ago

hinkley 5y ago

I think in general we put too much logic into our CI/CD configurations. There is an argument to be made for a minimalist CI/CD implementation that can handle task scheduling and dependencies, understands how to fetch and tag version control, count version numbers and not much else. Even extracting test result summaries, while handy, maybe should be handled another way. For many of us, if CI is down you can't deploy anything to production, not even roll back to a previous build. Everything but the credentials should be under version control, and the right people should be able to fire off a one-liner from a runbook that has two to four sanity checked arguments in order to trigger a deployment.

jeffjeff2 5y ago
A place I was at recently used makefiles in each project containing entries for "build", "deploy", "test" etc. All the CI did was use shared generic jobs to call the relevant makefile command. These could just as easily be run from a developers machine with the right credentials.
- john-tells-all 5y ago
  I strongly recommend this. Put all the CICD machinery inside the dev repos, letting all the devs see, understand, and modify the pipeline. With carefully chosen creds, Devs can run the CICD things directly, thus making feedback loops much faster in certain circumstances. Maybe don't let them delete databases, or run expensive VMs, but other than that go for it. In my experience if the pipeline is even slightly different Devs will treat it as a foreign object and will whine about it not working the way they want... they don't feel like they can change the pipeline, which is odd. Dude, it's code, just change it, I'm happy to approve your PR and/or help you make a change.
  - movedx 5y ago
    > I strongly recommend this. Put all the CICD machinery inside the dev repos, letting all the devs see, understand, and modify the pipeline. I disagree with this 1,000%. This is a massive security risk. From an InfoSec perspective it's a massive security risk because it implies the developers have access to AWS API's directly, otherwise how could they modify the pipeline or even engage it, when it manipulates infrastructure? And that's not even to mention all the other security implications that come with allowing developers to edit a pipeline... holy s-... I don't even want to imagine the chaos brought about by a quick edit to the `.gitlab-ci.yml` file that runs `terraform destroy -auto-approve` an hour before handing in one's notice. And no, code reviews aren't a sufficient barrier on their own. This is just such a bad idea on so many levels. > In my experience if the pipeline is even slightly different Devs will treat it as a foreign object and will whine about it not working the way they want... Who cares? That's not how DevOps works. It's not how a business works. Everyone has a part to play and it's not always going to be comfortable. Operations specialists are just that: specialists at handling the operations. Let them do it, just as they let developers use the tools they want or handle things the way they see best. Comfort doesn't apply.
    - hinkley 5y ago
      We're talking about putting all the code except credentials (or with revocable credentials) where developers can see it. IMO, the worst CI/CD tool on the market is Bamboo, and I've been using CI since you had to edit an XML file and restart. And the reason for that is information hiding. The Bamboo UI is completely undiscoverable. All of the things you can do with it are buried in the docs. Anything you don't have permission to do is eliminated from the UI, so unless you RTFM you don't even know what tools you have available to solve problems.

contingencies 5y ago

A distributed system is one where the failure of a machine you've never heard of stops you from getting any work done. - Leslie Lamport ... via https://github.com/globalcitizen/taoup https://github.com/globalcitizen/taoup