4 ms·

Slightly offtopic from what the article focuses on, but the thing that bothers me about all these built-in CI/CD things in software like Gitlab is that people s

by tazjin 9y ago

Slightly offtopic from what the article focuses on, but the thing that bothers me about all these built-in CI/CD things in software like Gitlab is that people seem to be perfectly content with building an artifact and then just ... pushing it out. In traditional deployments that may be some kind of "copy this thing over SSH and make it go!", in Kubernetes-land it's more like "lets just modify this API-state to point at the new image tag!". Neither of those produce a consistent record of changes applied and state mutation like that is very error-prone. We actually use Gitlab's CI at work, but we have our own deployment solutions built on it that end up making git commits into the NixOS & Kontemplate repositories which then run their own pipelines to deploy. This way we can always answer the question "what set of applications at which versions was deployed at $time?" and also roll back consistently.

akvadrako 9y ago
Sorry but it's not clear what you are saying. What is the fundamental difference between modifying your k8s deployment to point to a new tag and your solution? You can easily roll back (except with database migrations) and easily see what version is deployed when.
- dankohn1 9y ago
  The idea of keeping all state around your deployments in git is being called GitOps: https://www.weave.works/blog/gitops-operations-by-pull-request https://www.weave.works/blog/gitops-operations-by-pull-reque...
- marcinzm 9y ago
  Infrastructure as Code is a useful thing and this is essentially building on top of that idea. For example, you can spin up an exact copy of any environment automatically including all software running on it. If something utterly takes out an environment you can spin up a new copy of it and be certain it's identical to what was there before. You can rollback and manage multiple pieces of software at once if there's dependencies between them. You have an automatic and immutable audit log of who did what and when. Using git you can keep track of all infrastructure related changes in one place instead of trying to piece together information from multiple sources when debugging.
- tazjin 9y ago
  When you update a Kubernetes deployment in the API directly you're essentially modifying a global variable. If you do this as the result of some imperative process (e.g. a CI pipeline), you don't have any record telling you what the value should be - you only know what it currently is. You're also only modifying a single piece of your whole state at any given time, meaning that if you have the services A and B and their deployment pipelines independently modify state - you have nothing that declares any relationship between which versions of these should be deployed together. There's a little piece of infrastructure wisdom I've learned over the years, I refer to it sometimes as "tazjin's law": Any infrastructure component not controlled by a reconciliation process will eventually fail. Versions of dependent components will get out of sync, configuration is being deployed independently of the application, and so on. In order to reconcile your current state with your desired state you must know what your desired state is. Does that explain it?
  - akvadrako 9y ago
    That doesn't explain it. You always need some global variable saying what the current state is. In your case, maybe it's the branch pointer in your git tree. In the k8s deployment, it's a tag value.
    - tazjin 9y ago
      No, those don't contain the same information. Our Kontemplate repository contains the state of an entire cluster, including all versions and all configuration. A tag value is a single piece of mutable data that is in no relation to the other relevant data.
      - akvadrako 9y ago
        The tag points to the current version of your docker image, which contains the current configuration, except secrets which you don't want in a repo. Older deployments point to the older docker images which contain the older versions. Still don't see any advantage to justify the additional complexity in your system.
ianwalter 9y ago
What about Helm?
- tazjin 9y ago
  I aim for as few moving components at runtime as possible. Helm has tiller, which I'm not a big fan of. In addition its design (with warts like the separate versioning layer of charts) adds complexity that I don't think is necessary for most use-cases.

by tazjin 9y ago

akvadrako 9y ago

Sorry but it's not clear what you are saying. What is the fundamental difference between modifying your k8s deployment to point to a new tag and your solution? You can easily roll back (except with database migrations) and easily see what version is deployed when.

dankohn1 9y ago
The idea of keeping all state around your deployments in git is being called GitOps: https://www.weave.works/blog/gitops-operations-by-pull-request https://www.weave.works/blog/gitops-operations-by-pull-reque...
marcinzm 9y ago
Infrastructure as Code is a useful thing and this is essentially building on top of that idea. For example, you can spin up an exact copy of any environment automatically including all software running on it. If something utterly takes out an environment you can spin up a new copy of it and be certain it's identical to what was there before. You can rollback and manage multiple pieces of software at once if there's dependencies between them. You have an automatic and immutable audit log of who did what and when. Using git you can keep track of all infrastructure related changes in one place instead of trying to piece together information from multiple sources when debugging.
tazjin 9y ago
When you update a Kubernetes deployment in the API directly you're essentially modifying a global variable. If you do this as the result of some imperative process (e.g. a CI pipeline), you don't have any record telling you what the value should be - you only know what it currently is. You're also only modifying a single piece of your whole state at any given time, meaning that if you have the services A and B and their deployment pipelines independently modify state - you have nothing that declares any relationship between which versions of these should be deployed together. There's a little piece of infrastructure wisdom I've learned over the years, I refer to it sometimes as "tazjin's law": Any infrastructure component not controlled by a reconciliation process will eventually fail. Versions of dependent components will get out of sync, configuration is being deployed independently of the application, and so on. In order to reconcile your current state with your desired state you must know what your desired state is. Does that explain it?
- akvadrako 9y ago
  That doesn't explain it. You always need some global variable saying what the current state is. In your case, maybe it's the branch pointer in your git tree. In the k8s deployment, it's a tag value.
  - tazjin 9y ago
    No, those don't contain the same information. Our Kontemplate repository contains the state of an entire cluster, including all versions and all configuration. A tag value is a single piece of mutable data that is in no relation to the other relevant data.
    - akvadrako 9y ago
      The tag points to the current version of your docker image, which contains the current configuration, except secrets which you don't want in a repo. Older deployments point to the older docker images which contain the older versions. Still don't see any advantage to justify the additional complexity in your system.

ianwalter 9y ago

What about Helm?

tazjin 9y ago
I aim for as few moving components at runtime as possible. Helm has tiller, which I'm not a big fan of. In addition its design (with warts like the separate versioning layer of charts) adds complexity that I don't think is necessary for most use-cases.