3 ms·

We're in the process of evaluating tools to get away from 90% identical yaml files across environments and this seems like a good alternative to kustomize or he

We're in the process of evaluating tools to get away from 90% identical yaml files across environments and this seems like a good alternative to kustomize or helm. Do you have a good pattern on how to use it with CI/CD for deployments? The biggest challenge we've had after writing deployments is getting it setup to work with something like Jenkins (right now we have a custom bash script that does a bunch kubectl things). (PS any way this would help with static IPs on hosted Grafana.com Cloud to make access to firewalled datasources easier?)

jordanbeiber 7y ago
Here’s one example, not using any of these “templating turned programing languages” tools: We build an api and a cli util for these things. The api takes care of providing sane deployment manifests based on data from a service registry, which is populated using the cli. Cli can be run non-interactive for use in pipeline and other automations. The cli and api basically do any config and deplyment related tasks in concert with service discovery (which at this time is consul). Edit, rant: For automations json just owns yaml in all the ways! This yaml templating business should stop! I think it’s silly tbh. :) Our base k8s setup consisted of roughly 40k lines of yaml. Yaml is awesome for 10 lines of user input configuration, not 100, let alone 40k machine printed stuff. What happened here in this brave new DevOps world?!
maineldc 7y ago
For integrating k8s deploy into CI / CD, I really like Krane [0]. It has several key features: 1. A way to templatize YAML (though it could be used with Tanka too - the render command is split from the deploy command for this reason) 2. Monitors the rollout of resources - it's possible to detect successful and failed deployments more easily. 3. A way to run one-off commands during deployment. 4. Uses kubectl under the hood so easy to add to any workflow. It has other features that I don't use including secrets management. Net net - I definitely think it could elegantly replace a bunch of bash scripts that do kubectl things. [0] https://github.com/Shopify/krane https://github.com/Shopify/krane
transect 7y ago
For CI (testing) we use tekton [1] to run tests and pass around artifacts inside k8s. You can kick off and monitor tekton builds with their CLI tool, but we ended up building our own create/monitor/download-artifacts tool for it in ruby. We use an off the shelf CI server to kick that tool off, and it dumps back results the CI server can understand. One of the things we need to do is elastically scale the number of tasks (basically pods in tekton) that comprise our test suite run. This might be based on cluster utilization or whether it's the master branch. Since we have a single threaded test suite, we hoist parallelism up to the k8s level by breaking apart the tests into partitions each run by its own pod. For this we just render processed and parameterized erb to yaml. Eventually we'll dispense with yaml altogether and programmatically construct resources using a k8s REST api client. We haven't moved into CD with any of this tooling yet. [1] https://github.com/tektoncd/pipeline https://github.com/tektoncd/pipeline
shorez 7y ago
(Tanka dev here) I think this highly depends on your deployment process .. do you want full continuous deployment (CI deploying to the cluster)? In this case you could continue to use for example Jenkins to run `tk apply <environment>` on each merged PR. Another option would be to use an in-cluster CD agent (for example https://fluxcd.io/ https://fluxcd.io/), which uses Tanka to generate the yaml and applies it. Flux can be used with Tanka, needs some setup though: https://docs.fluxcd.io/en/1.17.0/references/fluxyaml-config-files.html https://docs.fluxcd.io/en/1.17.0/references/fluxyaml-config-.... I guess we could simplify this in the future. Feel free to reach out to me on Slack http://slack.raintank.io/ http://slack.raintank.io/ in the #tanka channel :D

jordanbeiber 7y ago

Here’s one example, not using any of these “templating turned programing languages” tools: We build an api and a cli util for these things. The api takes care of providing sane deployment manifests based on data from a service registry, which is populated using the cli. Cli can be run non-interactive for use in pipeline and other automations. The cli and api basically do any config and deplyment related tasks in concert with service discovery (which at this time is consul). Edit, rant: For automations json just owns yaml in all the ways! This yaml templating business should stop! I think it’s silly tbh. :) Our base k8s setup consisted of roughly 40k lines of yaml. Yaml is awesome for 10 lines of user input configuration, not 100, let alone 40k machine printed stuff. What happened here in this brave new DevOps world?!

maineldc 7y ago

For integrating k8s deploy into CI / CD, I really like Krane [0]. It has several key features: 1. A way to templatize YAML (though it could be used with Tanka too - the render command is split from the deploy command for this reason) 2. Monitors the rollout of resources - it's possible to detect successful and failed deployments more easily. 3. A way to run one-off commands during deployment. 4. Uses kubectl under the hood so easy to add to any workflow. It has other features that I don't use including secrets management. Net net - I definitely think it could elegantly replace a bunch of bash scripts that do kubectl things. [0] https://github.com/Shopify/krane https://github.com/Shopify/krane

transect 7y ago

For CI (testing) we use tekton [1] to run tests and pass around artifacts inside k8s. You can kick off and monitor tekton builds with their CLI tool, but we ended up building our own create/monitor/download-artifacts tool for it in ruby. We use an off the shelf CI server to kick that tool off, and it dumps back results the CI server can understand. One of the things we need to do is elastically scale the number of tasks (basically pods in tekton) that comprise our test suite run. This might be based on cluster utilization or whether it's the master branch. Since we have a single threaded test suite, we hoist parallelism up to the k8s level by breaking apart the tests into partitions each run by its own pod. For this we just render processed and parameterized erb to yaml. Eventually we'll dispense with yaml altogether and programmatically construct resources using a k8s REST api client. We haven't moved into CD with any of this tooling yet. [1] https://github.com/tektoncd/pipeline https://github.com/tektoncd/pipeline

shorez 7y ago

(Tanka dev here) I think this highly depends on your deployment process .. do you want full continuous deployment (CI deploying to the cluster)? In this case you could continue to use for example Jenkins to run `tk apply <environment>` on each merged PR. Another option would be to use an in-cluster CD agent (for example https://fluxcd.io/ https://fluxcd.io/), which uses Tanka to generate the yaml and applies it. Flux can be used with Tanka, needs some setup though: https://docs.fluxcd.io/en/1.17.0/references/fluxyaml-config-files.html https://docs.fluxcd.io/en/1.17.0/references/fluxyaml-config-.... I guess we could simplify this in the future. Feel free to reach out to me on Slack http://slack.raintank.io/ http://slack.raintank.io/ in the #tanka channel :D