4 ms·

Running kube is the most trivial part, especially with some of these tools like k3s/k0s. Hard part is deployments. Managing Kube's raw yaml files is tedious. W

Running kube is the most trivial part, especially with some of these tools like k3s/k0s. Hard part is deployments. Managing Kube's raw yaml files is tedious. What tools are y'all using there? I am having a tough time with Helm, but I feel like that is the standard answer these days. Back in ~2016 at FarmLogs we built and operated our own (Clojure!) deployment system that was backed by a Github repository. Developers worked with a DSL that was really just a slimmed down version of Kube yaml. This eliminated lots of boilerplate thanks to our company-wide conventions for building microservices. The master branch reflected the intended state of the cluster. It was easy to reason about. I have not found a modern/open-source tool that is similar, but I have to imagine it exists.

marcc 5y ago
I agree that "apps" and deploying to the cluster is the hard part. Helm is definitely the most popular choice, but there are others. kpt: https://github.com/GoogleContainerTools/kpt https://github.com/GoogleContainerTools/kpt kapp: https://carvel.dev/kapp/ https://carvel.dev/kapp/ cnab: https://cnab.io https://cnab.io There are many (many) more. But these are somewhat popular recently.
- whalesalad 5y ago
  Thank you for the links. I had not heard of cnab until now but that looks really interesting. I will look into all of these.
- ithkuil 5y ago
  I made a variation on the theme of kpt, you may find interesting: https://github.com/mkmik/knot8 https://github.com/mkmik/knot8 (Rendered manpage at https://knot8.io/ https://knot8.io/, if you like that exposition format)
q3k 5y ago
> Hard part is deployments. Managing Kube's raw yaml files is tedious. What tools are y'all using there? I am having a tough time with Helm, but I feel like that is the standard answer these days. kubecfg (with jsonnet) or CUE. Gives you actual composability and a powerful programming language for reducing boilerplate. Oh, and no text templating. With kubecfg deploying/updating is: kubecfg update foo.jsonnet No need to first build any Helm images or for any scaffolding. Just get `kubecfg` on your PATH and you're done. Version things the same way you version the rest of your code. Build your own abstractions that solve your particular problems as you go, instead of relying on someone's idea on how your deployments should work. Perfect for building orga-wide microservice deployment boilerplate.
dharmab 5y ago
For some more complex deployments (i.e. not simple batch jobs or web services), we've used the Kubernetes API directly from programming languages like Python or Go. Often as a controller running within the cluster. That allows us the full data structures toolset and freedom of a programming language. It reminds me of the old article "Beating the Averages": http://www.paulgraham.com/avg.html http://www.paulgraham.com/avg.html
- whalesalad 5y ago
  Afraid to admit I had not even considered that idea, good thinking.
- q3k 5y ago
  Yeah, just writing some code that talks to the Kubernetes API really ought to be more common. It's not a great API, but it certainly beats writing bash scripts that shell out to kubectl and yq. In the same vein, I wish people knew that YAML isn't even really a native Kubernetes thing, it's just one of the many ways you can serialize k8s objects...
  - dharmab 5y ago
    In my experience, engineers with a formal education in Data Structures and Algorithms tend "get it" in a way that others don't- that the underlying data structure is a set of maps/structs and YAML, JSON, etc. are just text formats.
  - sascha_sl 5y ago
    > writing bash scripts that shell out to kubectl and yq We did this for a while, from a container rendering out our stack in Go, when server-side apply was not a reality yet. The main issue is that lifecycle management of resources beyond writing code for a single deployment is hard to do with the typed clients, and the dynamic client used to be even more of a mess than it is now. There's a lot of implicit magic happening, baked into kubectl in a non-reusable way you just couldn't replicate before 1.16.
    - 5y ago
oweiler 5y ago
I've found kustomize to be really nice. Much less powerful than Helm, easy to learn, good enough for most tasks. https://github.com/kubernetes-sigs/kustomize https://github.com/kubernetes-sigs/kustomize
mbushey 5y ago
Helm is horrible. Please use Kustomize. Life will get better, it did for me. I also use Sops (with goabout/kustomize-sopssecretgenerator) and ArgoCD for a very slick well integrated system. The CRDs in ArgoCD are so well done that it feels like native K8s CD.
- booleanbetrayal 5y ago
  Big fan of ArgoCD here. The development team is also very responsive with good processes in place, from my experience in the repo.
- mfer 5y ago
  Kustomize and Helm take different routes. Kustomize expects the user to know Kubernetes resources and their schemas. Helm and charts target the chart creator being being an expert in Kubernetes while the chart consumer (the Helm user) doesn't need to be an expert. They can have a simple interface to deploy an application. One of the issues for those running applications in Kubernetes is the complexity. Kubernetes is complex. So much so that people like Kelsey Hightower have said Kubernetes is a platform for creating platforms. For most people to learn Kubernetes will slow down their velocity which is a negative. Most people need a simpler system on top of Kubernetes to make them effective.
  - sascha_sl 5y ago
    Helm itself is pretty awful through. For instance, when people upgraded to 1.16, where Kubernetes retired apps/v1beta1, Kubernetes automatically transparently converts these types. Helm, having once created these resources as v1beta1 will completely refuse to touch the deployment. Without some specialized extra tooling that people created in the wake of this, resolving this requires deleting the entire namespace and reinstalling the chart. Issue on helm repo closed as wontfix.
    - mfer 5y ago
      This still comes down to assumptions. If someone was using raw kubernetes manifests instead of Helm and charts they would need to know the schemas and details of those manifests. Then, when 1.16 came out they would need to update to the new apiVersion and modify their manifests for any differences in schema. This requires someone who knows and understands k8s resources. This IS NOT a typical app dev or person who wants to run apps in k8s. I've been in numerous circles of app folks who have complained about this expectation from the k8s community. On the main Helm project they have the stance that what's in a chart is up to the charts authors. Just like what's in a debian package is what's up to the debian package authors. That those folks should do a good job maintaining them. That's why it was a wontfix issue. What we see as an underlying issue with k8s is the experience that app devs and people who want to run apps have to deal with. It's currently like expecting an app dev to write in assembly and know when assembly codes are deprecated or changed. It's in the pre-high level language phase. This comparison to assembly is one that's come out of the k8s community itself.
jrsdav 5y ago
Have you heard of Flux? It's not a manifest abstraction tool per se (like Helm or Kustomize), but automates the management of your manifests using GitOps. It sounds like it might be what you're looking for. https://fluxcd.io/docs/concepts/ https://fluxcd.io/docs/concepts/
gray_-_wolf 5y ago
We are using tanka [0] and are failry happy with it. The jsonnet language allows nice layer of abstraction while still allowing to directly generate kubernetes resources when necessary. We wrote in-house library to map (very) simplified .jsonnet file to kuberenetes deployment; after all, 90% deployements are mostly same and differs in minor details only. Seems to work reasonably well. 0: https://github.com/grafana/tanka https://github.com/grafana/tanka
luxurytent 5y ago
This is how I feel too. At my workplace, we can easily spin up a kube cluster locally but our Helm charts aren't transportable for local development. They can be modified, torn apart, extracted from, but we don't have a seamless and elegant way to say "Hey, this thing in production, that's running via Helm/k8s, I want the same thing locally ... with these environment variables adjusted" I don't think that's a difficult thing to accomplish and there is appetite from others I work with to do so, but there's _effort_ behind it. Maybe there are smarter ways and we just don't know it yet.
birdyrooster 5y ago
jsonnet, works great and is composable however you like
stormbrew 5y ago
> Back in ~2016 at FarmLogs we built and operated our own (Clojure!) deployment system that was backed by a Github repository. Developers worked with a DSL that was really just a slimmed down version of Kube yaml. This eliminated lots of boilerplate thanks to our company-wide conventions for building microservices. The master branch reflected the intended state of the cluster. It was easy to reason about. The lack of this is what frustrates me so much about all of these things. Writing some bundle of yaml but just telling the management software "here go run this 'app'" with no state or version management of it makes me feel like I'm running a stack like it's a windows desktop. Just random shit everywhere and I don't know what any of it is. Maybe you're keeping it in a git repo but is what's running live what's in the git repo? Who the heck knows. All feels like a huge step backwards to me. Let me use version control as a source of truth, not just a spot to throw things in.
- topspin 5y ago
  "Maybe you're keeping it in a git repo but is what's running live what's in the git repo? Who the heck knows." See Argo CD.
mushishi 5y ago
In previous project, we had Kotlin DSL based definitions, and even though I only read it, it seemed to make a lot of sense: static typing, and reuse of code e.g., between QA and production environment definitions. I think it may have been this: https://github.com/fkorotkov/k8s-kotlin-dsl https://github.com/fkorotkov/k8s-kotlin-dsl

marcc 5y ago

I agree that "apps" and deploying to the cluster is the hard part. Helm is definitely the most popular choice, but there are others. kpt: https://github.com/GoogleContainerTools/kpt https://github.com/GoogleContainerTools/kpt kapp: https://carvel.dev/kapp/ https://carvel.dev/kapp/ cnab: https://cnab.io https://cnab.io There are many (many) more. But these are somewhat popular recently.

whalesalad 5y ago
Thank you for the links. I had not heard of cnab until now but that looks really interesting. I will look into all of these.
ithkuil 5y ago
I made a variation on the theme of kpt, you may find interesting: https://github.com/mkmik/knot8 https://github.com/mkmik/knot8 (Rendered manpage at https://knot8.io/ https://knot8.io/, if you like that exposition format)

q3k 5y ago

> Hard part is deployments. Managing Kube's raw yaml files is tedious. What tools are y'all using there? I am having a tough time with Helm, but I feel like that is the standard answer these days. kubecfg (with jsonnet) or CUE. Gives you actual composability and a powerful programming language for reducing boilerplate. Oh, and no text templating. With kubecfg deploying/updating is: kubecfg update foo.jsonnet No need to first build any Helm images or for any scaffolding. Just get `kubecfg` on your PATH and you're done. Version things the same way you version the rest of your code. Build your own abstractions that solve your particular problems as you go, instead of relying on someone's idea on how your deployments should work. Perfect for building orga-wide microservice deployment boilerplate.

dharmab 5y ago

For some more complex deployments (i.e. not simple batch jobs or web services), we've used the Kubernetes API directly from programming languages like Python or Go. Often as a controller running within the cluster. That allows us the full data structures toolset and freedom of a programming language. It reminds me of the old article "Beating the Averages": http://www.paulgraham.com/avg.html http://www.paulgraham.com/avg.html

whalesalad 5y ago
Afraid to admit I had not even considered that idea, good thinking.
q3k 5y ago
Yeah, just writing some code that talks to the Kubernetes API really ought to be more common. It's not a great API, but it certainly beats writing bash scripts that shell out to kubectl and yq. In the same vein, I wish people knew that YAML isn't even really a native Kubernetes thing, it's just one of the many ways you can serialize k8s objects...
- dharmab 5y ago
  In my experience, engineers with a formal education in Data Structures and Algorithms tend "get it" in a way that others don't- that the underlying data structure is a set of maps/structs and YAML, JSON, etc. are just text formats.
- sascha_sl 5y ago
  > writing bash scripts that shell out to kubectl and yq We did this for a while, from a container rendering out our stack in Go, when server-side apply was not a reality yet. The main issue is that lifecycle management of resources beyond writing code for a single deployment is hard to do with the typed clients, and the dynamic client used to be even more of a mess than it is now. There's a lot of implicit magic happening, baked into kubectl in a non-reusable way you just couldn't replicate before 1.16.
  - 5y ago

oweiler 5y ago

I've found kustomize to be really nice. Much less powerful than Helm, easy to learn, good enough for most tasks. https://github.com/kubernetes-sigs/kustomize https://github.com/kubernetes-sigs/kustomize

mbushey 5y ago

Helm is horrible. Please use Kustomize. Life will get better, it did for me. I also use Sops (with goabout/kustomize-sopssecretgenerator) and ArgoCD for a very slick well integrated system. The CRDs in ArgoCD are so well done that it feels like native K8s CD.

booleanbetrayal 5y ago
Big fan of ArgoCD here. The development team is also very responsive with good processes in place, from my experience in the repo.
mfer 5y ago
Kustomize and Helm take different routes. Kustomize expects the user to know Kubernetes resources and their schemas. Helm and charts target the chart creator being being an expert in Kubernetes while the chart consumer (the Helm user) doesn't need to be an expert. They can have a simple interface to deploy an application. One of the issues for those running applications in Kubernetes is the complexity. Kubernetes is complex. So much so that people like Kelsey Hightower have said Kubernetes is a platform for creating platforms. For most people to learn Kubernetes will slow down their velocity which is a negative. Most people need a simpler system on top of Kubernetes to make them effective.
- sascha_sl 5y ago
  Helm itself is pretty awful through. For instance, when people upgraded to 1.16, where Kubernetes retired apps/v1beta1, Kubernetes automatically transparently converts these types. Helm, having once created these resources as v1beta1 will completely refuse to touch the deployment. Without some specialized extra tooling that people created in the wake of this, resolving this requires deleting the entire namespace and reinstalling the chart. Issue on helm repo closed as wontfix.
  - mfer 5y ago
    This still comes down to assumptions. If someone was using raw kubernetes manifests instead of Helm and charts they would need to know the schemas and details of those manifests. Then, when 1.16 came out they would need to update to the new apiVersion and modify their manifests for any differences in schema. This requires someone who knows and understands k8s resources. This IS NOT a typical app dev or person who wants to run apps in k8s. I've been in numerous circles of app folks who have complained about this expectation from the k8s community. On the main Helm project they have the stance that what's in a chart is up to the charts authors. Just like what's in a debian package is what's up to the debian package authors. That those folks should do a good job maintaining them. That's why it was a wontfix issue. What we see as an underlying issue with k8s is the experience that app devs and people who want to run apps have to deal with. It's currently like expecting an app dev to write in assembly and know when assembly codes are deprecated or changed. It's in the pre-high level language phase. This comparison to assembly is one that's come out of the k8s community itself.

jrsdav 5y ago

Have you heard of Flux? It's not a manifest abstraction tool per se (like Helm or Kustomize), but automates the management of your manifests using GitOps. It sounds like it might be what you're looking for. https://fluxcd.io/docs/concepts/ https://fluxcd.io/docs/concepts/

gray_-_wolf 5y ago

We are using tanka [0] and are failry happy with it. The jsonnet language allows nice layer of abstraction while still allowing to directly generate kubernetes resources when necessary. We wrote in-house library to map (very) simplified .jsonnet file to kuberenetes deployment; after all, 90% deployements are mostly same and differs in minor details only. Seems to work reasonably well. 0: https://github.com/grafana/tanka https://github.com/grafana/tanka

luxurytent 5y ago

This is how I feel too. At my workplace, we can easily spin up a kube cluster locally but our Helm charts aren't transportable for local development. They can be modified, torn apart, extracted from, but we don't have a seamless and elegant way to say "Hey, this thing in production, that's running via Helm/k8s, I want the same thing locally ... with these environment variables adjusted" I don't think that's a difficult thing to accomplish and there is appetite from others I work with to do so, but there's _effort_ behind it. Maybe there are smarter ways and we just don't know it yet.

birdyrooster 5y ago

jsonnet, works great and is composable however you like

stormbrew 5y ago

> Back in ~2016 at FarmLogs we built and operated our own (Clojure!) deployment system that was backed by a Github repository. Developers worked with a DSL that was really just a slimmed down version of Kube yaml. This eliminated lots of boilerplate thanks to our company-wide conventions for building microservices. The master branch reflected the intended state of the cluster. It was easy to reason about. The lack of this is what frustrates me so much about all of these things. Writing some bundle of yaml but just telling the management software "here go run this 'app'" with no state or version management of it makes me feel like I'm running a stack like it's a windows desktop. Just random shit everywhere and I don't know what any of it is. Maybe you're keeping it in a git repo but is what's running live what's in the git repo? Who the heck knows. All feels like a huge step backwards to me. Let me use version control as a source of truth, not just a spot to throw things in.

topspin 5y ago
"Maybe you're keeping it in a git repo but is what's running live what's in the git repo? Who the heck knows." See Argo CD.

mushishi 5y ago

In previous project, we had Kotlin DSL based definitions, and even though I only read it, it seemed to make a lot of sense: static typing, and reuse of code e.g., between QA and production environment definitions. I think it may have been this: https://github.com/fkorotkov/k8s-kotlin-dsl https://github.com/fkorotkov/k8s-kotlin-dsl