4 ms·

I really want something that is Docker Compose but for Kubernetes. I mean that I can have a simple way to declaring resources in just like Docker Compose, but I

by stevefan1999 6mo ago

I really want something that is Docker Compose but for Kubernetes. I mean that I can have a simple way to declaring resources in just like Docker Compose, but I run the environment in Kubernetes so that I can get to test the behaviors when there are multiple copies of the softwares running together. I do rely on Kubernetes heavily for distributed and networked software deployment, so it is even better if we can emulate things like latency or burstable packet loss so that we can do a controlled chaos test for reliability test. I tried Skaffold, Tilt, Devspaces and Devpod/Coder v2, none of them are really simple like Compose.

nijave 6mo ago
Helm mostly does that. Not a huge fan of a text templating engine generating yaml but once you get your chart setup with a few variable inputs, you can continue using it for a bunch of other stuff with minimal new config. The inputs (values) are yaml so you can make it look exactly like a Docker Compose file if you want (wouldn't be surprised if there's some charts floating around that do that)
- calmoo 6mo ago
  I've recently been dipping my toes into k8s / kustomize / helm, and I recently had a situation where having a base deployment yml template that I wanted to reuse across various deployments. I had a look at Helm and I was frankly shocked how bad the templating was with Go templates, it was close to unreadable and felt very brittle!
  - freedomben 6mo ago
    I did that too, and ended up just skipping helm and using envsubst to interpolate the values I need at runtime from env vars. Nearly everyone preferred that approach. YMMV of course.
    - stackskipton 6mo ago
      BTW, this is what Flux V2 (GitOps) supports as well and we do at our company. It's worked well and covered almost all our use cases.
    - nijave 6mo ago
      I think that's a good approach if it works for your use case. Sometimes you might want something slightly more sophisticated like basic logic (loops/conditionals). In those cases, you can still use helm but you have an extremely simple template and avoid many of the "can't read this template" helm pitfalls.
  - regexorcist 6mo ago
    How so? I find it much the same to other templating engines like Jinja, though I'm definitely not a fan of the syntax. But that hardly matters anymore with LLMs.
    - deleted 6mo ago
      [deleted]
    - nijave 6mo ago
      I think it's a combination of text templating + yaml (which is whitespace delimited). When templating html with Jinja, it's not a big deal if the indentation isn't right. In helm, best case you get a syntax error, worst case you end up overwriting keys and producing something syntactically but not logically correct.
    - calmoo 6mo ago
      Specifically with YAML, Go templating requires some special operators to get anything useful done and it ends up being an unreadable mess.
  - nijave 6mo ago
    Yeah, that's fair. I don't think it's as bad if you make your own charts and can more liberally hardcode things. Community charts tend to have an insane amount of "knobs" so you can basically change everything being templated. I don't know if I'd necessarily call it brittle, though. You can use `helm template` and various linters to validate the generated yaml is correct (and use something like pre-commit to autorun)
  - mehackernewsacc 6mo ago
    Shared up the chain, but bringing your attention to it as well: https://nixidy.dev/ https://nixidy.dev/ solves all of these issues.
pmig 6mo ago
This is a problem we, as a company, have thought about a lot, but we always concluded that Kubernetes is already the simplest abstraction of a distributed system that is feasible for the diverse needs that the biggest companies out there have. We previously built a package manager for Kubernetes to abstract it in the simplest way possible `glasskube install app` but we failed because every abstraction needs to follow a "convention over configuration" pattern at some point. Also, we weren't able to monetize a package manager. With Distr (https://github.com/distr-sh/distr https://github.com/distr-sh/distr), we have actually been able to help companies not only package but distribute and either manage or give their customers a way to self-manage applications. Our customers are able to land on-premises contracts at enterprises way faster than before, which is also a clear ROI for paying for Distr. So, I don't think that you can get the flexibility of a distributed application orchestrator with a simple declarative YAML file if your target environments are diverse.
- freedomben 6mo ago
  Same, I've tried three or four times to make it work, including one attempt that just translated compose.yaml into k8s yaml, and every time I came away thinking, "just use k8s". K8s yaml looks complex, and can start to feel very boilerplate, but attempts to hide the complexity often just lead to something not-flexible-enough because it encodes convention over configuration, and inevitably some project runs into limitations and pretty soon you've just built an abstraction layer that leaks or is equally complex/verbose and now you have to learn something new. Just use k8s and follow similar patterns is the conclusion I've arrived at personally.
mehackernewsacc 6mo ago
You should take a look at https://nixidy.dev/ https://nixidy.dev/.
willswire 6mo ago
Have you looked into some of the Docker Compose Bridge transformers? We’re using it to scaffold out k8s apps for UDS from compose files https://github.com/defenseunicorns-labs/compose-bridge-uds/ https://github.com/defenseunicorns-labs/compose-bridge-uds/
suralind 6mo ago
This is a great idea for some vibe coding tbh and you can customize it to match exactly the resources that you need as everyone has slightly different requirements for their k8s stack. The simplest way to start probably would just be to create a custom Helm chart and define what you want in `values.yaml`.

nijave 6mo ago

Helm mostly does that. Not a huge fan of a text templating engine generating yaml but once you get your chart setup with a few variable inputs, you can continue using it for a bunch of other stuff with minimal new config. The inputs (values) are yaml so you can make it look exactly like a Docker Compose file if you want (wouldn't be surprised if there's some charts floating around that do that)

calmoo 6mo ago
I've recently been dipping my toes into k8s / kustomize / helm, and I recently had a situation where having a base deployment yml template that I wanted to reuse across various deployments. I had a look at Helm and I was frankly shocked how bad the templating was with Go templates, it was close to unreadable and felt very brittle!
- freedomben 6mo ago
  I did that too, and ended up just skipping helm and using envsubst to interpolate the values I need at runtime from env vars. Nearly everyone preferred that approach. YMMV of course.
  - stackskipton 6mo ago
    BTW, this is what Flux V2 (GitOps) supports as well and we do at our company. It's worked well and covered almost all our use cases.
  - nijave 6mo ago
    I think that's a good approach if it works for your use case. Sometimes you might want something slightly more sophisticated like basic logic (loops/conditionals). In those cases, you can still use helm but you have an extremely simple template and avoid many of the "can't read this template" helm pitfalls.
- regexorcist 6mo ago
  How so? I find it much the same to other templating engines like Jinja, though I'm definitely not a fan of the syntax. But that hardly matters anymore with LLMs.
  - deleted 6mo ago
    [deleted]
  - nijave 6mo ago
    I think it's a combination of text templating + yaml (which is whitespace delimited). When templating html with Jinja, it's not a big deal if the indentation isn't right. In helm, best case you get a syntax error, worst case you end up overwriting keys and producing something syntactically but not logically correct.
  - calmoo 6mo ago
    Specifically with YAML, Go templating requires some special operators to get anything useful done and it ends up being an unreadable mess.
- nijave 6mo ago
  Yeah, that's fair. I don't think it's as bad if you make your own charts and can more liberally hardcode things. Community charts tend to have an insane amount of "knobs" so you can basically change everything being templated. I don't know if I'd necessarily call it brittle, though. You can use `helm template` and various linters to validate the generated yaml is correct (and use something like pre-commit to autorun)
- mehackernewsacc 6mo ago
  Shared up the chain, but bringing your attention to it as well: https://nixidy.dev/ https://nixidy.dev/ solves all of these issues.

pmig 6mo ago

This is a problem we, as a company, have thought about a lot, but we always concluded that Kubernetes is already the simplest abstraction of a distributed system that is feasible for the diverse needs that the biggest companies out there have. We previously built a package manager for Kubernetes to abstract it in the simplest way possible `glasskube install app` but we failed because every abstraction needs to follow a "convention over configuration" pattern at some point. Also, we weren't able to monetize a package manager. With Distr (https://github.com/distr-sh/distr https://github.com/distr-sh/distr), we have actually been able to help companies not only package but distribute and either manage or give their customers a way to self-manage applications. Our customers are able to land on-premises contracts at enterprises way faster than before, which is also a clear ROI for paying for Distr. So, I don't think that you can get the flexibility of a distributed application orchestrator with a simple declarative YAML file if your target environments are diverse.

freedomben 6mo ago
Same, I've tried three or four times to make it work, including one attempt that just translated compose.yaml into k8s yaml, and every time I came away thinking, "just use k8s". K8s yaml looks complex, and can start to feel very boilerplate, but attempts to hide the complexity often just lead to something not-flexible-enough because it encodes convention over configuration, and inevitably some project runs into limitations and pretty soon you've just built an abstraction layer that leaks or is equally complex/verbose and now you have to learn something new. Just use k8s and follow similar patterns is the conclusion I've arrived at personally.

mehackernewsacc 6mo ago

You should take a look at https://nixidy.dev/ https://nixidy.dev/.

willswire 6mo ago

Have you looked into some of the Docker Compose Bridge transformers? We’re using it to scaffold out k8s apps for UDS from compose files https://github.com/defenseunicorns-labs/compose-bridge-uds/ https://github.com/defenseunicorns-labs/compose-bridge-uds/

suralind 6mo ago

This is a great idea for some vibe coding tbh and you can customize it to match exactly the resources that you need as everyone has slightly different requirements for their k8s stack. The simplest way to start probably would just be to create a custom Helm chart and define what you want in `values.yaml`.