16 ms·

I liked Mesos when I worked on it, and it's been replaced by more modern tools like Nomad, but every time I have to work on k8s it's .. well it's being promoted

I liked Mesos when I worked on it, and it's been replaced by more modern tools like Nomad, but every time I have to work on k8s it's .. well it's being promoted like a cult, has a cult following and I think the whole thing is set up to suck up complexity and layering. How do you deploy a thing to run on k8s? One would think you deploy a manifest to it and that's it. Like yaml or json or hcl or whatever. No. The built in thing is not good so someone wrote Helm. So you deploy helm manifests on there? No. Helm is not good either, you need helmsman or helmfile to further abstract things. How do you do networking? Layer a service mesh with custom overlay networking that abstract upon k8s clusterip. jeesh. why?

jjeaff 5y ago
You don't need any of that stuff. When I started with k8s, all I used were Yaml manifest files and kubectl apply. As I started using it more, i eventually moved up to using helm. I've been running production k8s for a few years now and haven't used helmsman or anything but helm yet.
- musingsole 5y ago
  > When I started with k8s But that's the problem right there. Shop after shop has invested in kubernetes with a team that was just learning it. And then they layered tool after tool to help massage pain points -- of which kubernetes has plenty. That leaves us at today where every shop effectively has a custom stack of kubernetes infrastructure cobbled together of various tools and philosophies. It's the same problem as JavaScript. There's nothing terrible about the technology -- on the contrary it's amazing stuff! But the context of how it gets used leads to a type of gridlock of opinionated tooling published just last week.
- smoyer 5y ago
  At its core, Kubernetes is a controller manager. It's true that most Kubernetes systems include a lot of the Hashicorp equivalents but you could in theory remove the controllers and associated CRDs. Kubernetes has gradually moved to what I think is a relatively sensible set of defaults which you can get in different flavors (k0s, k3s, microkube, kubespray, etc.) The comment about development and packaging tools can indeed present a problem. I tend to favor Helm for applications that need to be deployed more than once and Kustomize to differentiate between environments but I've definitely seen what I would consider horror stories. Even if you add a bit of GitOps, it's not too hard to understand. The horror stories seem to occur when teams try to map their previous workflows directly to k8s. Many times the worst k8s development workflows are created to enable teams that had broken workflows without k8s.
heinrichhartman 5y ago
> How do you deploy a thing to run on k8s? kubectrl apply -f deploy.yaml should work, no? What forces you to use the sugar-coating?
- sheeshkebab 5y ago
  Maybe bc no one uses that in reality and use helm instead?
  - saynay 5y ago
    Helm is useful if you need your software to run in many different places, and is widely known. This is why you see so many projects offering Helm charts; because you see them, they are set up to run in many environments. There is no reason to use it for your own software if you just have a single cluster.
    - voidfunc 5y ago
      Helms a pile of garbage but this isn’t really the fault of Helm. This is an issue with the chart or a failure to read the documentation of the chart. People have got to stop just blindly running stuff off the internet.
    - coredog64 5y ago
      At a previous employer, they’re building k8s clusters not for developers but for their infrastructure teams. In the past, where a vendor might have supplied an OVF file as the distributable product, they’re now providing Helm charts.
  - indigochill 5y ago
    I'll take that "no one" badge. I've never used Helm, always used kubectl. Well, until now, when I'm using Terraform (particularly helpful relative to kubectl when I need to provision cloud resources directly), but I've still never used Helm.
  - cassianoleal 5y ago
q3k 5y ago
> How do you deploy a thing to run on k8s? kubectl apply -f ~/git/infra/secretproject/prod.{json,yaml} One JSON/YAML file too uwieldy? Generate it using jsonnet/CUE/dhall/your favourite programming language. Or just talk directly to the Kubernetes API. You don't have to use Helm - in fact, you probably shouldn't be using Helm (as the whole idea of text templating YAML is... thoroughly ignorant in understanding what Kubernetes actually is). > Layer a service mesh with custom overlay networking that abstract upon k8s clusterip. You don't have to, and you probably don't need to. I'm happily running bare metal k8s production without anything more than Calico, Metallb and nginx-ingress-controller. That effectively gives me 'standard Kubernetes networking' ie. working Services and Ingresses which is plenty enough. And if you're using a cluster deployed by a cloud provider, you don't have to worry about any of this, all the hard decisions are made and the components are deployed and working.
- LambdaComplex 5y ago
  > the whole idea of text templating YAML is... thoroughly ignorant in understanding what Kubernetes actually is Could you expand on that? It sounds like an interesting position (bordering on the philosophical), but I don't know enough about Kubernetes to gauge its accuracy for myself.
  - cogman10 5y ago
    k8s is built for extension with validation of the extensions you add ( https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/ https://kubernetes.io/docs/concepts/extend-kubernetes/api-ex... ) Helm is just sort of dumb text manipulation with a TINY bit of deployment management built on top of it. There isn't really a whole lot that helm buys over extending k8s.
  - q3k 5y ago
    There's two things: 1) Text templating YAML is just bad. It's the serialized format of some structured data - instead of templating its text representation (and dealing with quoting, nindent, stringly-typed variables, and general YAML badness), just manipulate the structures directly before serializing them. For example, write some Python code that composes these structures in memory and then just serializes them to YAML before passing it over to kubectl. You can also use ready made tooling that uses domain-specific languages well suited to manipulating and emitting such structured configuration, like jsonnet/kubecfg (or CUE/Dhall/Nix combined with kubectl apply -f). Eschewing text templating is not only more productive by letting you build abstractions (eg. to deal with Kuberenetes' verbosity with labels/selectors, etc.), it also allows you to actually compose more complex deployments together, like 'this wordpress library uses this mariadb library' or 'this mariadb library can take a list of objects that will be upserted as users on startup'. 2) Those YAML/JSON manifests aren't even that native to Kubernetes. A lot of things go behind the scenes to actually upsert a manifest, as Kubernetes' resource/object model isn't nearly as straightforward as 'apply this YAML document full of stuff' would indicate (there's state to mix in, API/schema changes, optional/default fields, changes/annotations by other systems...). With k8s' Server-Side-Apply this can now be fairly transparent and you can pretend this is the case, but earlier tooling definitely had to be smarter in order to apply changes. Things like doing structure-level diffs between the previously applied intent and the current intent and the current state in order to build a set of mutations to apply. What this means is that Helm's entire 'serialized as YAML, manipulated at text level' stage is not only harmful and a pain in the neck to work with (see 1.) but also unnecessary (as 'flat YAML file' isn't any kind of canonical, ready-to-use representation that Helm had to use).
thefounder 5y ago
>> jeesh. why? If you look at Google they build a whole PaaS (Cloud Run/Appengine) on top of k8 and I guess that's the way it's meant to be used. IaaS -> K8 -> PaaS -> deploy.sh
- holografix 5y ago
  AppEngine does not run on K8s. Cloud Run _can_ run on GKE with Cloud Run for Anthos or oss kNative
clvx 5y ago
The worst part of k8s is definitely its configuration system. I don’t like helm as it’s combining packaging, templating and deployment lifecycle. I feel like each of this components should’ve been its own abstraction.
- q3k 5y ago
  It should've been, and it can be. There's nothing official or special about Helm, just don't use it.
- p_l 5y ago
  So just don't use Helm. It's not a core, crucial, or required part of k8s, it's just a reasonably popular related project... That I usually ban from environments before the rot can take root.
  - moljac024 5y ago
    what are the alternatives?
EdwardDiego 5y ago
Nothing about K8s forces you to use Helm or istio etc.
- musingsole 5y ago
  No, the developers about k8s force you to use Helm, istio, or etc. And then tech debt makes the decision unchangeable.
  - EdwardDiego 5y ago
    YMMV, I guess.
dikei 5y ago
> How do you do networking? Layer a service mesh with custom overlay networking that abstract upon k8s clusterip. Overlay network is easy if you don't need your pod and service IP to be routable from outside the cluster. It took me less than 1 afternoon to learn to use Kubespray to deploy an on-premise cluster with Calico for overlay network.
- scns 5y ago
  Zero snark, but you confirm parent somehow IMHO. Two more tools.
  - dikei 5y ago
    Well, Kubespray is just an Ansible-based installer that almost walk you through the process of setting up a K8S cluster. And Calico is the default option in Kubespray, so unless you want more customization, you don't need to learn anything about it.
peterthehacker 5y ago
So much flamebait. These kinds of comments aren’t good for anyone.
wvh 5y ago
Some parts of Kubernetes are perhaps unnecessary complex and we should keep our eyes open for alternatives and learn from different approaches, but deploying to Kubernetes really does not have to be that difficult unless you make it so. Plain manifests in a Git repository will do, and let something like Flux apply them for consistency. It really isn't harder than SSH and ad-hoc scripts or Ansible.
throwaway923840 5y ago
> jeesh. why? Poor design. It's been cobbled together slowly over the past 7 years. They built it to work only a certain way at first. But then somebody said, "oh, actually we forgot we need this other component too". But it didn't make sense with the current design. So they'd strap the new feature to the side with duct tape. Then they'd need another feature, and strap something else to the first strapped-on thing. Teams would ask for a new feature, and it would be shimmed in to the old design. All the while the design was led by an initial set of "opinionated" decisions that were inherently limiting. The end result is a mess that needs more and more abstraction. But this isn't a problem when you are already a billion-dollar company that can pay for other teams to build more abstractions and internal tools to deal with it. This is business as usual in any quasi-monolithic Enterprise project. Typically the way you escape that scenario is to have major refactors where you can overhaul the design and turn your 50 abstractions into 5. But instead, K8s decided to have very small release and support windows. This way they can refactor things and obsolete features within a short time frame. The end result is you either stick with one version for eternity (a security/operability problem) or you will be upgrading and deal with breaking changes every year for eternity.
birdyrooster 5y ago
Because you have different team members managing different aspects of your software's deployment (if you have complex systems that you have moved into kubernetes). You can have a team for: authnz/rbac, networking, containerization/virtualization, service load balancing, app development, and secrets all while no one is stepping on each others toes deploying changes while using common language to talk about resources and automation.
still_grokking 5y ago
> but every time I have to work on k8s it's .. well it's being promoted like a cult, has a cult following I would call this grass roots marketing. Some companies are very good at it. Maybe that ones have more know-how in the ad space than others?
shaklee3 5y ago
At some point you need to take a few minutes to learn new technologies in a tech career. Kubernetes is just not that hard if you spend a little time understanding it. You don't need a service mesh. Calico is very simple, or you can use cilium for something more advanced

jjeaff 5y ago

You don't need any of that stuff. When I started with k8s, all I used were Yaml manifest files and kubectl apply. As I started using it more, i eventually moved up to using helm. I've been running production k8s for a few years now and haven't used helmsman or anything but helm yet.

musingsole 5y ago
> When I started with k8s But that's the problem right there. Shop after shop has invested in kubernetes with a team that was just learning it. And then they layered tool after tool to help massage pain points -- of which kubernetes has plenty. That leaves us at today where every shop effectively has a custom stack of kubernetes infrastructure cobbled together of various tools and philosophies. It's the same problem as JavaScript. There's nothing terrible about the technology -- on the contrary it's amazing stuff! But the context of how it gets used leads to a type of gridlock of opinionated tooling published just last week.
smoyer 5y ago
At its core, Kubernetes is a controller manager. It's true that most Kubernetes systems include a lot of the Hashicorp equivalents but you could in theory remove the controllers and associated CRDs. Kubernetes has gradually moved to what I think is a relatively sensible set of defaults which you can get in different flavors (k0s, k3s, microkube, kubespray, etc.) The comment about development and packaging tools can indeed present a problem. I tend to favor Helm for applications that need to be deployed more than once and Kustomize to differentiate between environments but I've definitely seen what I would consider horror stories. Even if you add a bit of GitOps, it's not too hard to understand. The horror stories seem to occur when teams try to map their previous workflows directly to k8s. Many times the worst k8s development workflows are created to enable teams that had broken workflows without k8s.

heinrichhartman 5y ago

> How do you deploy a thing to run on k8s? kubectrl apply -f deploy.yaml should work, no? What forces you to use the sugar-coating?

sheeshkebab 5y ago
Maybe bc no one uses that in reality and use helm instead?
- saynay 5y ago
  Helm is useful if you need your software to run in many different places, and is widely known. This is why you see so many projects offering Helm charts; because you see them, they are set up to run in many environments. There is no reason to use it for your own software if you just have a single cluster.
  - voidfunc 5y ago
    Helms a pile of garbage but this isn’t really the fault of Helm. This is an issue with the chart or a failure to read the documentation of the chart. People have got to stop just blindly running stuff off the internet.
  - coredog64 5y ago
    At a previous employer, they’re building k8s clusters not for developers but for their infrastructure teams. In the past, where a vendor might have supplied an OVF file as the distributable product, they’re now providing Helm charts.
- indigochill 5y ago
  I'll take that "no one" badge. I've never used Helm, always used kubectl. Well, until now, when I'm using Terraform (particularly helpful relative to kubectl when I need to provision cloud resources directly), but I've still never used Helm.
- cassianoleal 5y ago

q3k 5y ago

> How do you deploy a thing to run on k8s? kubectl apply -f ~/git/infra/secretproject/prod.{json,yaml} One JSON/YAML file too uwieldy? Generate it using jsonnet/CUE/dhall/your favourite programming language. Or just talk directly to the Kubernetes API. You don't have to use Helm - in fact, you probably shouldn't be using Helm (as the whole idea of text templating YAML is... thoroughly ignorant in understanding what Kubernetes actually is). > Layer a service mesh with custom overlay networking that abstract upon k8s clusterip. You don't have to, and you probably don't need to. I'm happily running bare metal k8s production without anything more than Calico, Metallb and nginx-ingress-controller. That effectively gives me 'standard Kubernetes networking' ie. working Services and Ingresses which is plenty enough. And if you're using a cluster deployed by a cloud provider, you don't have to worry about any of this, all the hard decisions are made and the components are deployed and working.

LambdaComplex 5y ago
> the whole idea of text templating YAML is... thoroughly ignorant in understanding what Kubernetes actually is Could you expand on that? It sounds like an interesting position (bordering on the philosophical), but I don't know enough about Kubernetes to gauge its accuracy for myself.
- cogman10 5y ago
  k8s is built for extension with validation of the extensions you add ( https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/ https://kubernetes.io/docs/concepts/extend-kubernetes/api-ex... ) Helm is just sort of dumb text manipulation with a TINY bit of deployment management built on top of it. There isn't really a whole lot that helm buys over extending k8s.
- q3k 5y ago
  There's two things: 1) Text templating YAML is just bad. It's the serialized format of some structured data - instead of templating its text representation (and dealing with quoting, nindent, stringly-typed variables, and general YAML badness), just manipulate the structures directly before serializing them. For example, write some Python code that composes these structures in memory and then just serializes them to YAML before passing it over to kubectl. You can also use ready made tooling that uses domain-specific languages well suited to manipulating and emitting such structured configuration, like jsonnet/kubecfg (or CUE/Dhall/Nix combined with kubectl apply -f). Eschewing text templating is not only more productive by letting you build abstractions (eg. to deal with Kuberenetes' verbosity with labels/selectors, etc.), it also allows you to actually compose more complex deployments together, like 'this wordpress library uses this mariadb library' or 'this mariadb library can take a list of objects that will be upserted as users on startup'. 2) Those YAML/JSON manifests aren't even that native to Kubernetes. A lot of things go behind the scenes to actually upsert a manifest, as Kubernetes' resource/object model isn't nearly as straightforward as 'apply this YAML document full of stuff' would indicate (there's state to mix in, API/schema changes, optional/default fields, changes/annotations by other systems...). With k8s' Server-Side-Apply this can now be fairly transparent and you can pretend this is the case, but earlier tooling definitely had to be smarter in order to apply changes. Things like doing structure-level diffs between the previously applied intent and the current intent and the current state in order to build a set of mutations to apply. What this means is that Helm's entire 'serialized as YAML, manipulated at text level' stage is not only harmful and a pain in the neck to work with (see 1.) but also unnecessary (as 'flat YAML file' isn't any kind of canonical, ready-to-use representation that Helm had to use).

thefounder 5y ago

>> jeesh. why? If you look at Google they build a whole PaaS (Cloud Run/Appengine) on top of k8 and I guess that's the way it's meant to be used. IaaS -> K8 -> PaaS -> deploy.sh

holografix 5y ago
AppEngine does not run on K8s. Cloud Run _can_ run on GKE with Cloud Run for Anthos or oss kNative

clvx 5y ago

The worst part of k8s is definitely its configuration system. I don’t like helm as it’s combining packaging, templating and deployment lifecycle. I feel like each of this components should’ve been its own abstraction.

q3k 5y ago
It should've been, and it can be. There's nothing official or special about Helm, just don't use it.
p_l 5y ago
So just don't use Helm. It's not a core, crucial, or required part of k8s, it's just a reasonably popular related project... That I usually ban from environments before the rot can take root.
- moljac024 5y ago
  what are the alternatives?

EdwardDiego 5y ago

Nothing about K8s forces you to use Helm or istio etc.

musingsole 5y ago
No, the developers about k8s force you to use Helm, istio, or etc. And then tech debt makes the decision unchangeable.
- EdwardDiego 5y ago
  YMMV, I guess.

dikei 5y ago

> How do you do networking? Layer a service mesh with custom overlay networking that abstract upon k8s clusterip. Overlay network is easy if you don't need your pod and service IP to be routable from outside the cluster. It took me less than 1 afternoon to learn to use Kubespray to deploy an on-premise cluster with Calico for overlay network.

scns 5y ago
Zero snark, but you confirm parent somehow IMHO. Two more tools.
- dikei 5y ago
  Well, Kubespray is just an Ansible-based installer that almost walk you through the process of setting up a K8S cluster. And Calico is the default option in Kubespray, so unless you want more customization, you don't need to learn anything about it.

peterthehacker 5y ago

So much flamebait. These kinds of comments aren’t good for anyone.

wvh 5y ago

Some parts of Kubernetes are perhaps unnecessary complex and we should keep our eyes open for alternatives and learn from different approaches, but deploying to Kubernetes really does not have to be that difficult unless you make it so. Plain manifests in a Git repository will do, and let something like Flux apply them for consistency. It really isn't harder than SSH and ad-hoc scripts or Ansible.

throwaway923840 5y ago

> jeesh. why? Poor design. It's been cobbled together slowly over the past 7 years. They built it to work only a certain way at first. But then somebody said, "oh, actually we forgot we need this other component too". But it didn't make sense with the current design. So they'd strap the new feature to the side with duct tape. Then they'd need another feature, and strap something else to the first strapped-on thing. Teams would ask for a new feature, and it would be shimmed in to the old design. All the while the design was led by an initial set of "opinionated" decisions that were inherently limiting. The end result is a mess that needs more and more abstraction. But this isn't a problem when you are already a billion-dollar company that can pay for other teams to build more abstractions and internal tools to deal with it. This is business as usual in any quasi-monolithic Enterprise project. Typically the way you escape that scenario is to have major refactors where you can overhaul the design and turn your 50 abstractions into 5. But instead, K8s decided to have very small release and support windows. This way they can refactor things and obsolete features within a short time frame. The end result is you either stick with one version for eternity (a security/operability problem) or you will be upgrading and deal with breaking changes every year for eternity.

birdyrooster 5y ago

Because you have different team members managing different aspects of your software's deployment (if you have complex systems that you have moved into kubernetes). You can have a team for: authnz/rbac, networking, containerization/virtualization, service load balancing, app development, and secrets all while no one is stepping on each others toes deploying changes while using common language to talk about resources and automation.

still_grokking 5y ago

> but every time I have to work on k8s it's .. well it's being promoted like a cult, has a cult following I would call this grass roots marketing. Some companies are very good at it. Maybe that ones have more know-how in the ad space than others?

shaklee3 5y ago

At some point you need to take a few minutes to learn new technologies in a tech career. Kubernetes is just not that hard if you spend a little time understanding it. You don't need a service mesh. Calico is very simple, or you can use cilium for something more advanced