7 ms·

I run k8s in production and I think this is a really bizarre axe to grind that sort of smells like someone who got upset by how steep the kubernetes learning cu

by ixtli 6y ago

I run k8s in production and I think this is a really bizarre axe to grind that sort of smells like someone who got upset by how steep the kubernetes learning curve is. Which, in a way, is understandable. > 1) Add hundreds of new failure modes to your software In my entirely anecdotal experience, it removes error modes. It turns out that just because k8s offers a feature (it offers many!) doesn't mean you're required to use it. > 2) Move you from writing portable software configuration to writing thousands of lines of k8s-specific YAML Portability, put another way, is a requirement to maintain n different deployment mechanisms because you have no meaningful control over the system on which your software is deployed. This is unavoidable depending on what it is you're deploying and where, but for those of us who have the ability to make those decisions the old notion of portability described by automake and others is actually a bad contract to agree to. That said, the YAML but is horrible and should be abolished. This is a great criticism. > 3) Ensnare you in a mesh of questionably-good1 patterns like containerization and software defined networking The author isn't considering that I am not, and never will be, Google's entire SRE team. I consider it a general win to be able to apply some subset of their encoded knowledge to the problem of keeping my services running with minimal downtime even if you pick up some k8s-specific baggage along the way.

jonfw 6y ago
What's wrong with the YAML? It's easy to read and write, concise, and generally has sane defaults meaning you don't have to be overly verbose.
- GordonS 6y ago
  I'm personally fine with yaml, but specifically k8s yaml config is definitely not concise! Compare roughly equivalent Compose/Swarm and k8s configs, and you'll see that the k8s one is about 8x bigger.
- ixtli 6y ago
  You know i actually agree with you but, as a manager of other engineers, people never stop complaining about having to edit large amounts of YAML that describe complex objects. At a certain point im forced to acknowledge that its just not ergonomic. Now, I will say that perhaps whats actually happening is the objects that the describe are so complex and numerous that the problem is actually that you need better tools, not that YAML as a format is bad. YAML as used to configure, say Open API specifications, is something people always make my problem. I don't really know why this is because I don't have an issue hacking this stuff up in VIM but people really just don't seem to like it.
- dharmab 6y ago
  The YAML spec has a lot of ambiguity in it which makes correctly parsing it surprisingly complicated. YAML 1.1 (the most commonly implemented version) also lacks a great way of declaring sets (i.e. unordered collections of unique values). I actually quite like most of the language a lot but the corner cases can be quite complicated.
- mixedCase 6y ago
  Of all things I would definitely not call it easy to read and write. It has super-weak (see this for examples: https://github.com/cblp/yaml-sucks https://github.com/cblp/yaml-sucks) dynamic typing, which coupled with requiring awful templating to get anything non-trivial done it has been the cause of many outages. For real world Kubernetes needs to be addressed I'd argue we need Dhall or something with Dhall-like semantics and goals to have first class support.
- regularfry 6y ago
  Spending 6 hours tracking down a `loadBalancerIp` that should have been a `loadBalancerIP` feels like a problem that shouldn't have happened.
  - atombender 6y ago
    YAML can be validated using JSON Schema/OpenAPI, and somebody does maintain Kubernetes schema definitions, including Kustomize, which can then be used in an IDE for live validation. For example, the Kubernetes extension for VSCode does this. If you use Helm, writing and validating schemas for Helm value files is trickier, and not something I believe anyone is doing. It should be easier to do. Ideally you should simply put this at the top of a value file: $schema: ./schema.yaml But from what I know, there's no editor/extension, that supports validating through such a declaration. Secondly, Helm doesn't have a way to statically analyze value access. Something like {{ .Values.missTypedKey }} will only be caught when you generate the manifests.
  - merb 6y ago
    actually k8s validates the yaml, welp it doesn't it validates the json. basically yaml is only the format the operators get, internally it uses json.
- jacques_chester 6y ago
  Mostly that it's a single projection of an underlying data model. Where the projection is a snug fit, great. But typically folks want sets of things and suddenly it's a bit chatty.
- outworlder 6y ago
  > What's wrong with the YAML? Nothing at all. Until you start 'templateizing' it (looking at you, Helm). It's definitely better than JSON (it can even have comments, imagine that). Most people don't bother reading the spec or even examples though, and don't realize how good it actually is.
  - chupasaurus 6y ago
    > It's definitely better than JSON k8s API supports JSON almost exclusively, basically one could write in Whitespace then convert to it.
  - GordonS 6y ago
    > Nothing at all. Until you start 'templateizing' it (looking at you, Helm). This scourge affects JSON too, such as Azure's ARM templates, which also crowbar logic into the format. An abomination!
  - ithkuil 6y ago
    I wish more people stuck with templating yaml would just issue a toJson instead of fiddling with indentation levels
- tapoxi 6y ago
  In the Kubernetes and Ansible/Salt world it's frequently templatized, and then you need to worry about indentation that you place or your template engine places. Every day I pray that Helm will incorporate support for jsonnet so I never need to touch YAML again.
  - jonfw 6y ago
    Why not parse the YAML- manipulate it like you would any other data structure in your language of choice, and then generate the YAML from there? I don't understand why you would want to directly edit markup language without parsing it. YAML's goal is to represent a data structure so that a human can interact with it. It accomplishes that goal well. If you want to edit that data structure programmatically- no markup language makes sense to use, represent the data structure using your programming language's native representations.
  - aliswe 6y ago
    I made a script for expressing similar deployments (ie homepage-en, homepage-de) in a variety of contexts (test, prod) and their permutations in a single, succinct json file. It's a bash script though. Example json: { "deployments": [ { "name": "site-en", "env": { "hello": "world" } } ], "targets": [ { "name": "dev", "context": "aks-dev", "env": { "ipsum": "dolor" } } ] }
    - atombender 6y ago
      You should look into Kustomize. It's built into Kubectl, and lets you do something like this in a much more modular way.
  - deathanatos 6y ago
    I agree with, and have felt the same pain as you here. But this isn't YAML's fault, IMO, it is that the templaters, like Helm, aren't safe, and don't escape their outputs. If have a string, and I use that in the middle of a YAML template, I expect a string — I don't expect "300" to become (integer) 300; give me a function for that conversion. Any HTML templating system that behaved this way would be frowned upon due to the potential for XSS. There's little risk of XSS here, but it's still annoying to get the wrong results.
- ruuda 6y ago
  The features intended to make it easy to use, in the end lead to surprising behavior, often with values having non-string types where you expected strings. There are some examples in https://www.arp242.net/yaml-config.html https://www.arp242.net/yaml-config.html.
  - jonfw 6y ago
    I haven't had any issues with this in a Kubernetes context- but I can see how this could lead to issues in different contexts
- billfor 6y ago
  Having to do conditional logic in it, like: matchExpressions: - key: app operator: In values: - myservice Rather than, if (app=="myservice")
  - jonfw 6y ago
    Why would you do this in a Kubernetes context?
    - billfor 6y ago
      To specify a podAffinityTerm, amongst others.
    - blandflakes 6y ago
      Your position here seems to be that YAML is a reasonable configuration language. It kinda is. But that's not how it's used in Kubernetes. It's used as a declarative DSL that you use to specify your deployments, and it is HORRIFIC. Extremely nested things where the equivalent actual code to do that would be shorter. Things that are inexpressible. Backwards-incompatible changes to your YAML that actually make it impossible to deploy changes that bring your YAML up to spec if your cluster's master node upgraded first. YAML gives nothing here. jsonnet really isn't a drastic improvement, IMO, except that it lets you express some repetitive parts without having to figure out how many times a template engine is going to indent your YAML.
- SrslyJosh 6y ago
  YAML actually has one of the worst default behaviors I've seen: automatic conversion of certain strings to bools. This wouldn't be a problem if YAML required that all strings be quoted...but it doesn't, and there's no "strict" mode to force quotes, so users are extremely likely to end up doing something that behaves unexpectedly. If you're lucky, you'll spot the problem quickly instead of wasting half a day on it.
- mixmastamyk 6y ago
  There is a strict yaml alternative with edge cases removed I recommend.
- vorpalhex 6y ago
  YAML is easy to read and write in tutorial documents. It becomes a complete mess in real life and whitespace based syntax is horrendous to debug. We have had endless heartaches due to minor yaml issues and today employ tooling to minimize the amount of yaml engineers write to avoid these issues.
- wbl 6y ago
  I once put an IPv6 address into YAML. It didn't survive the round trip and nearly caused an incident.
shadowgovt 6y ago
It's always worth noting that k8s isn't designed to give outsiders the power of Borg the way Google SREs have the power of Borg. it's designed to give outsiders something better, based on learned experience of where the toil and cocked-foot-guns are working with Borg. Borg is, by modern standards, extremely legacy infrastructure and even Google is trying to replace it with something better internally.
KaiserPro 6y ago
the problem is that k8s is a half arsed implementation of borg/tupperware. Having supported a large team migrate from a custom system based on fleet (god know why they were allowed to waste two years on that) jump to k8s (around 1.2) it struck me how many features it actually lacked (like pod drain, although thats fixed now.) The worst offender in K8s world is the networking scheme. Its utterly batshit, and clearly designed by people who've never had to support anything out of hours. (statically assign 254 IPs to each host, or fuck with the routing table.) Combined with how extra ordinarily chatty the backplane is, makes it expensive and annoying to run at scale. Plus you have to continually upgrade, migrate and generally tweak the thing, every 6 months. Unless you have a big ops team, your frankly better off with lambdas.
- dilyevsky 6y ago
  I worked as borg sre for 6 years and the idea that tweaking the thing every 6 months is a lot is utterly hilarious to me
- rodgerd 6y ago
  > Plus you have to continually upgrade, migrate and generally tweak the thing, every 6 months. If you're AWS, Azure, or Google, this is a feature, not a bug. It makes it easy to sell k8s-as-a-service.
- neurostimulant 6y ago
  > The worst offender in K8s world is the networking scheme. Its utterly batshit, and clearly designed by people who've never had to support anything out of hours. (statically assign 254 IPs to each host, or fuck with the routing table.) Hmm, I thought you could specify larger node cidr at cluster creation time so you can have >254 ip addresses on each node? The default is /24 but you should be able to specify a larger block, unless you're on GKE (not sure if you can do that on GKE).
  - KaiserPro 6y ago
    Thats the point! The network provides simple and easily controllable ip allocation. You don't need to do any of it, just offload it to AWS/GC/Azure. Slam another network card/VLAN, give it a seperate ip range from the control network, and everything is handled for you, quickly, securely and efficiently.
Brian_K_White 6y ago
You define portability as requiring n thousand tight couples instead of one. I define portability as decoupling.

by ixtli 6y ago

jonfw 6y ago

What's wrong with the YAML? It's easy to read and write, concise, and generally has sane defaults meaning you don't have to be overly verbose.

GordonS 6y ago
I'm personally fine with yaml, but specifically k8s yaml config is definitely not concise! Compare roughly equivalent Compose/Swarm and k8s configs, and you'll see that the k8s one is about 8x bigger.
ixtli 6y ago
You know i actually agree with you but, as a manager of other engineers, people never stop complaining about having to edit large amounts of YAML that describe complex objects. At a certain point im forced to acknowledge that its just not ergonomic. Now, I will say that perhaps whats actually happening is the objects that the describe are so complex and numerous that the problem is actually that you need better tools, not that YAML as a format is bad. YAML as used to configure, say Open API specifications, is something people always make my problem. I don't really know why this is because I don't have an issue hacking this stuff up in VIM but people really just don't seem to like it.
dharmab 6y ago
The YAML spec has a lot of ambiguity in it which makes correctly parsing it surprisingly complicated. YAML 1.1 (the most commonly implemented version) also lacks a great way of declaring sets (i.e. unordered collections of unique values). I actually quite like most of the language a lot but the corner cases can be quite complicated.
mixedCase 6y ago
Of all things I would definitely not call it easy to read and write. It has super-weak (see this for examples: https://github.com/cblp/yaml-sucks https://github.com/cblp/yaml-sucks) dynamic typing, which coupled with requiring awful templating to get anything non-trivial done it has been the cause of many outages. For real world Kubernetes needs to be addressed I'd argue we need Dhall or something with Dhall-like semantics and goals to have first class support.
regularfry 6y ago
Spending 6 hours tracking down a `loadBalancerIp` that should have been a `loadBalancerIP` feels like a problem that shouldn't have happened.
- atombender 6y ago
  YAML can be validated using JSON Schema/OpenAPI, and somebody does maintain Kubernetes schema definitions, including Kustomize, which can then be used in an IDE for live validation. For example, the Kubernetes extension for VSCode does this. If you use Helm, writing and validating schemas for Helm value files is trickier, and not something I believe anyone is doing. It should be easier to do. Ideally you should simply put this at the top of a value file: $schema: ./schema.yaml But from what I know, there's no editor/extension, that supports validating through such a declaration. Secondly, Helm doesn't have a way to statically analyze value access. Something like {{ .Values.missTypedKey }} will only be caught when you generate the manifests.
- merb 6y ago
  actually k8s validates the yaml, welp it doesn't it validates the json. basically yaml is only the format the operators get, internally it uses json.
jacques_chester 6y ago
Mostly that it's a single projection of an underlying data model. Where the projection is a snug fit, great. But typically folks want sets of things and suddenly it's a bit chatty.
outworlder 6y ago
> What's wrong with the YAML? Nothing at all. Until you start 'templateizing' it (looking at you, Helm). It's definitely better than JSON (it can even have comments, imagine that). Most people don't bother reading the spec or even examples though, and don't realize how good it actually is.
- chupasaurus 6y ago
  > It's definitely better than JSON k8s API supports JSON almost exclusively, basically one could write in Whitespace then convert to it.
- GordonS 6y ago
  > Nothing at all. Until you start 'templateizing' it (looking at you, Helm). This scourge affects JSON too, such as Azure's ARM templates, which also crowbar logic into the format. An abomination!
- ithkuil 6y ago
  I wish more people stuck with templating yaml would just issue a toJson instead of fiddling with indentation levels
tapoxi 6y ago
In the Kubernetes and Ansible/Salt world it's frequently templatized, and then you need to worry about indentation that you place or your template engine places. Every day I pray that Helm will incorporate support for jsonnet so I never need to touch YAML again.
- jonfw 6y ago
  Why not parse the YAML- manipulate it like you would any other data structure in your language of choice, and then generate the YAML from there? I don't understand why you would want to directly edit markup language without parsing it. YAML's goal is to represent a data structure so that a human can interact with it. It accomplishes that goal well. If you want to edit that data structure programmatically- no markup language makes sense to use, represent the data structure using your programming language's native representations.
- aliswe 6y ago
  I made a script for expressing similar deployments (ie homepage-en, homepage-de) in a variety of contexts (test, prod) and their permutations in a single, succinct json file. It's a bash script though. Example json: { "deployments": [ { "name": "site-en", "env": { "hello": "world" } } ], "targets": [ { "name": "dev", "context": "aks-dev", "env": { "ipsum": "dolor" } } ] }
  - atombender 6y ago
    You should look into Kustomize. It's built into Kubectl, and lets you do something like this in a much more modular way.
- deathanatos 6y ago
  I agree with, and have felt the same pain as you here. But this isn't YAML's fault, IMO, it is that the templaters, like Helm, aren't safe, and don't escape their outputs. If have a string, and I use that in the middle of a YAML template, I expect a string — I don't expect "300" to become (integer) 300; give me a function for that conversion. Any HTML templating system that behaved this way would be frowned upon due to the potential for XSS. There's little risk of XSS here, but it's still annoying to get the wrong results.
ruuda 6y ago
The features intended to make it easy to use, in the end lead to surprising behavior, often with values having non-string types where you expected strings. There are some examples in https://www.arp242.net/yaml-config.html https://www.arp242.net/yaml-config.html.
- jonfw 6y ago
  I haven't had any issues with this in a Kubernetes context- but I can see how this could lead to issues in different contexts
billfor 6y ago
Having to do conditional logic in it, like: matchExpressions: - key: app operator: In values: - myservice Rather than, if (app=="myservice")
- jonfw 6y ago
  Why would you do this in a Kubernetes context?
  - billfor 6y ago
    To specify a podAffinityTerm, amongst others.
  - blandflakes 6y ago
    Your position here seems to be that YAML is a reasonable configuration language. It kinda is. But that's not how it's used in Kubernetes. It's used as a declarative DSL that you use to specify your deployments, and it is HORRIFIC. Extremely nested things where the equivalent actual code to do that would be shorter. Things that are inexpressible. Backwards-incompatible changes to your YAML that actually make it impossible to deploy changes that bring your YAML up to spec if your cluster's master node upgraded first. YAML gives nothing here. jsonnet really isn't a drastic improvement, IMO, except that it lets you express some repetitive parts without having to figure out how many times a template engine is going to indent your YAML.
SrslyJosh 6y ago
YAML actually has one of the worst default behaviors I've seen: automatic conversion of certain strings to bools. This wouldn't be a problem if YAML required that all strings be quoted...but it doesn't, and there's no "strict" mode to force quotes, so users are extremely likely to end up doing something that behaves unexpectedly. If you're lucky, you'll spot the problem quickly instead of wasting half a day on it.
mixmastamyk 6y ago
There is a strict yaml alternative with edge cases removed I recommend.
vorpalhex 6y ago
YAML is easy to read and write in tutorial documents. It becomes a complete mess in real life and whitespace based syntax is horrendous to debug. We have had endless heartaches due to minor yaml issues and today employ tooling to minimize the amount of yaml engineers write to avoid these issues.
wbl 6y ago
I once put an IPv6 address into YAML. It didn't survive the round trip and nearly caused an incident.

shadowgovt 6y ago

It's always worth noting that k8s isn't designed to give outsiders the power of Borg the way Google SREs have the power of Borg. it's designed to give outsiders something better, based on learned experience of where the toil and cocked-foot-guns are working with Borg. Borg is, by modern standards, extremely legacy infrastructure and even Google is trying to replace it with something better internally.

KaiserPro 6y ago

the problem is that k8s is a half arsed implementation of borg/tupperware. Having supported a large team migrate from a custom system based on fleet (god know why they were allowed to waste two years on that) jump to k8s (around 1.2) it struck me how many features it actually lacked (like pod drain, although thats fixed now.) The worst offender in K8s world is the networking scheme. Its utterly batshit, and clearly designed by people who've never had to support anything out of hours. (statically assign 254 IPs to each host, or fuck with the routing table.) Combined with how extra ordinarily chatty the backplane is, makes it expensive and annoying to run at scale. Plus you have to continually upgrade, migrate and generally tweak the thing, every 6 months. Unless you have a big ops team, your frankly better off with lambdas.

dilyevsky 6y ago
I worked as borg sre for 6 years and the idea that tweaking the thing every 6 months is a lot is utterly hilarious to me
rodgerd 6y ago
> Plus you have to continually upgrade, migrate and generally tweak the thing, every 6 months. If you're AWS, Azure, or Google, this is a feature, not a bug. It makes it easy to sell k8s-as-a-service.
neurostimulant 6y ago
> The worst offender in K8s world is the networking scheme. Its utterly batshit, and clearly designed by people who've never had to support anything out of hours. (statically assign 254 IPs to each host, or fuck with the routing table.) Hmm, I thought you could specify larger node cidr at cluster creation time so you can have >254 ip addresses on each node? The default is /24 but you should be able to specify a larger block, unless you're on GKE (not sure if you can do that on GKE).
- KaiserPro 6y ago
  Thats the point! The network provides simple and easily controllable ip allocation. You don't need to do any of it, just offload it to AWS/GC/Azure. Slam another network card/VLAN, give it a seperate ip range from the control network, and everything is handled for you, quickly, securely and efficiently.

Brian_K_White 6y ago

You define portability as requiring n thousand tight couples instead of one. I define portability as decoupling.