10 ms·

I wonder what the need for tools such as this or other "Kubernetes-by-example" type pages tell us about the complexity of configuring Kubernetes resources. Do

by tn890 6y ago

I wonder what the need for tools such as this or other "Kubernetes-by-example" type pages tell us about the complexity of configuring Kubernetes resources. Do we need a better layer of abstraction, i.e. better adoption and tighter integration for something like kustomize? Have we fucked up completely with Kubernetes due to it being outrageously complicated for simple tasks? How we redesign this to be simpler? Is the complexity even a problem for the target audience? I've no idea. I just know I'm a kubernetes admin and I can't write a deployment yaml without googling or copy/pasting.

runbsd 6y ago
A lot of other "bleeding edge" technologies that blow up and become difficult to use end up with new abstractions, and then abstractions for the abstractions, and the original problem the tool was meant to solve is now not the focus any longer.
- drybjed 6y ago
  https://xkcd.com/1319/ https://xkcd.com/1319/
  - runbsd 6y ago
    there is always a relevant xkcd!
throwaway894345 6y ago
I've always felt that Kubernetes isn't appropriate for most businesses to use directly, but rather it's a platform for simpler platforms--someone like Heroku would build on top of Kubernetes and expose a much simpler interface for their users so they don't have to think about SSL, DNS, logging, load balancing, auto-scaling, etc. Alternatively, maybe the problem is solved with "distributions" of Kubernetes analogous to Linux distros--preconfigured Kubernetes installations so you don't have to figure out how to configure your own Kubernetes service (arguably this is also what cloud providers give us with their managed k8s offerings). I'm curious to hear what others think.
- erohead 6y ago
  https://kubesail.com https://kubesail.com offers the exact service you describe!
  - erulabs 6y ago
    Thanks for the shout-out! We’re trying to be like an “open-Heroku” - back when I worked on OpenStack it was called “the open cloud” and no one knew what we were on about - open stack was enterprise as enterprise gets! Our goal is to make a Heroku-ish platform for getting an app online - but one that doesn’t hold you over a barrel later on. You can even host from a spare home server :) (Disclosure: I’m the CTO)
    - OJFord 6y ago
      Do you still expose the k8s bits? I'm interested in/have previously given up on a product where the control plane is managed for me, I can join the cluster with bare metal nodes, and then it's just Kubernetes.
      - erulabs 6y ago
        Yep - we expose the Kubernetes API (and add mutating controllers to help automate some of its actions). Right now we don't do "hosted control plane", rather - you can either use our hosted clusters where you have namespace-level access (so you still get to use Kubernetes, just not all of the resources, for example, no `Nodes`), or you can attach your own cluster to our UI (with the advantage that we can setup ingress and forward traffic to you, which is perfect for a home-hosting setup). Hosted control plane is something im watching closely though - I'd really love to offer that as a service, but since we're small, we're trying to focus hard on a core offering. Will be considered in the future though!
  - kutorio 6y ago
    kubesail has really nice UI for an hosted option. Another alternative I found is https://kalm.dev/ https://kalm.dev/, which is open sourced and geared towards configuring a heroku on top of a k8s cluster.
xyzzy_plugh 6y ago
Most of my colleagues find the yaml interface to be a lost cause. I tend to prefer to use the gRPC API as it's effectively the same declarative operations but you get to control it in software, which just feels right. YAML is just a crutch for specifying the API calls the client needs to make.
- EdSchouten 6y ago
  You mean the Kubernetes HTTP API? That one isn’t gRPC based, right?
  - xyzzy_plugh 6y ago
    Ah right, guess I was thinking of the protobuf over HTTP API. In any case, I find using the protobuf types in your language of choice pretty ergonomic compared to YAML.
    - teqnologiq 6y ago
      Would you happen to have any links to Github repos that would be an example of this? From light googling I only see examples of building gRPC APIs and microservices.
      - xyzzy_plugh 6y ago
        I don't have an example offhand, but mentioned skycfg in the neighbor comment, which may be of some interest to you. I've used https://github.com/ericchiang/k8s https://github.com/ericchiang/k8s in the past with great success.
    - gen220 6y ago
      This is one thing that mesos/aurora got right (the "turing-complete-by-design" config files, written in Python). Templating yaml is like templating html. It works but absolutely sucks at scale if your job is to maintain it. It's good to know that such an interface could conceivably be built for Kubernetes, using the HTTP API. Even better that it supports any programming language (since it's a network interface); I'd see that as a step-up from aurora configs, in fact. At some point, I imagine this will become the default way that Smart Organizations build around k8s, once k8s's core API is super-stable and people develop sound frameworks in $POPULAR_LANGUAGE for declaring your configuration. Then, we can then banish yaml to the same fate as json and xml. One can dream, at least. :)
kevinmgranger 6y ago
I think of Kubernetes as "the new Operating System", and these complex resources as fiddling with initscripts, fstab, /etc/interfaces, and so on. Writing an operator is like writing your own initscript. I wouldn't be surprised if we eventually see new abstractions for "you just want a plain 'ol deployment with CI/CD du jour, a persistent volume claim, and a service ingress, just like 90% of all other CRUD webapps? Sure, here's a simple resource for that." I think we'll start seeing a move towards more "opinionated" tools, just to outsource some of the decision making. No sense learning how to write your own pipelines if you can find a tool that says "we're gonna deploy every time you make a git tag and run `mvn package`, you figure the rest out".
- sushshshsh 6y ago
  Agreed. Just like how I don't want to roll my own desktop environment on Linux, I will happily accept a good config rollef for me, even if opinionated.
- taywrobel 6y ago
  Well prepare to be unsurprised! The CI/CD abstraction already exists in Tekton - https://cloud.google.com/tekton https://cloud.google.com/tekton The rest are soon to follow I’m sure.
  - jml78 6y ago
    If people think deployment yamls are complicated, what til they see tekton. Don’t get me wrong, I actually love tekton because I hate everything that is Jenkins and most other CICD tool integration with k8s. So tekton is just amazing in that regard. But there is a huge learning curve for people who are used to old school tools like Jenkins and its pipelines .
- oso2k 6y ago
  I think of writing Operators more as writing your own Device Driver but I can see the correlation you're trying to make.
- oauea 6y ago
wutsyrpt1 6y ago
Is it anymore complex than all the old ways? Was Apache, Asterisk, or loading and hardening a Linux host on bare metal easier? I seem to remember a lot of wrangling custom kernels to get Asterisk sounding just right, bizarre Apache, & network configs. It’s just text? It’s always going to turn into a nebulous mess without literal edges and boundaries. That’s Google’s play with it, IMO. Train tracks. Which is what I hate about it. Google hasn’t built a less Byzantine text mess. It’s built hype though, with a boring tool
- adwww 6y ago
  There are plenty of online tools to generate Nginx or Vagrant config files. I don't really get the love for hating on K8 complexity on here - nobody says it's the perfect tool for every use case... but when you do need it, it's amazing, and has many advantages over the current trend for serverless IMO.
- ex_amazon_sde 6y ago
  > Is it anymore complex than all the old ways? > Was Apache, Asterisk, or loading and hardening a Linux host on bare metal easier? Yes, and by far. Adding a layer on top of all the traditional Linux daemons, tools and libraries does not decrease the total complexity - quite the contrary. When you have a bug in an application that is related to something in on another layer you have to walk through the whole stack. Examples: A bug in a network card impacting only large UDP packets. A race condition of file access triggered by NFS or a storage device driver. A vulnerability based on a timing attack due to CPU caches. The deeper the stack, the worse.
  - doteka 6y ago
    But wouldn’t the point be that you don’t care about hardware level problems anymore? When I find a node with issues, I can just delete it from the pool and get a fresh one back. The bad network card? That’s for Google/Amazon/DigitalOcean to deal with. I find the bog-standard Prometheus chart provides me a pretty incredible level of monitoring out of the box, usually it’s pretty easy to pick the bad one out of a graph. Running your own VMs without something like k8s? Yeah this setup I can deploy and have working in an hour is gonna take you a week to set up properly. Standardization is valuable. Abstraction is valuable.
holografix 6y ago
I find K8s to be the perfect tool for enterprises. It provides the promise of portability, so the business feels less locked in to a vendor, while still being dense and impenetrable enough to them that IT won’t be done out of a job. Simpler, more efficient alternatives are seen as more expensive, because IT hides the true cost of just how much of their salaries is spent of non-productive pottering around with yaml. These systems and products also threaten to automate IT out of a job and don’t necessarily count much for their resumes. If you’re not a programmer what interest do you have in dumping all of your intricate, fragile environment for a PaaS? Not much
snom380 6y ago
Isn't that the case with any tool or service of some complexity? I'd probably have to look at docs or examples for most config files i write. There's trade-off between a tool being too specific to a usecase vs being too generic with the associated "boilerplate". But I don't know how much simpler Kubernetes could be made while still covering the intended scope?
- p_l 6y ago
  I would say kubernetes is very simple, in the same way lambda calculus or composing programs out of pipes is simple. It has very simple basic model, which allows you to recursively built more and more complex abstractions on top of it.
jayd16 6y ago
I like reading declarative docs because everything is spelled out but I hate writing them for the same reasons. More tooling to write good k8s yaml sounds good to me.
gk1 6y ago
It's a never-ending cycle. See: Front-end development. Until yet another abstraction emerges, anyone using Kubernetes for mission-critical applications should be using automated testing to prevent awful outcomes from a single botched line of YAML. There are a bunch of solutions emerging in this space, like Datree[0], made for the sole purpose of preventing Kubernetes misconfigurations. [0] https://datree.io https://datree.io (Disclaimer: I work with them.)
echelon 6y ago
> I can't write a deployment yaml without googling or copy/pasting What about Nginx or Apache configuration files? Could you write them without googling?
- goatinaboat 6y ago
  What about Nginx or Apache configuration files? Could you write them without googling? Easily, because noone writes them from scratch, they just modify the default one. For most of my Kubernetes work I "kubectl describe" something existing into YAML, modify it, then "kubectl apply" it back again.
  - marcc 6y ago
    You can do this without a cluster also: kubectl create deployment my-deployment --image nginx --dry-run -oyaml
secondcoming 6y ago
IMO it needs two things: - a dedicated editor with intelligent autocomplete - stop using YAML, it soon becomes unreadable. JSON is easier to grok.
- p_l 6y ago
  Unfortunately, JSON is much more annoying to edit. Best way is to stop using both, and generate the objects from higher level language, at least something like Jsonnet (which is really just a step up, so better not stop there but I will take what I can)
  - FridgeSeal 6y ago
    > generate the objects from higher level language I'm on the fence about this, something like Dahl seems good, but I am not a fan of using fully-fledged higher level languages (the Pulumi approach) as I've been burnt before with devs writing bad code and generating configs in the most confusing, convoluted way possible.
- crazy_hombre 6y ago
  Isn't YAML supposed to be a superset of JSON? So technically, you should be able to use JSON files right now (barring a few exceptions).
- goatinaboat 6y ago
  JSON is easier to grok. For anything non-trivial you will want inline comments. Also while any JSON can be expressed in YAML, the reverse is not true.
  - reificator 6y ago
    > For anything non-trivial you will want inline comments. Yes, JSON needs comment support back. > Also while any JSON can be expressed in YAML, the reverse is not true. This is a feature not a bug.
- reificator 6y ago
manigandham 6y ago
Complexity always exists somewhere. You can't remove it, only try it make it easier to deal with. What Kubernetes allows you to do is very complex so you need a capable system to express it all. YAML isn't always the best but it works fine for most and tools like these are very helpful. Do you use an IDE? Does that mean the language and framework is too complex? No, it's just a tool to help you get things done. More tools aren't a bad thing.
- haolez 6y ago
  Not all complexity is intrinsic, but all complexity incurs a cost.
- closeparen 6y ago
  Needing an IDE to have a reasonable time is a very common criticism against the expressive power of certain languages.
- enriched 6y ago
  Agree with this, and would say that often a good way to deal with complexity is encapsulation. I imagine there are some developers that say "why wouldn't you just write it in assembly" Because if you are used to that and write small low level binaries it might work well. But trying to write a larger app it becomes more difficult to reason about.
- gambler 6y ago
  >Complexity always exists somewhere. You can't remove it, only try it make it easier to deal with. This is simply not true. Most complexity in today's systems is completely avoidable. Most developers are just mentally stuck and don't even try. "Managing" complexity is a great way to achieve job security without becoming good at anything specific. Instead of learning how to design system people are learning how to write config files.
  - manigandham 6y ago
    That's a different argument. Not needing it doesn't mean it doesn’t exist. If you don't need Kubernetes then don't use it. But if you do then you can't make it any simpler than the complexity needed to deliver the functionality. Built in deployments, process health, logging, load balancing, security, high availability, config and secret management, volumes and persistence, and much more in exchange for some YAML files is a pretty good encapsulation of complexity though.
jrockway 6y ago
I don't think better abstraction is going to help the underlying problem: there are a lot of questions to answer, and you don't know the answer. Without going into the complexity of Deployments, consider the lowly Pod. What configuration does your app need? What is the name of the container that contains it? How much memory does it use? How much CPU does it need? What ports does it listen on? What HTTP endpoint handles the health check? Does that endpoint test liveness or readiness? What filesystems does it need? What setup needs to be done before the main container runs? Does it need any special resources like GPUs? The list goes on. The problem here is that when you're writing a Pod spec, you're building a single-purpose computer from scratch. In the traditional UNIX world, people answered most of these questions for you. How much RAM can my app use? However much I plugged in. How much CPU can my app use? All of them. What ports does it listen on? Any of them from 1024-65535. What filesystems does it need? Whichever ones I setup in /etc/fstab. I don't think it's a stretch to call UNIX's "yolo" approach problematic. It is great when you have one server running one app, but servers have gotten gigantic (with pricing to match) while applications have largely stayed the same size. This means you have to pack multiple apps onto one physical server, and to do that, there have to be rules. When you write a Kubernetes manifest, you are just answering every possible question upfront so that the entire system runs smoothly even if your individual component doesn't. It's the cost of having small apps on big computers. The problem comes from applications that you didn't write, or don't fully understand. Before you can understand how the application behaves, you have to write a manifest. But you don't know the answers to the questions like how much CPU you're going to use, or what the worst case memory usage is, etc. This causes a lot of cognitive dissonance, because the entire file is you admitting to the computer that you have no idea how to configure it. No abstraction layer is going to fix that problem, except by hiding those uncomfortable details from you. (And you will always regret using the "yolo" defaults -- who hasn't tried to SSH into a broken server only to have Linux helpfully OOMKill sshd or your bash instance when you're just trying to kill your malfunctioning app.) This is largely the fault of application developers. They aren't willing to commit to reasonable resource limits because they don't want to handle support requests that are related to underprovisioning. My experience is that applications that set limits pick them wrong. For example, GCP and DigitalOcean's managed Kubernetes offerings both install monitoring agents to support their dashboards; these apps ship with limits that are too low and any reasonable Prometheus installation will notice that they are being CPU throttled and warn you about it. Now you have to waste your day asking "is this a real problem?" Many open-source apps go the other way and pick resource limits that truly encapsulate the worst case and require individual nodes that are many times larger than the entire cluster. Yes, it would be nice if I gave each pod 32 CPUs and 128GiB of RAM... but I don't want to pay $2000/month/replica thankyouverymuch. (I've been on the other side of that where resources didn't cost me real money and happily used terabytes of RAM as cache.) Application-level configuration is also not in a great state. Everyone tries to sell you their curated defaults so they don't have to write any documentation beyond a "quick start". (I'm as guilty of that as anyone in fact!) The application will have some built-in defaults (so the developers writing the app can just "go run main.go" and get the config they need). Then someone comes along to make a Helm chart for you, and they change the defaults so that their local installation doesn't need any customization. This only causes problems because instead of an undocumented underlying application, now you have that AND an undocumented abstraction layer. You may find the answer to your question "how do I configure FooApp to bar?" but have no way of communicating that config through the Helm abstraction layer because the author of the Helm chart never thought anyone would do that. This rant has gotten quite long so I'll wrap it up. No abstraction layer is ever going to make it so you don't need to answer difficult questions. The actual list of questions to answer is available through "kubectl explain pod.spec" and friends, however.
YarickR2 6y ago
what's wrong with googling or copypasting ? toolset (helm template set) with basic blocks is added rather quickly, after that it could be as simple as running helm oneliner with needed variable values in the command line
spullara 6y ago
Open Stack by Google.
jwatte 6y ago
"Simple" solutions to "complex" problems, by necessity, only work for certain subsets of the problems. Modern distributed systems have tons of largely essential complexity. The only way to reduce it, is to reduce the expressive power of the solution, which might be great for the cases where you fit into the mold, but then fails spectacularly (either with excess resource consumption, or by not working at all) when you push against the boundaries.
renewiltord 6y ago
I always commit an example k8s YAML to any repo with k8s in it that describes how to get 2048 running. Honestly, I get the pieces and I know where to look to get what so I'm not too bothered. The problem with wrapper tools is that sometimes I can't get at the insides and then I have to learn the wrapper tool. So I'm going to just stick to raw Kube until one of the wrappers wins out.
auspex 6y ago
I think abstracting YAML generation to a tool is a good idea. Let the tool handle the underlying abstraction of converting the parameters to YAML. If k8s changes notation having a layer of abstraction (hopefully) allows me to run the same command to generate new YAML (very helpful in automation)

by tn890 6y ago

runbsd 6y ago

A lot of other "bleeding edge" technologies that blow up and become difficult to use end up with new abstractions, and then abstractions for the abstractions, and the original problem the tool was meant to solve is now not the focus any longer.

drybjed 6y ago
https://xkcd.com/1319/ https://xkcd.com/1319/
- runbsd 6y ago
  there is always a relevant xkcd!

throwaway894345 6y ago

I've always felt that Kubernetes isn't appropriate for most businesses to use directly, but rather it's a platform for simpler platforms--someone like Heroku would build on top of Kubernetes and expose a much simpler interface for their users so they don't have to think about SSL, DNS, logging, load balancing, auto-scaling, etc. Alternatively, maybe the problem is solved with "distributions" of Kubernetes analogous to Linux distros--preconfigured Kubernetes installations so you don't have to figure out how to configure your own Kubernetes service (arguably this is also what cloud providers give us with their managed k8s offerings). I'm curious to hear what others think.

erohead 6y ago
https://kubesail.com https://kubesail.com offers the exact service you describe!
- erulabs 6y ago
  Thanks for the shout-out! We’re trying to be like an “open-Heroku” - back when I worked on OpenStack it was called “the open cloud” and no one knew what we were on about - open stack was enterprise as enterprise gets! Our goal is to make a Heroku-ish platform for getting an app online - but one that doesn’t hold you over a barrel later on. You can even host from a spare home server :) (Disclosure: I’m the CTO)
  - OJFord 6y ago
    Do you still expose the k8s bits? I'm interested in/have previously given up on a product where the control plane is managed for me, I can join the cluster with bare metal nodes, and then it's just Kubernetes.
    - erulabs 6y ago
      Yep - we expose the Kubernetes API (and add mutating controllers to help automate some of its actions). Right now we don't do "hosted control plane", rather - you can either use our hosted clusters where you have namespace-level access (so you still get to use Kubernetes, just not all of the resources, for example, no `Nodes`), or you can attach your own cluster to our UI (with the advantage that we can setup ingress and forward traffic to you, which is perfect for a home-hosting setup). Hosted control plane is something im watching closely though - I'd really love to offer that as a service, but since we're small, we're trying to focus hard on a core offering. Will be considered in the future though!
- kutorio 6y ago
  kubesail has really nice UI for an hosted option. Another alternative I found is https://kalm.dev/ https://kalm.dev/, which is open sourced and geared towards configuring a heroku on top of a k8s cluster.

xyzzy_plugh 6y ago

Most of my colleagues find the yaml interface to be a lost cause. I tend to prefer to use the gRPC API as it's effectively the same declarative operations but you get to control it in software, which just feels right. YAML is just a crutch for specifying the API calls the client needs to make.

EdSchouten 6y ago
You mean the Kubernetes HTTP API? That one isn’t gRPC based, right?
- xyzzy_plugh 6y ago
  Ah right, guess I was thinking of the protobuf over HTTP API. In any case, I find using the protobuf types in your language of choice pretty ergonomic compared to YAML.
  - teqnologiq 6y ago
    Would you happen to have any links to Github repos that would be an example of this? From light googling I only see examples of building gRPC APIs and microservices.
    - xyzzy_plugh 6y ago
      I don't have an example offhand, but mentioned skycfg in the neighbor comment, which may be of some interest to you. I've used https://github.com/ericchiang/k8s https://github.com/ericchiang/k8s in the past with great success.
  - gen220 6y ago
    This is one thing that mesos/aurora got right (the "turing-complete-by-design" config files, written in Python). Templating yaml is like templating html. It works but absolutely sucks at scale if your job is to maintain it. It's good to know that such an interface could conceivably be built for Kubernetes, using the HTTP API. Even better that it supports any programming language (since it's a network interface); I'd see that as a step-up from aurora configs, in fact. At some point, I imagine this will become the default way that Smart Organizations build around k8s, once k8s's core API is super-stable and people develop sound frameworks in $POPULAR_LANGUAGE for declaring your configuration. Then, we can then banish yaml to the same fate as json and xml. One can dream, at least. :)

kevinmgranger 6y ago

I think of Kubernetes as "the new Operating System", and these complex resources as fiddling with initscripts, fstab, /etc/interfaces, and so on. Writing an operator is like writing your own initscript. I wouldn't be surprised if we eventually see new abstractions for "you just want a plain 'ol deployment with CI/CD du jour, a persistent volume claim, and a service ingress, just like 90% of all other CRUD webapps? Sure, here's a simple resource for that." I think we'll start seeing a move towards more "opinionated" tools, just to outsource some of the decision making. No sense learning how to write your own pipelines if you can find a tool that says "we're gonna deploy every time you make a git tag and run `mvn package`, you figure the rest out".

sushshshsh 6y ago
Agreed. Just like how I don't want to roll my own desktop environment on Linux, I will happily accept a good config rollef for me, even if opinionated.
taywrobel 6y ago
Well prepare to be unsurprised! The CI/CD abstraction already exists in Tekton - https://cloud.google.com/tekton https://cloud.google.com/tekton The rest are soon to follow I’m sure.
- jml78 6y ago
  If people think deployment yamls are complicated, what til they see tekton. Don’t get me wrong, I actually love tekton because I hate everything that is Jenkins and most other CICD tool integration with k8s. So tekton is just amazing in that regard. But there is a huge learning curve for people who are used to old school tools like Jenkins and its pipelines .
oso2k 6y ago
I think of writing Operators more as writing your own Device Driver but I can see the correlation you're trying to make.
oauea 6y ago

wutsyrpt1 6y ago

Is it anymore complex than all the old ways? Was Apache, Asterisk, or loading and hardening a Linux host on bare metal easier? I seem to remember a lot of wrangling custom kernels to get Asterisk sounding just right, bizarre Apache, & network configs. It’s just text? It’s always going to turn into a nebulous mess without literal edges and boundaries. That’s Google’s play with it, IMO. Train tracks. Which is what I hate about it. Google hasn’t built a less Byzantine text mess. It’s built hype though, with a boring tool

adwww 6y ago
There are plenty of online tools to generate Nginx or Vagrant config files. I don't really get the love for hating on K8 complexity on here - nobody says it's the perfect tool for every use case... but when you do need it, it's amazing, and has many advantages over the current trend for serverless IMO.
ex_amazon_sde 6y ago
> Is it anymore complex than all the old ways? > Was Apache, Asterisk, or loading and hardening a Linux host on bare metal easier? Yes, and by far. Adding a layer on top of all the traditional Linux daemons, tools and libraries does not decrease the total complexity - quite the contrary. When you have a bug in an application that is related to something in on another layer you have to walk through the whole stack. Examples: A bug in a network card impacting only large UDP packets. A race condition of file access triggered by NFS or a storage device driver. A vulnerability based on a timing attack due to CPU caches. The deeper the stack, the worse.
- doteka 6y ago
  But wouldn’t the point be that you don’t care about hardware level problems anymore? When I find a node with issues, I can just delete it from the pool and get a fresh one back. The bad network card? That’s for Google/Amazon/DigitalOcean to deal with. I find the bog-standard Prometheus chart provides me a pretty incredible level of monitoring out of the box, usually it’s pretty easy to pick the bad one out of a graph. Running your own VMs without something like k8s? Yeah this setup I can deploy and have working in an hour is gonna take you a week to set up properly. Standardization is valuable. Abstraction is valuable.

holografix 6y ago

I find K8s to be the perfect tool for enterprises. It provides the promise of portability, so the business feels less locked in to a vendor, while still being dense and impenetrable enough to them that IT won’t be done out of a job. Simpler, more efficient alternatives are seen as more expensive, because IT hides the true cost of just how much of their salaries is spent of non-productive pottering around with yaml. These systems and products also threaten to automate IT out of a job and don’t necessarily count much for their resumes. If you’re not a programmer what interest do you have in dumping all of your intricate, fragile environment for a PaaS? Not much

snom380 6y ago

Isn't that the case with any tool or service of some complexity? I'd probably have to look at docs or examples for most config files i write. There's trade-off between a tool being too specific to a usecase vs being too generic with the associated "boilerplate". But I don't know how much simpler Kubernetes could be made while still covering the intended scope?

p_l 6y ago
I would say kubernetes is very simple, in the same way lambda calculus or composing programs out of pipes is simple. It has very simple basic model, which allows you to recursively built more and more complex abstractions on top of it.

jayd16 6y ago

I like reading declarative docs because everything is spelled out but I hate writing them for the same reasons. More tooling to write good k8s yaml sounds good to me.

gk1 6y ago

It's a never-ending cycle. See: Front-end development. Until yet another abstraction emerges, anyone using Kubernetes for mission-critical applications should be using automated testing to prevent awful outcomes from a single botched line of YAML. There are a bunch of solutions emerging in this space, like Datree[0], made for the sole purpose of preventing Kubernetes misconfigurations. [0] https://datree.io https://datree.io (Disclaimer: I work with them.)

echelon 6y ago

> I can't write a deployment yaml without googling or copy/pasting What about Nginx or Apache configuration files? Could you write them without googling?

goatinaboat 6y ago
What about Nginx or Apache configuration files? Could you write them without googling? Easily, because noone writes them from scratch, they just modify the default one. For most of my Kubernetes work I "kubectl describe" something existing into YAML, modify it, then "kubectl apply" it back again.
- marcc 6y ago
  You can do this without a cluster also: kubectl create deployment my-deployment --image nginx --dry-run -oyaml

secondcoming 6y ago

IMO it needs two things: - a dedicated editor with intelligent autocomplete - stop using YAML, it soon becomes unreadable. JSON is easier to grok.

p_l 6y ago
Unfortunately, JSON is much more annoying to edit. Best way is to stop using both, and generate the objects from higher level language, at least something like Jsonnet (which is really just a step up, so better not stop there but I will take what I can)
- FridgeSeal 6y ago
  > generate the objects from higher level language I'm on the fence about this, something like Dahl seems good, but I am not a fan of using fully-fledged higher level languages (the Pulumi approach) as I've been burnt before with devs writing bad code and generating configs in the most confusing, convoluted way possible.
crazy_hombre 6y ago
Isn't YAML supposed to be a superset of JSON? So technically, you should be able to use JSON files right now (barring a few exceptions).
goatinaboat 6y ago
JSON is easier to grok. For anything non-trivial you will want inline comments. Also while any JSON can be expressed in YAML, the reverse is not true.
- reificator 6y ago
  > For anything non-trivial you will want inline comments. Yes, JSON needs comment support back. > Also while any JSON can be expressed in YAML, the reverse is not true. This is a feature not a bug.
reificator 6y ago

manigandham 6y ago

Complexity always exists somewhere. You can't remove it, only try it make it easier to deal with. What Kubernetes allows you to do is very complex so you need a capable system to express it all. YAML isn't always the best but it works fine for most and tools like these are very helpful. Do you use an IDE? Does that mean the language and framework is too complex? No, it's just a tool to help you get things done. More tools aren't a bad thing.

haolez 6y ago
Not all complexity is intrinsic, but all complexity incurs a cost.
closeparen 6y ago
Needing an IDE to have a reasonable time is a very common criticism against the expressive power of certain languages.
enriched 6y ago
Agree with this, and would say that often a good way to deal with complexity is encapsulation. I imagine there are some developers that say "why wouldn't you just write it in assembly" Because if you are used to that and write small low level binaries it might work well. But trying to write a larger app it becomes more difficult to reason about.
gambler 6y ago
>Complexity always exists somewhere. You can't remove it, only try it make it easier to deal with. This is simply not true. Most complexity in today's systems is completely avoidable. Most developers are just mentally stuck and don't even try. "Managing" complexity is a great way to achieve job security without becoming good at anything specific. Instead of learning how to design system people are learning how to write config files.
- manigandham 6y ago
  That's a different argument. Not needing it doesn't mean it doesn’t exist. If you don't need Kubernetes then don't use it. But if you do then you can't make it any simpler than the complexity needed to deliver the functionality. Built in deployments, process health, logging, load balancing, security, high availability, config and secret management, volumes and persistence, and much more in exchange for some YAML files is a pretty good encapsulation of complexity though.

jrockway 6y ago

I don't think better abstraction is going to help the underlying problem: there are a lot of questions to answer, and you don't know the answer. Without going into the complexity of Deployments, consider the lowly Pod. What configuration does your app need? What is the name of the container that contains it? How much memory does it use? How much CPU does it need? What ports does it listen on? What HTTP endpoint handles the health check? Does that endpoint test liveness or readiness? What filesystems does it need? What setup needs to be done before the main container runs? Does it need any special resources like GPUs? The list goes on. The problem here is that when you're writing a Pod spec, you're building a single-purpose computer from scratch. In the traditional UNIX world, people answered most of these questions for you. How much RAM can my app use? However much I plugged in. How much CPU can my app use? All of them. What ports does it listen on? Any of them from 1024-65535. What filesystems does it need? Whichever ones I setup in /etc/fstab. I don't think it's a stretch to call UNIX's "yolo" approach problematic. It is great when you have one server running one app, but servers have gotten gigantic (with pricing to match) while applications have largely stayed the same size. This means you have to pack multiple apps onto one physical server, and to do that, there have to be rules. When you write a Kubernetes manifest, you are just answering every possible question upfront so that the entire system runs smoothly even if your individual component doesn't. It's the cost of having small apps on big computers. The problem comes from applications that you didn't write, or don't fully understand. Before you can understand how the application behaves, you have to write a manifest. But you don't know the answers to the questions like how much CPU you're going to use, or what the worst case memory usage is, etc. This causes a lot of cognitive dissonance, because the entire file is you admitting to the computer that you have no idea how to configure it. No abstraction layer is going to fix that problem, except by hiding those uncomfortable details from you. (And you will always regret using the "yolo" defaults -- who hasn't tried to SSH into a broken server only to have Linux helpfully OOMKill sshd or your bash instance when you're just trying to kill your malfunctioning app.) This is largely the fault of application developers. They aren't willing to commit to reasonable resource limits because they don't want to handle support requests that are related to underprovisioning. My experience is that applications that set limits pick them wrong. For example, GCP and DigitalOcean's managed Kubernetes offerings both install monitoring agents to support their dashboards; these apps ship with limits that are too low and any reasonable Prometheus installation will notice that they are being CPU throttled and warn you about it. Now you have to waste your day asking "is this a real problem?" Many open-source apps go the other way and pick resource limits that truly encapsulate the worst case and require individual nodes that are many times larger than the entire cluster. Yes, it would be nice if I gave each pod 32 CPUs and 128GiB of RAM... but I don't want to pay $2000/month/replica thankyouverymuch. (I've been on the other side of that where resources didn't cost me real money and happily used terabytes of RAM as cache.) Application-level configuration is also not in a great state. Everyone tries to sell you their curated defaults so they don't have to write any documentation beyond a "quick start". (I'm as guilty of that as anyone in fact!) The application will have some built-in defaults (so the developers writing the app can just "go run main.go" and get the config they need). Then someone comes along to make a Helm chart for you, and they change the defaults so that their local installation doesn't need any customization. This only causes problems because instead of an undocumented underlying application, now you have that AND an undocumented abstraction layer. You may find the answer to your question "how do I configure FooApp to bar?" but have no way of communicating that config through the Helm abstraction layer because the author of the Helm chart never thought anyone would do that. This rant has gotten quite long so I'll wrap it up. No abstraction layer is ever going to make it so you don't need to answer difficult questions. The actual list of questions to answer is available through "kubectl explain pod.spec" and friends, however.

YarickR2 6y ago

what's wrong with googling or copypasting ? toolset (helm template set) with basic blocks is added rather quickly, after that it could be as simple as running helm oneliner with needed variable values in the command line

spullara 6y ago

Open Stack by Google.

jwatte 6y ago

"Simple" solutions to "complex" problems, by necessity, only work for certain subsets of the problems. Modern distributed systems have tons of largely essential complexity. The only way to reduce it, is to reduce the expressive power of the solution, which might be great for the cases where you fit into the mold, but then fails spectacularly (either with excess resource consumption, or by not working at all) when you push against the boundaries.

renewiltord 6y ago

I always commit an example k8s YAML to any repo with k8s in it that describes how to get 2048 running. Honestly, I get the pieces and I know where to look to get what so I'm not too bothered. The problem with wrapper tools is that sometimes I can't get at the insides and then I have to learn the wrapper tool. So I'm going to just stick to raw Kube until one of the wrappers wins out.

auspex 6y ago

I think abstracting YAML generation to a tool is a good idea. Let the tool handle the underlying abstraction of converting the parameters to YAML. If k8s changes notation having a layer of abstraction (hopefully) allows me to run the same command to generate new YAML (very helpful in automation)