7 ms·

Kubernetes as a platform vs. Kubernetes as an API

66 points by mariuz 4y ago

ianpurton 4y ago
> What is kubernetes? For me it's a platform to deploy apps without tying myself into a particular vendor.
- spyremeown 4y ago
  One of the things I hear the most about Kubernetes is the complexity. Honestly, it's not that complex from an user perspective. It's not much more complicated than Docker, really. It's also super nice to just write two yaml files and have everything running nicely with rollouts, auto-scaling, node pinning etc. It just works and I never have to even think about the internals (although I took the time to learn most of them). Adding: heck, I'll run a single node k3s on my computer (which takes one (1) command) instead of using docker-compose just because I like it better than compose.
  - ninkendo 4y ago
    Yeah most of the (IMO) legitimate complaints about the difficulty of k8s are about how hard it is to correctly set up and administer. (Networking setup, making ingress work, cloud service integration, etc.) Especially when you get into permissions/namespaces and trying to get that right for a decent size organization and trying to configure things for least privilege. Simply using it isn’t that hard IMO.
    - steveBK123 4y ago
      Right but this is the irony I think. If you need the scale provided by k8s, you are probably in a large org. If you are in a large org, you are probably intermediated from running k8s directly by some kind of devops/cloudops team who needs to sort out all the "hard administration stuff" you listed. Setting up a science project at home in your toy AWS account I'm sure is no big deal. But this is true of every tech- "works on my box" is always easier than "doing this in enterprise". The measure of a technology is how easy it is to use in the real production environment.
      - trallnag 4y ago
        Being in a large organization does not equal having a dedicated team for running Kubernetes clusters. For example Daimler (the company behind brands like Mercedes) has hundreds of Kubernetes clusters across hundreds of small product teams.
    - jasonjayr 4y ago
      IMHO, the biggest annoyance and difficulty with K8s, is figuring out the helm chart ecosystem with the crazy templated YAML. Some of those documents can get quite convoluted. One family of charts (say, from Bitami) will have one set of conventions for values.yaml, and other families will have a slightly different way of organizing this.
      - akvadrako 4y ago
        You can just not use helm. Kustomize, which is built-in, does almost everything needed and is much simpler.
        itintheory 4y ago
        But then you have an entirely different challenge if you want to use a project that only provides a chart as a deployment package / docs.
        jpdb 4y ago
        Try checking out kpt, it does packages and hydration. https://kpt.dev/ https://kpt.dev/
  - danpalmer 4y ago
    Kubernetes is complex if you don't need what it provides. However if you do need what it provides in _some way_, Kubernetes provides a pretty nice, coherent way of doing it.
    - steveBK123 4y ago
      The problem is a lot of people read about k8s and decided stuff that could be done with a queue, 3 micro services, and 4 cores of CPU running on a RasPi somehow needs ... k8s.
      - danpalmer 4y ago
        This might be true, but one of the under-appreciated benefits of Kubernetes is the control around deployments. 3 services and a queue isn't much... unless you have high uptime requirements, need to maintain capacity during releases, want a careful rollout process, etc. It's possible to orchestrate complex release flows without an orchestrator, but it's often easier with one.
  - bravetraveler 4y ago
    I find consuming K8s to be like this - it's great, provides facilities for common everyday things I find building K8s (well) to be wildly different. CNIs, storage classes/providers, yada yada. For a 'solution', you have to build a lot of it. It's in a weird spot to me, I like/enjoy it - but only because I have other people helping me tend to it edit: Think of a typical 'cloud' -- it's typically made of compute, storage, and networking. My 'thing' about K8s is it's hyper-focused on the compute part, touted as complete, but (understandably) lacking in terms of storage/networking OOTB Being able to easily build (development) clusters perpetuates this a bit more than I care for. They let you ignore the parts of production-worthy K8s, worrying only about the workload.
    - adra 4y ago
      I still remember a recent company I worked at. They had a strict on prem datacenter ethos. Their procurement was so bad that you had to know your hardware requirements months ahead of time to get what you ask for (pre-covid mind you). If you under provisioned, you were boned. What naturally happened? Teams allocated a ton of resources that ended up sitting around in bitrot hell. I think someone took a sample of all system CPUs and found something like 6% averages. Pathetically underutilized. If this was k8s, it means the company can provision for their general needs, and teams can requisition what they needs easily through technological means instead of bureaucratic ones. Crap like storage classes, etc are sure more annoying than simple apps teams, but it's way easier than needing to manage the bare metal and provisioning / maintenance / migration that is often completely invisible to apps teams. Oh my app got restarted an hour ago, why was that? Oh it turned out a node had it's hard-drive started to fail and the node automatically drained it's load and moved them to a spare capacity node without you. Plus it often saves the need to page a human to run around frantically to figure out why all these services are crashing or misbehaving.
      - ilyt 4y ago
        You're comparing it with "buy server(s) per project approach" but many companies went to "buy a bunch of servers and run VMs under some controller" long before k8s existed. Hell, that's whole VMware business model. Sure containers allow for more dynamic fitting, especially memory wise but that's not the biggest benefit. The difference here is not between utilization but the fact that k8s allows developer to deploy "smaller unit" - they don't need to manage whole VM, they just need to push a container to it. And keep the deployment config near the app itself and not a document to throw across the wall to ops to realize.
  - MuffinFlavored 4y ago
    Could you share those 2 YAML files? I feel like it's not hard to do k8s YAML the wrong way and miss out on the auto-rollout, auto-scaling, etc. features
    - trallnag 4y ago
      You are implying that he needs all these automations.
  - ilyt 4y ago
    It moves most/all of the complexity on ops side. or in "whatever automation thingy you use to deploy it". Yes, easy on dev, but compared to "just docker" or "a VM with some systemd service running your app" it is hideously complex
mac-chaffee 4y ago
Amazon publishing a blog that blatantly argues for vendor lock-in? Who would have guessed? One of the big benefits of Kubernetes is the cloud-agnostic API for running pretty much any application. CustomResources can help blur that boundary and make it easier to use some managed services, but actually arguing in favor of that approach for everything is a pure advertising tactic. It's no secret that managed services are several times more expensive than equivalent EC2 time. Sometimes it's worth it. But to throw Kubernetes into the mix just to run some controllers that deploy those services will pointlessly add to your monthly bill. Running any Kubernetes cluster is not cheap, even if it's just a bunch of controllers.
- mreferre 4y ago
  [ Disclaimer: Author of the blog here ] I knew this blog was very polarizing so this is a fair comment. I have called out advantages and disadvantages in the first scenario ("The pros of this approach is that the customer is decoupling heavily from the infrastructure underneath thus enabling better portability of their entire platform. The cons of doing this is that the customer is signing up for a high amount of work.") and, in retrospect, I should have called them out explicitly for the last scenario (I have just assumed readers could infer they would be the opposite). I did mention towards the end that "customers [that can benefit from the third model ] are coming from an Amazon ECS background, they are all in into Amazon ECS and AWS" but I take to heart the suggestion that I should have been more explicit. Thanks.
- joshribakoff 4y ago
  > It’s arguably an extreme option and one that doesn’t even exist today How is it blatantly arguing for this thing when it literally calls it extreme and something that doesn’t exist? I think a more fair criticism would be the fact they don’t call out vendor lock-in as a tradeoff > to throw Kubernetes into the mix just to run some controllers that deploy those services will pointlessly add to your monthly bill. I think you have a valid point it may be dubious but the article does outline purported reasons it could make sense for some people. I dont think there is a “right way” and a “wrong way” here, rather there is the way “you would do it” and “the way they’re proposing as an option”
linuxftw 4y ago
There has been a lot of prior work in this space. There's crossplane, the Terraform Operator, and a ton of roll-your-own solutions for tackling this vary problem. Alibaba also was early to have the concept of a 'virtual node' where you can deploy seemingly infinite number of pods, they'll handle all the orchestration below. I was reading the article thinking "Yes, finally, some code or use to implement interacting with AWS without a 3rd party lib" and then I got to this section at the bottom: > In this post, we showed you the flexibility of Kubernetes. It’s arguably an extreme option and one that doesn’t even exist today (the Amazon ECS ACK controller isn’t available but please thumb up this roadmap proposal if you are intrigued). If anything, it proves that Kubernetes can be a lot of things to a lot of different people and defining it remains a work in progress (at least in my head). As usual, Amazon is playing catch-up in actually orchestrating their services in a non-AWS-centric way. Very deceptive to create a blog post and not disclose it's not real software at the end.
- coredog64 4y ago
  The ACK exists and allows management of AWS services. What doesn’t exist is the ECS-specific ACK.
IceWreck 4y ago
The last one (Kubernetes is my control plane but not my data plane) is completely stupid. Instead of that why not use terraform or equivalents ? Paying for EKS just to use its API to provision and manage other AWS resources is an interesting idea, but not something I'd use in production.
jhoelzel 4y ago
I love how conveniently on-premise kubernetes is missing. Or self-managed in the cloud. The whole point of kubernetes is to not think about your servers anymore because they in essence can be replaced at any time. This would only provide absolute vendor lock-in. Of course you can use any external service there is (like a database) but you should never put all your eggs into one basked and trust only one hoster. The principle remains the same: its just someone elses computer. Storage and networking are solved in a lot of places at a fraction of the cost and that is really just the beginning., but also the whole point of the exercise: to be able to move your workloads from wherever to wherever
- awill 4y ago
  >>The whole point of kubernetes is to not think about your servers anymore because they in essence can be replaced at any time. Isn't that also the case for VMs?
  - pixl97 4y ago
    How are you orchestrating the configuration of your VMs?
    - mmh0000 4y ago
      Terraform (for cloudy stuff), kickstart and ansible (for on-prem stuff) would be my first go-to(s) for automatic installation, configuration, and config management. https://www.terraform.io/ https://www.terraform.io/ https://en.wikipedia.org/wiki/Kickstart_(Linux) https://en.wikipedia.org/wiki/Kickstart_(Linux) https://www.ansible.com/ https://www.ansible.com/
      - jhoelzel 4y ago
        The question was not "what tools do you use" but how do you achieve orchestration on the same level as kubernetes. Your machines are still named. Your resources still multiplicative. Your setup is not idempotent. Don't get me wrong, i have been using ansible and terraform for over a decade too, but this truly is an apple and oranges comparison.

66 points by mariuz 4y ago

ianpurton 4y ago

> What is kubernetes? For me it's a platform to deploy apps without tying myself into a particular vendor.

spyremeown 4y ago
One of the things I hear the most about Kubernetes is the complexity. Honestly, it's not that complex from an user perspective. It's not much more complicated than Docker, really. It's also super nice to just write two yaml files and have everything running nicely with rollouts, auto-scaling, node pinning etc. It just works and I never have to even think about the internals (although I took the time to learn most of them). Adding: heck, I'll run a single node k3s on my computer (which takes one (1) command) instead of using docker-compose just because I like it better than compose.
- ninkendo 4y ago
  Yeah most of the (IMO) legitimate complaints about the difficulty of k8s are about how hard it is to correctly set up and administer. (Networking setup, making ingress work, cloud service integration, etc.) Especially when you get into permissions/namespaces and trying to get that right for a decent size organization and trying to configure things for least privilege. Simply using it isn’t that hard IMO.
  - steveBK123 4y ago
    Right but this is the irony I think. If you need the scale provided by k8s, you are probably in a large org. If you are in a large org, you are probably intermediated from running k8s directly by some kind of devops/cloudops team who needs to sort out all the "hard administration stuff" you listed. Setting up a science project at home in your toy AWS account I'm sure is no big deal. But this is true of every tech- "works on my box" is always easier than "doing this in enterprise". The measure of a technology is how easy it is to use in the real production environment.
    - trallnag 4y ago
      Being in a large organization does not equal having a dedicated team for running Kubernetes clusters. For example Daimler (the company behind brands like Mercedes) has hundreds of Kubernetes clusters across hundreds of small product teams.
  - jasonjayr 4y ago
    IMHO, the biggest annoyance and difficulty with K8s, is figuring out the helm chart ecosystem with the crazy templated YAML. Some of those documents can get quite convoluted. One family of charts (say, from Bitami) will have one set of conventions for values.yaml, and other families will have a slightly different way of organizing this.
    - akvadrako 4y ago
      You can just not use helm. Kustomize, which is built-in, does almost everything needed and is much simpler.
      - itintheory 4y ago
        But then you have an entirely different challenge if you want to use a project that only provides a chart as a deployment package / docs.
        jpdb 4y ago
        Try checking out kpt, it does packages and hydration. https://kpt.dev/ https://kpt.dev/
- danpalmer 4y ago
  Kubernetes is complex if you don't need what it provides. However if you do need what it provides in _some way_, Kubernetes provides a pretty nice, coherent way of doing it.
  - steveBK123 4y ago
    The problem is a lot of people read about k8s and decided stuff that could be done with a queue, 3 micro services, and 4 cores of CPU running on a RasPi somehow needs ... k8s.
    - danpalmer 4y ago
      This might be true, but one of the under-appreciated benefits of Kubernetes is the control around deployments. 3 services and a queue isn't much... unless you have high uptime requirements, need to maintain capacity during releases, want a careful rollout process, etc. It's possible to orchestrate complex release flows without an orchestrator, but it's often easier with one.
- bravetraveler 4y ago
  I find consuming K8s to be like this - it's great, provides facilities for common everyday things I find building K8s (well) to be wildly different. CNIs, storage classes/providers, yada yada. For a 'solution', you have to build a lot of it. It's in a weird spot to me, I like/enjoy it - but only because I have other people helping me tend to it edit: Think of a typical 'cloud' -- it's typically made of compute, storage, and networking. My 'thing' about K8s is it's hyper-focused on the compute part, touted as complete, but (understandably) lacking in terms of storage/networking OOTB Being able to easily build (development) clusters perpetuates this a bit more than I care for. They let you ignore the parts of production-worthy K8s, worrying only about the workload.
  - adra 4y ago
    I still remember a recent company I worked at. They had a strict on prem datacenter ethos. Their procurement was so bad that you had to know your hardware requirements months ahead of time to get what you ask for (pre-covid mind you). If you under provisioned, you were boned. What naturally happened? Teams allocated a ton of resources that ended up sitting around in bitrot hell. I think someone took a sample of all system CPUs and found something like 6% averages. Pathetically underutilized. If this was k8s, it means the company can provision for their general needs, and teams can requisition what they needs easily through technological means instead of bureaucratic ones. Crap like storage classes, etc are sure more annoying than simple apps teams, but it's way easier than needing to manage the bare metal and provisioning / maintenance / migration that is often completely invisible to apps teams. Oh my app got restarted an hour ago, why was that? Oh it turned out a node had it's hard-drive started to fail and the node automatically drained it's load and moved them to a spare capacity node without you. Plus it often saves the need to page a human to run around frantically to figure out why all these services are crashing or misbehaving.
    - ilyt 4y ago
      You're comparing it with "buy server(s) per project approach" but many companies went to "buy a bunch of servers and run VMs under some controller" long before k8s existed. Hell, that's whole VMware business model. Sure containers allow for more dynamic fitting, especially memory wise but that's not the biggest benefit. The difference here is not between utilization but the fact that k8s allows developer to deploy "smaller unit" - they don't need to manage whole VM, they just need to push a container to it. And keep the deployment config near the app itself and not a document to throw across the wall to ops to realize.
- MuffinFlavored 4y ago
  Could you share those 2 YAML files? I feel like it's not hard to do k8s YAML the wrong way and miss out on the auto-rollout, auto-scaling, etc. features
  - trallnag 4y ago
    You are implying that he needs all these automations.
- ilyt 4y ago
  It moves most/all of the complexity on ops side. or in "whatever automation thingy you use to deploy it". Yes, easy on dev, but compared to "just docker" or "a VM with some systemd service running your app" it is hideously complex

mac-chaffee 4y ago

Amazon publishing a blog that blatantly argues for vendor lock-in? Who would have guessed? One of the big benefits of Kubernetes is the cloud-agnostic API for running pretty much any application. CustomResources can help blur that boundary and make it easier to use some managed services, but actually arguing in favor of that approach for everything is a pure advertising tactic. It's no secret that managed services are several times more expensive than equivalent EC2 time. Sometimes it's worth it. But to throw Kubernetes into the mix just to run some controllers that deploy those services will pointlessly add to your monthly bill. Running any Kubernetes cluster is not cheap, even if it's just a bunch of controllers.

mreferre 4y ago
[ Disclaimer: Author of the blog here ] I knew this blog was very polarizing so this is a fair comment. I have called out advantages and disadvantages in the first scenario ("The pros of this approach is that the customer is decoupling heavily from the infrastructure underneath thus enabling better portability of their entire platform. The cons of doing this is that the customer is signing up for a high amount of work.") and, in retrospect, I should have called them out explicitly for the last scenario (I have just assumed readers could infer they would be the opposite). I did mention towards the end that "customers [that can benefit from the third model ] are coming from an Amazon ECS background, they are all in into Amazon ECS and AWS" but I take to heart the suggestion that I should have been more explicit. Thanks.
joshribakoff 4y ago
> It’s arguably an extreme option and one that doesn’t even exist today How is it blatantly arguing for this thing when it literally calls it extreme and something that doesn’t exist? I think a more fair criticism would be the fact they don’t call out vendor lock-in as a tradeoff > to throw Kubernetes into the mix just to run some controllers that deploy those services will pointlessly add to your monthly bill. I think you have a valid point it may be dubious but the article does outline purported reasons it could make sense for some people. I dont think there is a “right way” and a “wrong way” here, rather there is the way “you would do it” and “the way they’re proposing as an option”

linuxftw 4y ago

There has been a lot of prior work in this space. There's crossplane, the Terraform Operator, and a ton of roll-your-own solutions for tackling this vary problem. Alibaba also was early to have the concept of a 'virtual node' where you can deploy seemingly infinite number of pods, they'll handle all the orchestration below. I was reading the article thinking "Yes, finally, some code or use to implement interacting with AWS without a 3rd party lib" and then I got to this section at the bottom: > In this post, we showed you the flexibility of Kubernetes. It’s arguably an extreme option and one that doesn’t even exist today (the Amazon ECS ACK controller isn’t available but please thumb up this roadmap proposal if you are intrigued). If anything, it proves that Kubernetes can be a lot of things to a lot of different people and defining it remains a work in progress (at least in my head). As usual, Amazon is playing catch-up in actually orchestrating their services in a non-AWS-centric way. Very deceptive to create a blog post and not disclose it's not real software at the end.

coredog64 4y ago
The ACK exists and allows management of AWS services. What doesn’t exist is the ECS-specific ACK.

IceWreck 4y ago

The last one (Kubernetes is my control plane but not my data plane) is completely stupid. Instead of that why not use terraform or equivalents ? Paying for EKS just to use its API to provision and manage other AWS resources is an interesting idea, but not something I'd use in production.

jhoelzel 4y ago

I love how conveniently on-premise kubernetes is missing. Or self-managed in the cloud. The whole point of kubernetes is to not think about your servers anymore because they in essence can be replaced at any time. This would only provide absolute vendor lock-in. Of course you can use any external service there is (like a database) but you should never put all your eggs into one basked and trust only one hoster. The principle remains the same: its just someone elses computer. Storage and networking are solved in a lot of places at a fraction of the cost and that is really just the beginning., but also the whole point of the exercise: to be able to move your workloads from wherever to wherever

awill 4y ago
>>The whole point of kubernetes is to not think about your servers anymore because they in essence can be replaced at any time. Isn't that also the case for VMs?
- pixl97 4y ago
  How are you orchestrating the configuration of your VMs?
  - mmh0000 4y ago
    Terraform (for cloudy stuff), kickstart and ansible (for on-prem stuff) would be my first go-to(s) for automatic installation, configuration, and config management. https://www.terraform.io/ https://www.terraform.io/ https://en.wikipedia.org/wiki/Kickstart_(Linux) https://en.wikipedia.org/wiki/Kickstart_(Linux) https://www.ansible.com/ https://www.ansible.com/
    - jhoelzel 4y ago
      The question was not "what tools do you use" but how do you achieve orchestration on the same level as kubernetes. Your machines are still named. Your resources still multiplicative. Your setup is not idempotent. Don't get me wrong, i have been using ansible and terraform for over a decade too, but this truly is an apple and oranges comparison.