11 ms·

The Kubernetes Effect

163 points by sdiepend 9y ago

sacheendra 9y ago
A consequence of the "Kubernetes Effect" is that while distributed systems are easy to build and use, a lot of developers lose sight of the fundamental problems which make distributed systems difficult. For example, the sidecar in a sidecar pattern might fail while the application is running and the system can get stuck in weird states. The developer still needs to understand fundamentally how the system works. Eschewing deeper knowledge just because it is easy to use is trap in this case. While the article compares Kubernetes to JVM, Kubernetes can fail in a lot more hard to debug ways than the JVM right now. I don't know if this semantic gap between distributed systems like Kubernetes and monolithic systems like JVM can ever be bridged.
- majidazimi 9y ago
  > A consequence of the "Kubernetes Effect" is that while distributed systems are easy to build and use, a lot of developers lose sight of the fundamental problems which make distributed systems difficult. I would extend this to cloud as well. The more prevalent cloud becomes, the more ignorant developers become. It's like: I have Mathematica license, who cares how to calculate function derivative?
  - RRRA 9y ago
    More generally, society's achievement currently rely on a workforce that gets more and more specialized. We are bound to fragment every sector into sub-niche where specialists in functions, general programming and infrastructure resources cooperate on their boundaries without being able to quite understand what the others are doing.
  - user5994461 9y ago
    The cloud is awesome. It's infinitely more efficient than dealing with Dell and HP.
- speby 9y ago
  > ... distributed systems are easy to build and use ... I would not say distributed systems are easy to build or use. I think Kubernetes makes distributed systems _easier_ but definitely not easy in general, or at scale. Just easier than doing it all by hand/manually.
falcolas 9y ago
<rant> And yet finding people who can reliably install K8s from scratch, who understand what's going on under the hood, remains remarkably close to 0. How many people can, within a few hours, tell you how Kubernetes runs DNS, and how it routes packets between containers by default? How do you run an integrated DNS which uses, say, my_service.service.my_namespace instead of my_service.my_namespace? I've found that most installs of k8s have been made using defaults, using tooling that Google has provided. We hired one such administrator, but when asked anything outside of how to run kubectl, they just shrugged and said "it never came up". The codebase is vast, complicated, and there are few experts who live outside of Google. And it's getting more vast, more complicated on a quarterly basis. It bothers me how far operations has gone from "providing reliable systems on which we run software" to "offload work onto the developer at any cost". </rant> I realize that a lot of this is because of scarcity. The good devops folks (i.e. those who are both competent generalist sysadmins and competent generalist programmers) are few and expensive. That makes pre-packaged "full stack" solutions like GAE, Kubernetes, and Fargate very appealing to leadership. "You don't need an operations department to act as a huge drain on your revenue, just re-use your developers" holds a lot of appeal for those high up in the food chain. It's even initially appealing to developers! But in the end, it makes as much sense as re-using your developers to do customer service.
- deleted 9y ago
  [deleted]
- caniszczyk 9y ago
  How many people understand how the Linux kernel works from top to bottom? There are more than a handful of cloud providers (AWS, Azure, Microsoft, Alibaba etc) that offer a completely managed Kubernetes experience, for most folks, this will be good enough and you don't need to understand everything in order to take advantage of Kubernetes, similar how you don't need to understand how the kernel (think POSIX) works: https://www.cncf.io/certification/software-conformance/ https://www.cncf.io/certification/software-conformance/
  - falcolas 9y ago
    You're right. You don't have to know anything about Linux to run software on it... until you do. Until you have to understand and modify swap. Until you have to understand and change the various schedulers (for both processes and disk operations). Until you have to troubleshoot networking problems. Until you have to change a kernel setting to avoid a 0-day exploit. Until you have to encrypt all communication because a client said so. Being on AWS or Azure or Microsoft doesn't shield you from these needs.
archgrove 9y ago
I'll stand by my assertion that for 99% of users (maybe even 99.99%), Kubernetes offers entirely the wrong abstraction. They don't want to run a container, they want to run an application (Node, Go, Ruby, Python, Java, whatever). The prevailing mythology is you should "containerize" everything and give it to a container orchestrator to run, but why? They had one problem, "Run an app". Now they have two, "Run a container that runs an app" and "maintain a container". Just give the app to a PAAS, and go home early. Most startups - most large companies - would be far better served with a real PAAS, rather than container orchestration. My encounters with container orchestrators is that ops teams spent inordinate amounts of time trying to bend them into a PAAS, rather than just starting with one. This is why I don't understand why this article lumps, e.g. Cloud Foundry in with K8S - they solve entirely different problems. My advice to almost every startup I speak to is "Just use Heroku; solve your business problems first". The article also mentions it enables "new set of distributed primitives and runtime for creating distributed systems that spread across multiple processes and nodes". I'll throw out my other assertion, which I always though was axiomatic - you want your system to be the least distributed you can make it at all times. Distributed systems are harder to reason about, harder to write, and harder to maintain. They fail in strange ways, and are so hard to get right, I'd bet I can find a hidden problem in yours within an hour of starting code review. Most teams running a non-trivial distributed system are coasting on luck rather than skill. This is not a reflection on them - just an inherent problem with building distributed logic. Computers are fast, and you are not Google. I've helped run multiple thousand TPS using Cloudfoundry, driving one of Europe's biggest retailers using just a few services. I'm now helping a startup unpick it's 18 "service" containerised system back to something that can actually be maintained. TLDR; containers as production app deployment artefacts have, in the medium and long term, caused more problems than they've solved for almost every case I've seen.
- merb 9y ago
  since you mentioned Cloudfoundry... I think it's a thousand times easier to get up and running with k8s, than with Cloudfoundry on Bare-Metal (no Cloud). It's also a thousand times easier to maintain. (Thanks CoreOS) Basically if you want a managed simple no maintance, no cost bare-metal K8S installation you basically just use tectonic/kubeadm and you get something which is self-containing, or close to self-containing. and the only things you need to get it done is actually way easier than reading through cf docs (I'm pretty sure bare-metal isn't even supported that easily). running some services on top of it is than pretty simple, especially if you want to use a single ip, insteand of roundobin dns (https://github.com/kubernetes/contrib/tree/master/keepalived-vip https://github.com/kubernetes/contrib/tree/master/keepalived...) and if you have k8s running, adding some PaaS layer on top (openshift) can be pretty simple.
mixmastamyk 9y ago
Any one have opinions on the Kubernetes admin cert linked? Wonder if it would significantly help in getting work and skipping the interview BS? https://www.cncf.io/certification/expert/ https://www.cncf.io/certification/expert/
- markbnj 9y ago
  I've been working with kubernetes since 2015 and running production workloads on GKE since 2016. Since you asked for opinions, mine is that the certs don't matter very much. CNCF plays them up, and they will probably have some impact in larger orgs as enterprises get on the train, but in the open source community from which most of the kubernetes momentum emanates there has never really been a ton of respect for formal certification programs, and this doesn't feel any different to me.
  - deleted 9y ago
    [deleted]
  - dsnuh 9y ago
    Just wanted to chime in to generally second this opinion, but with one exeception as it relates to hiring. While I don't think having a cert would help you get hired necessarily, it would probably influence a decision to get an interview. What really matters is if you know how to do real-world operational tasks with the knowledge, which will show up if your technical interviewers know k8s. If you are the first person they are hiring at the company to begin their k8s project, then you might have a real advantage with a cert. Personally, I've never been one to give undue respect to many of the certs on the basis of having them alone, but it can depend on where you interview. Some places love certs.
    - mixmastamyk 9y ago
      Yep, I suspected as much. Was just jealous of some nurses I know where their job interviews consist of: Is your license up to date? Are you an ax-murderer? Can you start Monday?
thedevopsguy 9y ago
Kubernetes by itself may be daunting for most teams. But I'm not sure I understand the backlash. Once you've built your application and it's been packaged (containerized) and deployed why would anyone care how its run. Also running a container in production and orchestration seem to be conflated somewhat in this thread and the use cases are very different. You can think of Kubernetes as an Automated SysAdmin . This is a bit reductive I know but it is useful to think of this way. You ask the sysadmin to run something for you and they tell you how to package it (tgz, war, zip etc) and they run it for you on hardware. The level of engagement that a dev has with getting his app running on hardware is no different to that of dealing with a sysadmin and with the admin requesting that your app is packagedin a container. Kubernetes out of the box will give you most of this functionality as long as you keep state outside of the cluster. There are also options on how to make the experience smoother. There also these tools to help too: * Openshift * Kubernetes + Rancher * Mesos If you need orchestration and scheduling. I am a little perplexed.

163 points by sdiepend 9y ago

A consequence of the "Kubernetes Effect" is that while distributed systems are easy to build and use, a lot of developers lose sight of the fundamental problems which make distributed systems difficult. For example, the sidecar in a sidecar pattern might fail while the application is running and the system can get stuck in weird states. The developer still needs to understand fundamentally how the system works. Eschewing deeper knowledge just because it is easy to use is trap in this case. While the article compares Kubernetes to JVM, Kubernetes can fail in a lot more hard to debug ways than the JVM right now. I don't know if this semantic gap between distributed systems like Kubernetes and monolithic systems like JVM can ever be bridged.

majidazimi 9y ago
> A consequence of the "Kubernetes Effect" is that while distributed systems are easy to build and use, a lot of developers lose sight of the fundamental problems which make distributed systems difficult. I would extend this to cloud as well. The more prevalent cloud becomes, the more ignorant developers become. It's like: I have Mathematica license, who cares how to calculate function derivative?
- RRRA 9y ago
  More generally, society's achievement currently rely on a workforce that gets more and more specialized. We are bound to fragment every sector into sub-niche where specialists in functions, general programming and infrastructure resources cooperate on their boundaries without being able to quite understand what the others are doing.
- user5994461 9y ago
  The cloud is awesome. It's infinitely more efficient than dealing with Dell and HP.
speby 9y ago
> ... distributed systems are easy to build and use ... I would not say distributed systems are easy to build or use. I think Kubernetes makes distributed systems _easier_ but definitely not easy in general, or at scale. Just easier than doing it all by hand/manually.

falcolas 9y ago

<rant> And yet finding people who can reliably install K8s from scratch, who understand what's going on under the hood, remains remarkably close to 0. How many people can, within a few hours, tell you how Kubernetes runs DNS, and how it routes packets between containers by default? How do you run an integrated DNS which uses, say, my_service.service.my_namespace instead of my_service.my_namespace? I've found that most installs of k8s have been made using defaults, using tooling that Google has provided. We hired one such administrator, but when asked anything outside of how to run kubectl, they just shrugged and said "it never came up". The codebase is vast, complicated, and there are few experts who live outside of Google. And it's getting more vast, more complicated on a quarterly basis. It bothers me how far operations has gone from "providing reliable systems on which we run software" to "offload work onto the developer at any cost". </rant> I realize that a lot of this is because of scarcity. The good devops folks (i.e. those who are both competent generalist sysadmins and competent generalist programmers) are few and expensive. That makes pre-packaged "full stack" solutions like GAE, Kubernetes, and Fargate very appealing to leadership. "You don't need an operations department to act as a huge drain on your revenue, just re-use your developers" holds a lot of appeal for those high up in the food chain. It's even initially appealing to developers! But in the end, it makes as much sense as re-using your developers to do customer service.

deleted 9y ago
[deleted]
caniszczyk 9y ago
How many people understand how the Linux kernel works from top to bottom? There are more than a handful of cloud providers (AWS, Azure, Microsoft, Alibaba etc) that offer a completely managed Kubernetes experience, for most folks, this will be good enough and you don't need to understand everything in order to take advantage of Kubernetes, similar how you don't need to understand how the kernel (think POSIX) works: https://www.cncf.io/certification/software-conformance/ https://www.cncf.io/certification/software-conformance/
- falcolas 9y ago
  You're right. You don't have to know anything about Linux to run software on it... until you do. Until you have to understand and modify swap. Until you have to understand and change the various schedulers (for both processes and disk operations). Until you have to troubleshoot networking problems. Until you have to change a kernel setting to avoid a 0-day exploit. Until you have to encrypt all communication because a client said so. Being on AWS or Azure or Microsoft doesn't shield you from these needs.

archgrove 9y ago

I'll stand by my assertion that for 99% of users (maybe even 99.99%), Kubernetes offers entirely the wrong abstraction. They don't want to run a container, they want to run an application (Node, Go, Ruby, Python, Java, whatever). The prevailing mythology is you should "containerize" everything and give it to a container orchestrator to run, but why? They had one problem, "Run an app". Now they have two, "Run a container that runs an app" and "maintain a container". Just give the app to a PAAS, and go home early. Most startups - most large companies - would be far better served with a real PAAS, rather than container orchestration. My encounters with container orchestrators is that ops teams spent inordinate amounts of time trying to bend them into a PAAS, rather than just starting with one. This is why I don't understand why this article lumps, e.g. Cloud Foundry in with K8S - they solve entirely different problems. My advice to almost every startup I speak to is "Just use Heroku; solve your business problems first". The article also mentions it enables "new set of distributed primitives and runtime for creating distributed systems that spread across multiple processes and nodes". I'll throw out my other assertion, which I always though was axiomatic - you want your system to be the least distributed you can make it at all times. Distributed systems are harder to reason about, harder to write, and harder to maintain. They fail in strange ways, and are so hard to get right, I'd bet I can find a hidden problem in yours within an hour of starting code review. Most teams running a non-trivial distributed system are coasting on luck rather than skill. This is not a reflection on them - just an inherent problem with building distributed logic. Computers are fast, and you are not Google. I've helped run multiple thousand TPS using Cloudfoundry, driving one of Europe's biggest retailers using just a few services. I'm now helping a startup unpick it's 18 "service" containerised system back to something that can actually be maintained. TLDR; containers as production app deployment artefacts have, in the medium and long term, caused more problems than they've solved for almost every case I've seen.

merb 9y ago
since you mentioned Cloudfoundry... I think it's a thousand times easier to get up and running with k8s, than with Cloudfoundry on Bare-Metal (no Cloud). It's also a thousand times easier to maintain. (Thanks CoreOS) Basically if you want a managed simple no maintance, no cost bare-metal K8S installation you basically just use tectonic/kubeadm and you get something which is self-containing, or close to self-containing. and the only things you need to get it done is actually way easier than reading through cf docs (I'm pretty sure bare-metal isn't even supported that easily). running some services on top of it is than pretty simple, especially if you want to use a single ip, insteand of roundobin dns (https://github.com/kubernetes/contrib/tree/master/keepalived-vip https://github.com/kubernetes/contrib/tree/master/keepalived...) and if you have k8s running, adding some PaaS layer on top (openshift) can be pretty simple.

mixmastamyk 9y ago

Any one have opinions on the Kubernetes admin cert linked? Wonder if it would significantly help in getting work and skipping the interview BS? https://www.cncf.io/certification/expert/ https://www.cncf.io/certification/expert/

markbnj 9y ago
I've been working with kubernetes since 2015 and running production workloads on GKE since 2016. Since you asked for opinions, mine is that the certs don't matter very much. CNCF plays them up, and they will probably have some impact in larger orgs as enterprises get on the train, but in the open source community from which most of the kubernetes momentum emanates there has never really been a ton of respect for formal certification programs, and this doesn't feel any different to me.
- deleted 9y ago
  [deleted]
- dsnuh 9y ago
  Just wanted to chime in to generally second this opinion, but with one exeception as it relates to hiring. While I don't think having a cert would help you get hired necessarily, it would probably influence a decision to get an interview. What really matters is if you know how to do real-world operational tasks with the knowledge, which will show up if your technical interviewers know k8s. If you are the first person they are hiring at the company to begin their k8s project, then you might have a real advantage with a cert. Personally, I've never been one to give undue respect to many of the certs on the basis of having them alone, but it can depend on where you interview. Some places love certs.
  - mixmastamyk 9y ago
    Yep, I suspected as much. Was just jealous of some nurses I know where their job interviews consist of: Is your license up to date? Are you an ax-murderer? Can you start Monday?

thedevopsguy 9y ago

Kubernetes by itself may be daunting for most teams. But I'm not sure I understand the backlash. Once you've built your application and it's been packaged (containerized) and deployed why would anyone care how its run. Also running a container in production and orchestration seem to be conflated somewhat in this thread and the use cases are very different. You can think of Kubernetes as an Automated SysAdmin . This is a bit reductive I know but it is useful to think of this way. You ask the sysadmin to run something for you and they tell you how to package it (tgz, war, zip etc) and they run it for you on hardware. The level of engagement that a dev has with getting his app running on hardware is no different to that of dealing with a sysadmin and with the admin requesting that your app is packagedin a container. Kubernetes out of the box will give you most of this functionality as long as you keep state outside of the cluster. There are also options on how to make the experience smoother. There also these tools to help too: * Openshift * Kubernetes + Rancher * Mesos If you need orchestration and scheduling. I am a little perplexed.