19 ms·

Will Kubernetes Collapse Under the Weight of Its Complexity?

205 points by tim_sw 8y ago

deleted 8y ago
[deleted]
outside1234 8y ago
> kubectl scaffold mysql --generate This exists, its called helm, which in fact delivers the productivity gains the author is looking for.
- ridruejo 8y ago
  Correct, you can check out Kubeapps for a collection of apps packaged as Helm charts https://hub.kubeapps.com https://hub.kubeapps.com
- hyperrail 8y ago
  In the following paragraph, the author considers and appears to reject that solution: >Maybe the combination of operators and Helm charts covers this, but I don’t think that will cut it. Then we’re forcing developers to learn about two other things in addition to Kubernetes. Even if it’s just increasing the vocabulary and installing a new command line tool, it’s extra effort and thought. These things need to be first-class citizens and part of the Kubernetes out-of-the-box experience. They need to be accessible via kubectl.
  - justinjlynn 8y ago
    So, instead of keeping cross-cutting concerns separate and permitting them to evolve separately, the author wants K8S to become more complex, because they'd rather learn one more complex thing than multiple simpler things? Seems legit. I mean, why should I install multiple libraries? Why can't glibc wash my car? Will glibc collapse under the weight of it's own complexity -- even though it can't wash my car like I want it to?
    - dominotw 8y ago
      I think more charitable interpretation is that kuberbetes should be designed in such a way that these tools are a natural part of it. Think of active record being part of rails. Sure you can use rails without ar but that doesn't make sense.
      - justinjlynn 8y ago
        Using Ruby on Rails without ActiveRecord does make sense for a number of applications. For example, an edge caching API which uses Redis on the backend. Helm is a package manager for K8S but it isn't the package manager for K8S, which is probably why it's kept separate. ActiveRecord was the ORM for Rails, before it was extracted and called "ActiveRecord" as a separate thing. Helm, however, is coming from the opposite direction -- and until it becomes something you can't use K8S without, then I'm happy to see them kept separate.
- GiorgioG 8y ago
  I have Kubernetes in Action on my desk and I haven’t cracked it open yet because Kubernetes seems monstrously complex. Sure helm gets you up and running. When something goes wrong in prod at 3am, what do you do?
  - manigandham 8y ago
    What do you do now, without Helm or Kubernetes?
  - barrkel 8y ago
    K8s is actually fairly simple and self-evident once you understand about 3 core ideas: etcd being the repository of state, in particular the spec; controllers with control loops bringing status into line with spec (the core mechanism in k8s, this is key); and a familiarity with the options on pods & deployments, for initialisation, service discovery, liveness, readiness, etc. that let the system make decisions globally while you only worry about local status (this is most of what you need to know as a dev deploying a service). Don't buy the FUD. There's a lot of it about. K8s commoditises cloud providers. It's a strategic weapon against AWS lock-in.
  - core-questions 8y ago
    This is my issue as well. I'm not comfortable running magic commands that create vast realms of infrastructure that I am responsible for but do not understand. It's one thing if it's all going to run on someone else's Kube, but even still, I need to be able to troubleshoot the apps on top, which are my responsibility.
- flor1s 8y ago
  Also there is Draft (https://kubernetes.io/blog/2017/05/draft-kubernetes-container-development/ https://kubernetes.io/blog/2017/05/draft-kubernetes-containe...)
- geerlingguy 8y ago
  As someone new to Kubernetes, Helm is _really_ not appealing; it masks a _ton_ of complexity, and is akin to telling someone new to Docker to run all their applications off the 'Docker Library' official images on Docker Hub. With few exceptions, those cookie cutter examples are not ideal for running production applications; they're decent starting points for learning how they could be built, just like Helm could be (if it were a little less "here's a 1,000 line Kubernetes configuration file in a box, good luck!"-ish).
packetized 8y ago
tl;dr: author believes “No”. Betteridge’s Law still applies.
- baxtr 8y ago
  Copied from Wikipedia because I didn’t know: Betteridge's law of headlines is an adage that states: "Any headline that ends in a question mark can be answered by the word no." It is named after Ian Betteridge, a British technology journalist, although the principle is much older. As with similar "laws" (e.g., Murphy's law), it is intended to be humorous rather than the literal truth.
- packetized 8y ago
  Not sure why the downvotes; merely pointing out that Betteridge’s Law is once again proven out.
nicodjimenez 8y ago
The funny thing about Kubernetes is that I read somewhere that it's not actually widely used within Google (please correct me if I'm wrong). Given that's it's a complicated piece of software and that Google has extremely complicated requirements, that's pretty concerning. To me, we've barely starting to solve deployments, the idea of Kubernetes seems a little ahead of it's time. If you're using GPC I do hear it's really great. But then, what's the huge benefit of Kubernetes in the first place? A tiny bit less of vendor lock in?
- bhouston 8y ago
  Kubernetes is newer than Google's Borg infrastructure I understand. https://kubernetes.io/blog/2015/04/borg-predecessor-to-kubernetes/ https://kubernetes.io/blog/2015/04/borg-predecessor-to-kuber...
- mooreds 8y ago
  >I read somewhere that it's not actually widely used within Google Borg and Container Engine: https://www.quora.com/Does-Google-use-the-Open-Source-Kubernetes-or-a-version-of-Borg-for-their-own-container-management https://www.quora.com/Does-Google-use-the-Open-Source-Kubern...
- kozikow 8y ago
  If someone thinks kubernetes is complicated, they would cry during their first month with the borg.
  - ebikelaw 8y ago
    I would argue that borg has much less initial complexity than kubernetes. Because of Google's integrated systems you can take any local binary and run it in borg with a five-line config file and one command. You can't get started that quickly with kubernetes. If you understand the implications of running N replicas of a command on <= N machines in the cloud, borg does exactly what you want. Most of the people I met at Google complaining about borg just didn't understand how a program gets executed at all ... borg was the least of their problems. Also borg is easier to pronounce.
nimish 8y ago
kubernetes is a classic case of a tool designed for consultants and companies to sell consulting services (Including cloud services, which is why every cloud provider leapt onto it). In like 90% of the cases when someone used Kubernetes, Docker Swarm would have easily sufficed.
- chrisweekly 8y ago
  Docker Swarm is a hot mess.
  - jhanschoo 8y ago
    I'm inclined to agree with you, but you need to substantiate your thesis with evidence.
    - raarts 8y ago
      If you are, so would you.
  - coding123 8y ago
    I've used it extensively and have been happy with it. I've never seen more fud in HN than in swarm vs kub threads.
  - tuananh 8y ago
    isn't docker swarm declared dead by docker team?
    - coding123 8y ago
      This is FUD
    - BretFisher 8y ago
      nope https://www.bretfisher.com/is-swarm-dead-answered-by-a-docker-captain/ https://www.bretfisher.com/is-swarm-dead-answered-by-a-docke...
  - 8y ago
pat2man 8y ago
Running OpenShift has felt like early Rails to me. You can get started with a hosted version, switch to on AWS easily and dive deep down into Kubernetes whenever you feel ready. It is also opinionated so it’s easy to get started on the golden path and modify it to suit your needs. The only real frustration has been upgrading clusters which has gotten easier each new release.
- keir-rex 8y ago
  I’m new to Openshift (about a month in) after a year of low level (hand rolled HA cluster) kubernetes experience. I don’t rate the experience in Openshift. It seems like they are trying to tack things on which are superfluous to most teams requirements, loosely defined, and not well advertised. I’m constantly trying to figure out what it’s hiding from the Kubernetes layer or what it is being manipulated to provide its behaviour. I personally wouldn’t recommend Openshift->Kubernetes but the other way round would be a better approach once you know you need the additional functionality. (Edit: fix typo)
jrs95 8y ago
The complexity of Kubernetes largely reflects the complexity of the problem. Nobody has delivered anything significantly simpler that hasn’t had a much smaller scope, and those tools approach the same level of complexity when composed with others to get the same level of functionality. But setting up Kubernetes is both well documented and automated on multiple cloud providers, whereas something like Nomad & Consul doesn’t really have a good end to end walk through to get you to the same destination as Kubernetes. I suppose if you’re fine with pushing that complexity into service clients you can avoid the need for a lot beyond what Nomad and Consul give you by themselves — but then you end up with the downsides that something like the Netflix microservices stack gets you. Fat clients ultimately leave the developer with more complexity, and Kubernetes helps you eliminate that in favor of more SRE/Ops/whatever-you’d-like-to-call-it complexity. Since cloud platforms can take a lot of the edge off of Ops complexity, that’s my preferred approach.
- atombender 8y ago
  I would also argue that Kubernetes is less complex than it seems at first glance. Yes, if you look at all the possible parts, and at the current monolithic codebase, there's a lot of complexity. It also supports umpteen cloud providers, volume providers, networking stacks, etc., and comes with a whole swathe of bootstrapping tools for various environments (e.g. AWS). But if you strip it down, Kubernetes is "simple": There's a consistent object store made out of JSON structures, and then there's a bunch of controllers listening to changes to that store to make stuff real. That is the core. Everything is, in principle, controllers mediating between the data model and the real world. Very elegant and orthogonal. You also have an API, a scheduler, and a thing called Kubelet that runs on each node to manage containers and report node-specific metrics. And of course you have Docker, though with 1.10 you can more easily run dockerless via containerd, which is a great thing indeed. The complexity comes from the operational part, when the pieces come together. And as you say, there's not really any way around it.
  - jrs95 8y ago
    It's certainly a lot easier to do "kubernetes the hard way" than it is to do "Linux from scratch", and I don't see a lot of FUD about how we should stop using Linux because it's too complex. That might seem like an extreme comparison, but I think the number & quality of comparable alternatives is similar between the two.
gtirloni 8y ago
TL;DR; Kubernetes needs to continue to focus on the developer experience but it's good enough for InfluxData's new cloud offering. The project has been listening: https://github.com/kubernetes/community/blob/master/sig-apps/README.md https://github.com/kubernetes/community/blob/master/sig-apps...
- hueving 8y ago
  Forming a SIG is just paying lip service. It remains to be seen if anything will really change or if kubernetes will suffocate itself with its own bloat.
bigdubs 8y ago
Kubernetes isn't supposed to be simple; it's supposed to be a box of tools that you pull from to represent literally any workload. Once you know what tools to ignore, and build scripts around the ones you need, it's very powerful. This line of thinking is like faulting the golang stdlib for having a lot of useful stuff in it.
- rorykoehler 8y ago
  Everything should be as simple as possible. It's the mark of good design
  - davidp 8y ago
    As simple as possible, but no simpler.
  - nine_k 8y ago
    There's no single design that suits everyone. It's a toolbox. The job of choosing the right tool is on you.
    - MBCook 8y ago
      There’s a big difference between a toolbox and a box with those same tools jumbled up inside. One is organized and sensible. The other is a mess. Just because they are both technically capable of the same things doesn’t mean we shouldn’t hold ourselves to the higher standard. (Haven’t used Kubernetes, just going with the analogy)
      - manigandham 8y ago
        Kubernetes has some bad organization and some terrible naming due to it's rapid pace of development, but it's getting better and is much nicer than any other existing system that can provide the same functionality. I think you'd be surprised how quickly you can understand > 80% of it with just an hour spent reading the docs.
      - 8y ago
Pyxl101 8y ago
Is Kubernetes really scalable to a meaningful extent? I feel like if I was going to set up a Kubernetes system, I'd need to plan from the beginning to have multiple clusters anyway, and then the utility of all the scheduling features would be considerably diminished since I'd have to plan to load balance applications across the clusters in some custom way. Yuck. The Kubernetes website (1) currently claims that it supports clusters of up to 5000 nodes, which is a decent amount but not enough to avoid having multiple clusters. Does anyone have experience operating multiple production clusters in a single territory as partitions for scaling reasons? What's the experience like? (1) https://kubernetes.io/docs/admin/cluster-large/ https://kubernetes.io/docs/admin/cluster-large/
- jrs95 8y ago
  If you have a need for more than 5000 nodes in a cluster, it should take a trivial amount of your resources to solve that problem IMO. The vast majority of Kuberentes users probably won’t even need a 500 node cluster, let alone a 5000 node cluster. It makes sense for them to prioritize optimizing for other things like developer experience to help address some of the concerns mentioned in this article over supporting even larger clusters right now.
- rickycook 8y ago
  nodes meaning compute nodes; not containers in case there was confusion
- manigandham 8y ago
  5000 nodes has proven to be a high enough ceiling for basically all workloads, especially given the size of individual servers available now in clouds. Easier to run smaller numbers of bigger servers, as always. Also Kubernetes does support federation for cross-cluster deployments (now named multi-cluster). Some cloud services even support ingress load balancing across these, or you can do that part yourself by simple sticking with the same ports, but it all works fine today. Nothing custom needed.
- geerlingguy 8y ago
  The other important distinction there is the current (soft-ish) limit of 110 pods per node; if you want to run hundreds or thousands of pods on a few giant (beefy CPU/RAM) nodes, you need to rethink that strategy and go with many smaller nodes, with the pods evenly spread across them.
superzadeh 8y ago
This is exactly one of the reason we picked kontena.io in our startup, and never regretted it. Also super excited about their approach to run on top of k8 with pharos.sh.
mwcampbell 8y ago
> available 99.5% of the time with decent alerting for operators to kick it An operator should never have to "kick" a service. It should repair itself, except for the occasional hardware replacement if one is working with bare metal. And for anything that's being sold as a product, as opposed to an internal tool, I think 99.9% availability should be the minimum. But I don't know enough about Kubernetes to say whether it's overkill at the scale of just a few servers.
- 49bc 8y ago
  How many “kicks” does it take to build a reasonably resilient distributed service? A lot!
  - wang_li 8y ago
    CAP theorem says it's impossible.
    - Xorlev 8y ago
      That isn't what the CAP theorem says at all. It says you need to pick your guarantees. Additionally, resilience is in the eye of the beholder: might mean AP for a service that needs to be up at the cost of consistency or CP for a service that detects when it can't achieve quorum and fail gratefully.
- smudgymcscmudge 8y ago
  > An operator should never have to "kick" a service. Have you ever been a sysadmin? There are very few services that don’t need a kick every now and then.
  - mwcampbell 8y ago
    Yes I have. And if a service ever needs a manual kick, I consider that a bug. At least when running in a public cloud.
bitL 8y ago
Kubernetes looks to me like one of those prototypical technologies where LEGO-style usage of Deep Learning in helping to set it up for whatever scenario is needed would be already doable and beneficial. I am wondering if Google is working on it already.
flor1s 8y ago
Isn't one of the appeals of Kubernetes to have a portable cloud environment, which means I can easily switch between Windows Azure, Google Cloud, Amazon Web Services, on premise, and even Minikube (localhost). Is there any simpler alternative for that?
- bdcravens 8y ago
  For many use cases, I'd imagine an abstraction atop of Kubernetes (like OpenShift or Rancher) is a good fit.
hharnisch 8y ago
I've gone to the last few KubeCons and given talks at two of them and I'd also consider myself to be more of an app developer than ops. The tone has been very much that Kubernetes is deeper in the stack than most developers want or need to be thinking about. Mantras like "kubectl is the new ssh" have become super popular. So Kubernetes ends up being the platform you build your tools developers deploy their applications with -- if you work on ops. The problem seems to be that there's not a lot of agreement on what those tools actually look like. What Kubernetes does end up doing is providing a consistent API to deploy workloads across (many, but not all) cloud providers. Over time we'll see better and better developer facing solutions built on top of Kubernetes, rather than part of Kubernetes.
- flor1s 8y ago
  The problem with saying "kubectl is the new ssh" is that it is simply not true in my opinion. Something more akin to "kubectl is to controlling a cluster as ssh is to controlling a server" would be more accurate I think. The point the OP makes about "Most Developers Don’t Have Google-Scale Problems" is true, I don't think you should use Kubernetes if your app consists of just a website and a database. But do people working on such (relatively simple) apps really consider using Kubernetes?
  - sidlls 8y ago
    Yes, as part of a terrible feedback loop and resume driven development.
  - geerlingguy 8y ago
    IMO, only if you're working on like 100+ of them. If you just maintain a website using a traditional LAMP/LEMP stack, Rails, Node.js, or something like that it still makes more sense (unless you want to be 'trendy') to stick to primitives or use more managed hosting. But if you're maintaining a fleet of independent sites, Kubernetes' scheduling can make sense, despite the inherent complexity (TBH, you're going to have a similar level of complexity managing the same kind of scale with any other tool).
manigandham 8y ago
Microsoft Word is also incredibly complex software with decades of development and features, and yet it's just a word processor. Everyone uses a small subset of the actual functionality which is why the entire system can be complex and simple at the same time, depending on your needs. It's exactly the same with Kubernetes. It's just clustering software that ties multiple servers together to give you an PaaS-like workspace to run containers, but there are 1000s of details you can use if you need them to build much more intelligence into operations. If you don't need them, you can just run a single container and still benefit from simple declarative deployments and automated oversight.
- MBCook 8y ago
  But it’s trivial to start typing in Word. Spell checking is easy, as are basic formatting operations. Loading/saving work the way you’d expect. Yes you can write a dissertation with a ton of support from Word to make you life easier, but doing simple things is simple. It sounds like that’s what missing from kubectl. Even for a small start it takes a lot of knowledge. To continue the Word analogy, that sounds like LaTeX. It’s very powerful, but no normal person is going to get started for the first time very fast for basic tasks. Certainly not compared to Word. The Rails analogy from the article seems very apt.
  - tootie 8y ago
    It's true and I think that there's a base case for orchestration that is to autoscale nodes based CPU/memory consumption behind a load balancer that would get 90% of projects off the ground without needing to think about yaml files.
  - manigandham 8y ago
    Doing simple things in Kubernetes is also simple. It's 1 line to get a pod up and running: kubectl run appname --image=yourimage You can then graduate to a basic YAML config file with a few lines and update with: kubectl apply -f yourfile.yaml At some point, if you want to build a distributed application, you need to know the concepts involved. Looking at it relatively, you needed to learn and deal with many more low-level details before Kubernetes existed, so it's actually quite an improvement to what we had just a few years ago.
jtwaleson 8y ago
Heroku / Cloud Foundry offer exactly what the author points towards: a very simple user interface for developers. InfluxDB will of course needs stateful applications so it's not a good use case for them. The Cloud Foundry community has started exploring a switch from their own container management system to K8s. If that becomes real, CF would "just" become a nice user interface on top of k8s. The right move imho.
- EtienneK 8y ago
  This is basically what OpenShift is. Highly recommended.
- troytop 8y ago
  The cloudfoundry-incubator/eirini project is where the CF+K8s scheduler work is going on. Related to this, SUSE and IBM have already released distributions of CF that run on Kubernetes. I'm biased (I work on one of these) but I really think this is the most expedient way to enable PaaS features on K8s.
stormbeard 8y ago
> Scaffold generators for common elements would be great. Need a MySQL database? Having a command like [...] to create a stateful set, service and everything else necessary would go a long way. Then just a single command to deploy the scaffold into your k8s environment and you’d have a production-worthy database in just a few console commands. The same could be created for the popular application frameworks, message brokers, and anything else we can think of. Rook sort of does this. You deploy a Rook operator, then just one other kubectl command to get an object store, database, shared filesystem, etc... https://blog.rook.io/rooks-framework-for-cloud-native-storage-orchestration-c66278014df7 https://blog.rook.io/rooks-framework-for-cloud-native-storag...
hardwaresofton 8y ago
This blog post starts under a false premise. Kubernetes is not for app developers, it is the substrate on which applications, databases, and other workloads run. Just like you wouldn't want an application developer SSHing into machines in production (assuming you have ops people), you don't want them to use kubernetes, except kubernetes has done one better -- it's abstracted so well (especially with the introduction and widespread use of Custom Resource Definitions AKA CRDs) that you can let them write resource definitions, which are declarative representations of the resources they will need for their application, and run those. Coming from someone who gave a talk at Kubecon I'm very surprised to read something like this. Maybe I'm the one with the misunderstanding, but I'm going to try and refute the things this article said/is implying. 1. Kubernetes is complex This is kind of right, but it's also kind of not -- Kubernetes is essentially complex, given that it encourages write-once solutions to all the problems it faces. Here are the pieces that make a basic Kubernetes "cluster": - apiserver => you send commands to this to change/query cluster state - controller-manager => works to make your ensure that the cluster is in the state you want it to be (making workloads replicate/restart/etc) - scheduler => figures out where to put workloads - kubelet => runs containers -- one on each node that can do work - kube-proxy => maintains the routing infrastructure necessary to enable containers on any node to hit a container on another one. All of those pieces are needed -- the only concession I would make is that they could all be in the same daemon (one executable), but that's actually worse at scale, and harder to debug -- all of these services can produce a lot of logs. 2. Application developers can't use kubernetes as it is Application developers can use kubernetes as it is. Learning to write a kubernetes resource definition is not any harder than figuring out the conventions and configuration you have to write for Heroku, or AWS ElasticBeanstalk, or AWS ECS. In fact, I would argue that it's simpler. We've touched on another problem here -- the competitor for kubernetes is not SSH, it's not heroku -- it's tools like CloudFormation/ECS. I don't know if you've CloudFormation, but it's kind of a clusterfuck, hard to set up quite right, and the dynamic yaml approach they've taken is enough rope for one clever developer to hang you and your whole team with. Bold prediction, but I think AWS is going to abandon CloudFormation and ECS in favor of Kubernetes resources once it stabilizes. OK, let's say you disagreed with everything I've said up until this point -- at the very least, you can deploy tools like the following to your kubernetes cluster: https://gitkube.sh https://gitkube.sh => heroku workflow https://helm.sh https://helm.sh => cloud-formation/elastic-beanstalk workflow (with kubernetes primitives) And presto, you have a completely different interface to your cluster, WITHOUT changing anything fundamental underneath. 3. Developers who only focus on the application-level are the goal Why would you even want this? Not only is it basically impossible to hide the underlying infrastructure so well that the application developer doesn't have to know about it, it's arguably not even a good idea. Take session management -- if you want to handle it in the context of more than one frontend running at a time, you generally outsource that state to a cache like redis. An application developer who grew up in this imaginary world where app developers never touch infrastructure is not who I want solving this issue, assuming there isn't a qualified ops person. If you needed to optimize even further, app-local caches could be deployed, but this requires knowledge of "sticky sessions" -- this very much is a deployment/infrastructure specific question, again, that app-only developer is just about useless here. I'm no hiring manager but the desire to stay an "application" developer who only worries about that part of the stack when the "application" as a whole is so much more would be a red flag for me. Even if you were delivering a desktop application, the developer who worries about underlying OS-specific enhancements (for example knowing how to optimize the app for MacOS) is the one I want, the one I want to pay the big bucks for. 4. It's hard to deploy the usual app+backing store+caching+worker pool structure The author touches on this a little bit with the "maybe operators and helm charts solve this", and that's exactly what the operator pattern (Custom Resource Definitions, AKA CRDs, plus custom controllers) were meant to solve -- now you can actually give declarative specifications of what you want your Redis/Postgres/Celery/whatever cluster to look like, and `kubectl apply`, and the platform handles it. There's arguably no difference here between how you'd use this and a tool like heroku. Also, for the record you can trivially extend `kubectl`: https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/ https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plug...
enos_feedler 8y ago
Kubernetes is the Android of the datacenter. Android OEMs are Samsung, HTC, Huawei, etc. Kubernetes OEMs are Google, Amazon, Microsoft etc. With Android, a large % of consumer market gains access to an app ecosystem by choosing Android. Similarly, businesses will _eventually_ gain access to an ecosystem of B2B applications by adopting kubernetes. Kubernetes will be the OS for business.
- dfee 8y ago
  For a number of reasons, I think this metaphor fails. K8s appears more open than Android, is not largely controlled by one corporation, doesn’t come crippled with vendor services, etc.
  - enos_feedler 8y ago
    No metaphor is perfect. I highlighted where I thought it makes sense (in the ecosystem sense of apps and modules being built on top of it).
tootie 8y ago
I think https://draft.sh https://draft.sh is trying to address this but it's still early in development.
hliyan 8y ago
This whole image, to me, represents a big problem with software engineering today: https://twitter.com/dankohn1/status/989956137603747840 https://twitter.com/dankohn1/status/989956137603747840 The industry is full of engineers who are experts in weirdly named "technologies" (which are really just products and libraries) but have no idea how the actual technologies (e.g. TCP/IP, file systems, memory hierarchy etc.) work. I don't know what to think when I meet engineers who know how to setup an ELB on AWS but don't quite understand what a socket is...
- ramblerman 8y ago
  That you both have your domains of knowledge? I mean should the electrical engineer worry that you know what a socket is but not how it actually travels over the wire?
  - Vendan 8y ago
    Manchester encoding? (assuming twisted pair ethernet, at least last time I looked into this stuff). And no, while I don't expect a developer to be able to build a NIC from first principles, I do expect someone writing a distributed application to at least know the difference between TCP and UDP. (cause that's actually HIGHLY relevant).
  - hliyan 8y ago
    Actually, the correct analogy here is an electrical engineer who knows the the General Electric Foobar technology and can use its console to design and build an electrical system, while another engineer knows the Alacatel FizzBuzz technology and believes its superior because it can build larger systems. Neither truly understands how resistance or current actually works, because they're all abstracted away under system specific terminologies.
- tananaev 8y ago
  I have a mixed feeling about this. On one hand I definitely agree that it's good to know lower level technologies and that's something I always ask people in the interviews. I think it's important because I know it. On the other hand there is no end on how low you can go in the technology stack. Do you need to know how sockets works underneath? Low level network protocols? Do you need to know how hardware works because it probably influenced decisions that were made on the low level of the software stack? I know what socket is, but maybe the only reason I know it is because I'm old and I started coding at the time when raw sockets was the only way to implement network communication. When we have modern libraries and frameworks do we really need low level knowledge?
zzbzq 8y ago
The way the author's 'scaffolding' idea should work is that you start but not using kubernetes at all, rather than using a easier version of it. Of course, 'not using it' already exists.. But what doesn't exist is the sweet way to transition. In particular, you can start off with Azure AppService, AWS Elastic Beanstalk, Google AppEngine. You could also go serverless. All these approaches allow rapid development and deployment with low ops overhead, and they'll actually scale and heal well. Ultimately, the services are doing the k8s type of stuff for you. To state that inversely, running kubernetes is like trying to run your own PaaS. (When put that way, it sounds dubious that so many people are trying to jump into k8s, but I'm not an expert on the $$ economics of devops.) The next gen evolution of the cloud platforms could really take this migration from PaaS to IaaS to a whole new level than it's at right now.
spockz 8y ago
Firstly, I had the same experience on at kubecon this year, most people I encountered were ops engineers or infra engineers. Maybe the application developers were there but they were less vocal. I can also imagine the subjects being rather specific and deep for your average application developer. Secondly, aren’t solutions like Helm supposed to take away the need for scaffolding? The problem with scaffolding is staying up to date with the new templates and usually results in the deployments not being updated anymore. Additionally, I have to say that getting started with K8s was quite easy because we already had experience with Docker. OpenShift was similar and has source2build which is very convenient. So I don’t percieve K8s to be hard to start to use. To use it ‘correctly’ and use all potential, yes that is harder but that holds for any product.
hennsen 8y ago
Basically agree that simple things should be simple and complex things possible - and that’s from my experience not yet met with standard k8s. And having no „end users“ (app developers) on a conference about tools that should serve exactly these is an interesting observation to investigate further. Having to install one mire tool to get ready for production apps installed with helm in one command is not asked too much, though. Then, slighty unrelated, but it comes to my mind: i wonder if this happy path thing works with influx, where the author is working. Can i have a simple single command and install everything i need to look at logs from my app and db server, see most important performance stats and http/ip access logs, geaphical as well as with notifications if certain, easily to be entered ( and in case of cou, io, ram and diskspace reasonable defaults like 80% or so) thresholds are met? Can i do that with only the free open source tools as the author expects it from the k8s ecosystem? Or do i get it when buying influx‘s professional service? So maybe it’s the job of, and an opportunity for, commercial companies to develop and sell such simplifying tools. At some place, developers time to develop all these things must be paid for - if millions of developers just use the perfectly polished open source tools - and a high percentage doesn’t even help in development with big reports, not to think of patches, what are the developers going to live from while doing the polishing/ simplifying?
- pauldix 8y ago
  Author here. We have more work to do to make the happy path with the set of Influx tools (the TICK stack) more turn key and easier to run. The entire feature set is available in the open source versions. The thing we keep commercial is HA and scale out clustering (either we operate for you on AWS or you buy on-premise). Our work on 2.0 of the platform should make the happy path much easier, but software is a process of continuous improvement. So I'd expect 3.0 to be even better and so on.
  - hennsen 8y ago
    Thanks! Sounds great and I‘ll look into it for the next project/use case!
yeukhon 8y ago
One major reason OpenStack is a messy project today is the amount of companies involved in thr foundation early on. It was a big problem when RedHat vs HP vs Rackspace vs XYZ.
EngineerBetter 8y ago
I was at KubeCon, and had a similar experience. Lots of engineers excites about all the technical possibilities, and less discussion of developer productivity. It reminds me a little of in the 00's everyone thought their company should write its own CMS. I think we're in danger of every writing their own PaaS. This is why things like Deis and Cloud Foundry exist. Most app developers should not have to understand the full depth and breadth of Kubernetes.
- Jyaif 8y ago
  Amen. Use PaaS as much as possible and fallback to running VMs as a last resort.
reacharavindh 8y ago
I'm a sysadmin and I attended KubeCon recently. I came back with a similar thought flow in mind. This one anecdote nailed the problem in my opinion - "Kubernetes makes simple things hard, and hard things possible." So, if you dont have things that you think are impossible, just don't pay the complexity tax. Real-life example : https://www.reddit.com/r/devops/comments/8byasq/is_kubernetes_worth_it/ https://www.reddit.com/r/devops/comments/8byasq/is_kubernete... Paraphrasing for discussion: Poster: A rails project with deployed to 6 servers running in production currently. Poster: During the asset compilation process, the servers often freeze. Poster: I need to manually remove servers from the load balancer and deploy one by one. Poster:I looked a lot into kubernetes and production containerization lately, and as far as I read it, it should solve the deployment and uptime issues. I imagine it'd be a lot easier to just switch containers instead of deploying with capistrano. I also really like the self-healing capabilities a lot. So, he/she hopes that Kubernetes will magically solve his problem(asset compilation freezes the server). I suppose in his/her mind, Kubernetes is the snake oil. Things that he/she failed to put thought into (and rather got revved up about Kubernetes): * Could I setup CI with a script that will perform the asset compilation once on one server and just rsync the final result to the prod servers? * Could I spend a couple of hours understanding the asset compilation process and find out why it freezes the server? * Could I learn more about load balancing, rolling deploys? I think this is the real problem in the tech field. People are running after shiny tools and hope to through tools at their problems all the while ignoring the basics. In this particular case, I think if they had a grey beard sysadmin who was grumpy to the devs, and enforced a strict release process, everyone would've been happier.
gant 8y ago
I think people misunderstand why Kubernetes exists. It is the reverse OpenStack. Kubernetes has the potential to be the one unified API of the cloud. A middleware for proprietary cloud APIs. A few resources, like load balancers, are already at a point where you barely have to care about the underlying cloud provider. With operators and aggregated API servers (especially if they'll be offered as a service) provisioning resources could follow one well-known standard. Calling it now, within the next 2 years cloud database providers like Compose will offer a way to CRUD resources via CRD/apimachinery-compatible services. Few more years and we'll have a generic yaml spec for these resources that work out of the box on multiple providers (probably with a bunch of annotations that are vendor specific).
erikb 8y ago
Exactly what I'm always saying. It's also nearly impossible in an Enterprise IT environment to get Kubernetes working on your laptop. Minikube and Docker Edge both seem to fail way too often. As a developer one wants to spin up a system to work on, then work on it, then push results to some repo. And this loop simply isn't possible (yet?). Also what the author didn't mention is that even the vanilla k8s stuff is already super complex. Let's assume you manage to setup a cluster somewhere and it continues to work for more than 2 days (rarely seen in real world work environments). Then you are faced with deploying your hello world app to work on. Just for a simple single-server nginx deployment with no files and no config, you already need to understand mutliple objects: deployments, replicasets, pods, containers, nodes, hosts, services, nodePorts, port-forwards, maybe even ingresses and controllers. That means until you feel well, even in a perfect environment, you need several days or weeks. And the documentation is not really helping you there. Yes, it's better than most enterprise grade documentation out there, but still it assumes a lot of stuff upfront. For instance, why should a developer even know what an ingress is and that it might be something he needs? Combine that huge learning overhead with nearly impossible network debugging, beta level stability, and the low possibility to "just use it", then you have a system that most developers will never touch. Docker itself might survive though, and one of the good things of this CNCF world is that other alternatives to docker also get a chance to improve on the existing system.
- tuananh 8y ago
  it's been extremely easy with Docker for desktop. Just 1 click a way from "enable kubernetes"... in Docker preferences.
  - erikb 8y ago
    That worked for me for 2 days. Then a manager was breathing down my neck to debug his cluster. So due to stress I overwrote the kubeconfig file with my managers kubeconfig. Now I can't find a way to get the docker kubeconfig back, resetting the cluster, reinstalling kubernetes etc does nothing, and now that I have uninstalled and reinstalled docker, kubernetes doesn't even come up anymore. I'm also not allowed to share debug information with the Docker company. No real help from Docker developers without debug info. Because they can't tell me how docker generates a kubeconfig or where it writes its logs without seeing my diagnostics output... Not the vanilla perfect world use case is what makes your software but how well your software manages edge cases.
cmorgan8506 8y ago
As an app developer, I can vouch for finding k8 to feel frustratingly complex. My current client work has recently shifted to using k8. I took the time to get minikube working locally to get a better understanding. It definitely helped, but I find the layers of abstraction hard to grok after not working with it for a while. I can see the value the tool offers, but I get the feeling it's supposed to be reserved for higher degrees of scale than the average 2-8 node app. Black box is how I feel about it sometimes. Hopefully I'll get more one on one time with it in the future. It seems like a really cool technology.
austinshea 8y ago
These articles seem common, recently. I don’t really understand what this sentiment is about. It’s a really useful orchestrator, in some cases. In other circumstances, it’s unnecessary complexity.
mverwijs 8y ago
"The number of things you need to create and worry about are a barrier to starting a project and iterating on it quickly." As an SRE / Ops person it is my take that Kubernetes addresses many of the complexity your startup should be worrying about, but doesn't because "it is not a feature". Yes, it is massively complex. That is because Ops is massively complex.
- deleted 8y ago
  [deleted]
mancerayder 8y ago
This is my personal experience on the matter as a DevOps consultant who periodically interviews in the traditional manner (as opposed to getting gigs from people that already worked with me). Despite having 15+ years of *nix experience, including internals, having a track record of building large scalable infra and knowing a few different programming languages, what happened to me was this: I was getting filtered out because I didn't have Docker and Kubernetes and even (at one point) Cloudformation and/or Terraform. No problem - I learned those things (minus Kubernetes, so far) quite quickly. Much more quickly than the grueling trial-by-fire years of Unix administration. I like to know how things work, not just how to use them. So if you wonder and worry about the state of enterprise IT some days, look no further than hiring managers themselves, who will pick a 25 year-old who writes YAML for some abstraction-of-an-abstraction system that does infrastructure under the hood, infrastructure that people kind of don't really try very hard to understand. After all, it's disposable thanks to infrastructure-as-code, right? How do I know this? Well, I've seen shop after shop that's suffered a spaghetti infrastructure, using all the latest and greatest, from AWS and Kubernetes and Docker and other abstraction layers above AWS. And what happens is that it gets so complex that no one knows what's really going on, and at the very least two common symptoms arise: people are terrified during releases and they take hours, with many people on a call together very late at night; they spend a fortune on extra instances (in the case of AWS) because they haven't properly worked out environment separations (they had trouble keeping them the same, or one of many other problems). A talented dev manager I used to work with used to complain that they had trouble hiring people who knew Javascript well, but they had expert after expert of some fancy JS framework try to interview, unable to answer the fundamentals-types of questions. I think it's similar with enterprise infrastructure. I don't know what to say. I hope things go full swing and people who know how things work under the hood can charge consulting dollars for fixing the fuckups. It's not enough to know YAML, you also need to have wisdom in maintaining complex infrastructure, understand the delicate balance between change and stability, and be able to troubleshoot when it goes wrong WITHOUT just 'rinsing and repeating' where you learn absolutely no lessons at all. [edit:] One theory for all of this. Some of the big shops (Google, FB, Netflix, etc.) did it right, and now everyone is trying to copy the style of infrastructure management, except doesn't have the talent or wisdom to do it well.
jgr447 8y ago
Not having used it in any meaningful way, after a lot of reading I am sometimes still unclear with the value proposition. The value imho could be in being able to package distributed applications and deploy across cloud providers or on prem, seamlessly. I don't think this is true though short of doing a lot of effort to abstract access to a gcp/aws/azure managed service (say, a db), which is probably a bad idea. If you take that away, then a lot of the replication, autoscaling, load balancing, failover etc. can be implemented using cloud providers without having to manage the complexity of k8s. Hope to be proven wrong here.

205 points by tim_sw 8y ago

deleted 8y ago

[deleted]

> kubectl scaffold mysql --generate This exists, its called helm, which in fact delivers the productivity gains the author is looking for.

ridruejo 8y ago
Correct, you can check out Kubeapps for a collection of apps packaged as Helm charts https://hub.kubeapps.com https://hub.kubeapps.com
hyperrail 8y ago
In the following paragraph, the author considers and appears to reject that solution: >Maybe the combination of operators and Helm charts covers this, but I don’t think that will cut it. Then we’re forcing developers to learn about two other things in addition to Kubernetes. Even if it’s just increasing the vocabulary and installing a new command line tool, it’s extra effort and thought. These things need to be first-class citizens and part of the Kubernetes out-of-the-box experience. They need to be accessible via kubectl.
- justinjlynn 8y ago
  So, instead of keeping cross-cutting concerns separate and permitting them to evolve separately, the author wants K8S to become more complex, because they'd rather learn one more complex thing than multiple simpler things? Seems legit. I mean, why should I install multiple libraries? Why can't glibc wash my car? Will glibc collapse under the weight of it's own complexity -- even though it can't wash my car like I want it to?
  - dominotw 8y ago
    I think more charitable interpretation is that kuberbetes should be designed in such a way that these tools are a natural part of it. Think of active record being part of rails. Sure you can use rails without ar but that doesn't make sense.
    - justinjlynn 8y ago
      Using Ruby on Rails without ActiveRecord does make sense for a number of applications. For example, an edge caching API which uses Redis on the backend. Helm is a package manager for K8S but it isn't the package manager for K8S, which is probably why it's kept separate. ActiveRecord was the ORM for Rails, before it was extracted and called "ActiveRecord" as a separate thing. Helm, however, is coming from the opposite direction -- and until it becomes something you can't use K8S without, then I'm happy to see them kept separate.
GiorgioG 8y ago
I have Kubernetes in Action on my desk and I haven’t cracked it open yet because Kubernetes seems monstrously complex. Sure helm gets you up and running. When something goes wrong in prod at 3am, what do you do?
- manigandham 8y ago
  What do you do now, without Helm or Kubernetes?
- barrkel 8y ago
  K8s is actually fairly simple and self-evident once you understand about 3 core ideas: etcd being the repository of state, in particular the spec; controllers with control loops bringing status into line with spec (the core mechanism in k8s, this is key); and a familiarity with the options on pods & deployments, for initialisation, service discovery, liveness, readiness, etc. that let the system make decisions globally while you only worry about local status (this is most of what you need to know as a dev deploying a service). Don't buy the FUD. There's a lot of it about. K8s commoditises cloud providers. It's a strategic weapon against AWS lock-in.
- core-questions 8y ago
  This is my issue as well. I'm not comfortable running magic commands that create vast realms of infrastructure that I am responsible for but do not understand. It's one thing if it's all going to run on someone else's Kube, but even still, I need to be able to troubleshoot the apps on top, which are my responsibility.
flor1s 8y ago
Also there is Draft (https://kubernetes.io/blog/2017/05/draft-kubernetes-container-development/ https://kubernetes.io/blog/2017/05/draft-kubernetes-containe...)
geerlingguy 8y ago
As someone new to Kubernetes, Helm is _really_ not appealing; it masks a _ton_ of complexity, and is akin to telling someone new to Docker to run all their applications off the 'Docker Library' official images on Docker Hub. With few exceptions, those cookie cutter examples are not ideal for running production applications; they're decent starting points for learning how they could be built, just like Helm could be (if it were a little less "here's a 1,000 line Kubernetes configuration file in a box, good luck!"-ish).

packetized 8y ago

tl;dr: author believes “No”. Betteridge’s Law still applies.

baxtr 8y ago
Copied from Wikipedia because I didn’t know: Betteridge's law of headlines is an adage that states: "Any headline that ends in a question mark can be answered by the word no." It is named after Ian Betteridge, a British technology journalist, although the principle is much older. As with similar "laws" (e.g., Murphy's law), it is intended to be humorous rather than the literal truth.
packetized 8y ago
Not sure why the downvotes; merely pointing out that Betteridge’s Law is once again proven out.

nicodjimenez 8y ago

The funny thing about Kubernetes is that I read somewhere that it's not actually widely used within Google (please correct me if I'm wrong). Given that's it's a complicated piece of software and that Google has extremely complicated requirements, that's pretty concerning. To me, we've barely starting to solve deployments, the idea of Kubernetes seems a little ahead of it's time. If you're using GPC I do hear it's really great. But then, what's the huge benefit of Kubernetes in the first place? A tiny bit less of vendor lock in?

bhouston 8y ago
Kubernetes is newer than Google's Borg infrastructure I understand. https://kubernetes.io/blog/2015/04/borg-predecessor-to-kubernetes/ https://kubernetes.io/blog/2015/04/borg-predecessor-to-kuber...
mooreds 8y ago
>I read somewhere that it's not actually widely used within Google Borg and Container Engine: https://www.quora.com/Does-Google-use-the-Open-Source-Kubernetes-or-a-version-of-Borg-for-their-own-container-management https://www.quora.com/Does-Google-use-the-Open-Source-Kubern...
kozikow 8y ago
If someone thinks kubernetes is complicated, they would cry during their first month with the borg.
- ebikelaw 8y ago
  I would argue that borg has much less initial complexity than kubernetes. Because of Google's integrated systems you can take any local binary and run it in borg with a five-line config file and one command. You can't get started that quickly with kubernetes. If you understand the implications of running N replicas of a command on <= N machines in the cloud, borg does exactly what you want. Most of the people I met at Google complaining about borg just didn't understand how a program gets executed at all ... borg was the least of their problems. Also borg is easier to pronounce.

nimish 8y ago

kubernetes is a classic case of a tool designed for consultants and companies to sell consulting services (Including cloud services, which is why every cloud provider leapt onto it). In like 90% of the cases when someone used Kubernetes, Docker Swarm would have easily sufficed.

chrisweekly 8y ago
Docker Swarm is a hot mess.
- jhanschoo 8y ago
  I'm inclined to agree with you, but you need to substantiate your thesis with evidence.
  - raarts 8y ago
    If you are, so would you.
- coding123 8y ago
  I've used it extensively and have been happy with it. I've never seen more fud in HN than in swarm vs kub threads.
- tuananh 8y ago
  isn't docker swarm declared dead by docker team?
  - coding123 8y ago
    This is FUD
  - BretFisher 8y ago
    nope https://www.bretfisher.com/is-swarm-dead-answered-by-a-docker-captain/ https://www.bretfisher.com/is-swarm-dead-answered-by-a-docke...
- 8y ago

pat2man 8y ago

Running OpenShift has felt like early Rails to me. You can get started with a hosted version, switch to on AWS easily and dive deep down into Kubernetes whenever you feel ready. It is also opinionated so it’s easy to get started on the golden path and modify it to suit your needs. The only real frustration has been upgrading clusters which has gotten easier each new release.

keir-rex 8y ago
I’m new to Openshift (about a month in) after a year of low level (hand rolled HA cluster) kubernetes experience. I don’t rate the experience in Openshift. It seems like they are trying to tack things on which are superfluous to most teams requirements, loosely defined, and not well advertised. I’m constantly trying to figure out what it’s hiding from the Kubernetes layer or what it is being manipulated to provide its behaviour. I personally wouldn’t recommend Openshift->Kubernetes but the other way round would be a better approach once you know you need the additional functionality. (Edit: fix typo)

jrs95 8y ago

The complexity of Kubernetes largely reflects the complexity of the problem. Nobody has delivered anything significantly simpler that hasn’t had a much smaller scope, and those tools approach the same level of complexity when composed with others to get the same level of functionality. But setting up Kubernetes is both well documented and automated on multiple cloud providers, whereas something like Nomad & Consul doesn’t really have a good end to end walk through to get you to the same destination as Kubernetes. I suppose if you’re fine with pushing that complexity into service clients you can avoid the need for a lot beyond what Nomad and Consul give you by themselves — but then you end up with the downsides that something like the Netflix microservices stack gets you. Fat clients ultimately leave the developer with more complexity, and Kubernetes helps you eliminate that in favor of more SRE/Ops/whatever-you’d-like-to-call-it complexity. Since cloud platforms can take a lot of the edge off of Ops complexity, that’s my preferred approach.

atombender 8y ago
I would also argue that Kubernetes is less complex than it seems at first glance. Yes, if you look at all the possible parts, and at the current monolithic codebase, there's a lot of complexity. It also supports umpteen cloud providers, volume providers, networking stacks, etc., and comes with a whole swathe of bootstrapping tools for various environments (e.g. AWS). But if you strip it down, Kubernetes is "simple": There's a consistent object store made out of JSON structures, and then there's a bunch of controllers listening to changes to that store to make stuff real. That is the core. Everything is, in principle, controllers mediating between the data model and the real world. Very elegant and orthogonal. You also have an API, a scheduler, and a thing called Kubelet that runs on each node to manage containers and report node-specific metrics. And of course you have Docker, though with 1.10 you can more easily run dockerless via containerd, which is a great thing indeed. The complexity comes from the operational part, when the pieces come together. And as you say, there's not really any way around it.
- jrs95 8y ago
  It's certainly a lot easier to do "kubernetes the hard way" than it is to do "Linux from scratch", and I don't see a lot of FUD about how we should stop using Linux because it's too complex. That might seem like an extreme comparison, but I think the number & quality of comparable alternatives is similar between the two.

gtirloni 8y ago

TL;DR; Kubernetes needs to continue to focus on the developer experience but it's good enough for InfluxData's new cloud offering. The project has been listening: https://github.com/kubernetes/community/blob/master/sig-apps/README.md https://github.com/kubernetes/community/blob/master/sig-apps...

hueving 8y ago
Forming a SIG is just paying lip service. It remains to be seen if anything will really change or if kubernetes will suffocate itself with its own bloat.

bigdubs 8y ago

Kubernetes isn't supposed to be simple; it's supposed to be a box of tools that you pull from to represent literally any workload. Once you know what tools to ignore, and build scripts around the ones you need, it's very powerful. This line of thinking is like faulting the golang stdlib for having a lot of useful stuff in it.

rorykoehler 8y ago
Everything should be as simple as possible. It's the mark of good design
- davidp 8y ago
  As simple as possible, but no simpler.
- nine_k 8y ago
  There's no single design that suits everyone. It's a toolbox. The job of choosing the right tool is on you.
  - MBCook 8y ago
    There’s a big difference between a toolbox and a box with those same tools jumbled up inside. One is organized and sensible. The other is a mess. Just because they are both technically capable of the same things doesn’t mean we shouldn’t hold ourselves to the higher standard. (Haven’t used Kubernetes, just going with the analogy)
    - manigandham 8y ago
      Kubernetes has some bad organization and some terrible naming due to it's rapid pace of development, but it's getting better and is much nicer than any other existing system that can provide the same functionality. I think you'd be surprised how quickly you can understand > 80% of it with just an hour spent reading the docs.
    - 8y ago

Pyxl101 8y ago

Is Kubernetes really scalable to a meaningful extent? I feel like if I was going to set up a Kubernetes system, I'd need to plan from the beginning to have multiple clusters anyway, and then the utility of all the scheduling features would be considerably diminished since I'd have to plan to load balance applications across the clusters in some custom way. Yuck. The Kubernetes website (1) currently claims that it supports clusters of up to 5000 nodes, which is a decent amount but not enough to avoid having multiple clusters. Does anyone have experience operating multiple production clusters in a single territory as partitions for scaling reasons? What's the experience like? (1) https://kubernetes.io/docs/admin/cluster-large/ https://kubernetes.io/docs/admin/cluster-large/

jrs95 8y ago
If you have a need for more than 5000 nodes in a cluster, it should take a trivial amount of your resources to solve that problem IMO. The vast majority of Kuberentes users probably won’t even need a 500 node cluster, let alone a 5000 node cluster. It makes sense for them to prioritize optimizing for other things like developer experience to help address some of the concerns mentioned in this article over supporting even larger clusters right now.
rickycook 8y ago
nodes meaning compute nodes; not containers in case there was confusion
manigandham 8y ago
5000 nodes has proven to be a high enough ceiling for basically all workloads, especially given the size of individual servers available now in clouds. Easier to run smaller numbers of bigger servers, as always. Also Kubernetes does support federation for cross-cluster deployments (now named multi-cluster). Some cloud services even support ingress load balancing across these, or you can do that part yourself by simple sticking with the same ports, but it all works fine today. Nothing custom needed.
geerlingguy 8y ago
The other important distinction there is the current (soft-ish) limit of 110 pods per node; if you want to run hundreds or thousands of pods on a few giant (beefy CPU/RAM) nodes, you need to rethink that strategy and go with many smaller nodes, with the pods evenly spread across them.

superzadeh 8y ago

This is exactly one of the reason we picked kontena.io in our startup, and never regretted it. Also super excited about their approach to run on top of k8 with pharos.sh.

mwcampbell 8y ago

> available 99.5% of the time with decent alerting for operators to kick it An operator should never have to "kick" a service. It should repair itself, except for the occasional hardware replacement if one is working with bare metal. And for anything that's being sold as a product, as opposed to an internal tool, I think 99.9% availability should be the minimum. But I don't know enough about Kubernetes to say whether it's overkill at the scale of just a few servers.

49bc 8y ago
How many “kicks” does it take to build a reasonably resilient distributed service? A lot!
- wang_li 8y ago
  CAP theorem says it's impossible.
  - Xorlev 8y ago
    That isn't what the CAP theorem says at all. It says you need to pick your guarantees. Additionally, resilience is in the eye of the beholder: might mean AP for a service that needs to be up at the cost of consistency or CP for a service that detects when it can't achieve quorum and fail gratefully.
smudgymcscmudge 8y ago
> An operator should never have to "kick" a service. Have you ever been a sysadmin? There are very few services that don’t need a kick every now and then.
- mwcampbell 8y ago
  Yes I have. And if a service ever needs a manual kick, I consider that a bug. At least when running in a public cloud.

bitL 8y ago

Kubernetes looks to me like one of those prototypical technologies where LEGO-style usage of Deep Learning in helping to set it up for whatever scenario is needed would be already doable and beneficial. I am wondering if Google is working on it already.

flor1s 8y ago

Isn't one of the appeals of Kubernetes to have a portable cloud environment, which means I can easily switch between Windows Azure, Google Cloud, Amazon Web Services, on premise, and even Minikube (localhost). Is there any simpler alternative for that?

bdcravens 8y ago
For many use cases, I'd imagine an abstraction atop of Kubernetes (like OpenShift or Rancher) is a good fit.

hharnisch 8y ago

I've gone to the last few KubeCons and given talks at two of them and I'd also consider myself to be more of an app developer than ops. The tone has been very much that Kubernetes is deeper in the stack than most developers want or need to be thinking about. Mantras like "kubectl is the new ssh" have become super popular. So Kubernetes ends up being the platform you build your tools developers deploy their applications with -- if you work on ops. The problem seems to be that there's not a lot of agreement on what those tools actually look like. What Kubernetes does end up doing is providing a consistent API to deploy workloads across (many, but not all) cloud providers. Over time we'll see better and better developer facing solutions built on top of Kubernetes, rather than part of Kubernetes.

flor1s 8y ago
The problem with saying "kubectl is the new ssh" is that it is simply not true in my opinion. Something more akin to "kubectl is to controlling a cluster as ssh is to controlling a server" would be more accurate I think. The point the OP makes about "Most Developers Don’t Have Google-Scale Problems" is true, I don't think you should use Kubernetes if your app consists of just a website and a database. But do people working on such (relatively simple) apps really consider using Kubernetes?
- sidlls 8y ago
  Yes, as part of a terrible feedback loop and resume driven development.
- geerlingguy 8y ago
  IMO, only if you're working on like 100+ of them. If you just maintain a website using a traditional LAMP/LEMP stack, Rails, Node.js, or something like that it still makes more sense (unless you want to be 'trendy') to stick to primitives or use more managed hosting. But if you're maintaining a fleet of independent sites, Kubernetes' scheduling can make sense, despite the inherent complexity (TBH, you're going to have a similar level of complexity managing the same kind of scale with any other tool).

manigandham 8y ago

Microsoft Word is also incredibly complex software with decades of development and features, and yet it's just a word processor. Everyone uses a small subset of the actual functionality which is why the entire system can be complex and simple at the same time, depending on your needs. It's exactly the same with Kubernetes. It's just clustering software that ties multiple servers together to give you an PaaS-like workspace to run containers, but there are 1000s of details you can use if you need them to build much more intelligence into operations. If you don't need them, you can just run a single container and still benefit from simple declarative deployments and automated oversight.

MBCook 8y ago
But it’s trivial to start typing in Word. Spell checking is easy, as are basic formatting operations. Loading/saving work the way you’d expect. Yes you can write a dissertation with a ton of support from Word to make you life easier, but doing simple things is simple. It sounds like that’s what missing from kubectl. Even for a small start it takes a lot of knowledge. To continue the Word analogy, that sounds like LaTeX. It’s very powerful, but no normal person is going to get started for the first time very fast for basic tasks. Certainly not compared to Word. The Rails analogy from the article seems very apt.
- tootie 8y ago
  It's true and I think that there's a base case for orchestration that is to autoscale nodes based CPU/memory consumption behind a load balancer that would get 90% of projects off the ground without needing to think about yaml files.
- manigandham 8y ago
  Doing simple things in Kubernetes is also simple. It's 1 line to get a pod up and running: kubectl run appname --image=yourimage You can then graduate to a basic YAML config file with a few lines and update with: kubectl apply -f yourfile.yaml At some point, if you want to build a distributed application, you need to know the concepts involved. Looking at it relatively, you needed to learn and deal with many more low-level details before Kubernetes existed, so it's actually quite an improvement to what we had just a few years ago.

jtwaleson 8y ago

Heroku / Cloud Foundry offer exactly what the author points towards: a very simple user interface for developers. InfluxDB will of course needs stateful applications so it's not a good use case for them. The Cloud Foundry community has started exploring a switch from their own container management system to K8s. If that becomes real, CF would "just" become a nice user interface on top of k8s. The right move imho.

EtienneK 8y ago
This is basically what OpenShift is. Highly recommended.
troytop 8y ago
The cloudfoundry-incubator/eirini project is where the CF+K8s scheduler work is going on. Related to this, SUSE and IBM have already released distributions of CF that run on Kubernetes. I'm biased (I work on one of these) but I really think this is the most expedient way to enable PaaS features on K8s.

stormbeard 8y ago

> Scaffold generators for common elements would be great. Need a MySQL database? Having a command like [...] to create a stateful set, service and everything else necessary would go a long way. Then just a single command to deploy the scaffold into your k8s environment and you’d have a production-worthy database in just a few console commands. The same could be created for the popular application frameworks, message brokers, and anything else we can think of. Rook sort of does this. You deploy a Rook operator, then just one other kubectl command to get an object store, database, shared filesystem, etc... https://blog.rook.io/rooks-framework-for-cloud-native-storage-orchestration-c66278014df7 https://blog.rook.io/rooks-framework-for-cloud-native-storag...

hardwaresofton 8y ago

This blog post starts under a false premise. Kubernetes is not for app developers, it is the substrate on which applications, databases, and other workloads run. Just like you wouldn't want an application developer SSHing into machines in production (assuming you have ops people), you don't want them to use kubernetes, except kubernetes has done one better -- it's abstracted so well (especially with the introduction and widespread use of Custom Resource Definitions AKA CRDs) that you can let them write resource definitions, which are declarative representations of the resources they will need for their application, and run those. Coming from someone who gave a talk at Kubecon I'm very surprised to read something like this. Maybe I'm the one with the misunderstanding, but I'm going to try and refute the things this article said/is implying. 1. Kubernetes is complex This is kind of right, but it's also kind of not -- Kubernetes is essentially complex, given that it encourages write-once solutions to all the problems it faces. Here are the pieces that make a basic Kubernetes "cluster": - apiserver => you send commands to this to change/query cluster state - controller-manager => works to make your ensure that the cluster is in the state you want it to be (making workloads replicate/restart/etc) - scheduler => figures out where to put workloads - kubelet => runs containers -- one on each node that can do work - kube-proxy => maintains the routing infrastructure necessary to enable containers on any node to hit a container on another one. All of those pieces are needed -- the only concession I would make is that they could all be in the same daemon (one executable), but that's actually worse at scale, and harder to debug -- all of these services can produce a lot of logs. 2. Application developers can't use kubernetes as it is Application developers can use kubernetes as it is. Learning to write a kubernetes resource definition is not any harder than figuring out the conventions and configuration you have to write for Heroku, or AWS ElasticBeanstalk, or AWS ECS. In fact, I would argue that it's simpler. We've touched on another problem here -- the competitor for kubernetes is not SSH, it's not heroku -- it's tools like CloudFormation/ECS. I don't know if you've CloudFormation, but it's kind of a clusterfuck, hard to set up quite right, and the dynamic yaml approach they've taken is enough rope for one clever developer to hang you and your whole team with. Bold prediction, but I think AWS is going to abandon CloudFormation and ECS in favor of Kubernetes resources once it stabilizes. OK, let's say you disagreed with everything I've said up until this point -- at the very least, you can deploy tools like the following to your kubernetes cluster: https://gitkube.sh https://gitkube.sh => heroku workflow https://helm.sh https://helm.sh => cloud-formation/elastic-beanstalk workflow (with kubernetes primitives) And presto, you have a completely different interface to your cluster, WITHOUT changing anything fundamental underneath. 3. Developers who only focus on the application-level are the goal Why would you even want this? Not only is it basically impossible to hide the underlying infrastructure so well that the application developer doesn't have to know about it, it's arguably not even a good idea. Take session management -- if you want to handle it in the context of more than one frontend running at a time, you generally outsource that state to a cache like redis. An application developer who grew up in this imaginary world where app developers never touch infrastructure is not who I want solving this issue, assuming there isn't a qualified ops person. If you needed to optimize even further, app-local caches could be deployed, but this requires knowledge of "sticky sessions" -- this very much is a deployment/infrastructure specific question, again, that app-only developer is just about useless here. I'm no hiring manager but the desire to stay an "application" developer who only worries about that part of the stack when the "application" as a whole is so much more would be a red flag for me. Even if you were delivering a desktop application, the developer who worries about underlying OS-specific enhancements (for example knowing how to optimize the app for MacOS) is the one I want, the one I want to pay the big bucks for. 4. It's hard to deploy the usual app+backing store+caching+worker pool structure The author touches on this a little bit with the "maybe operators and helm charts solve this", and that's exactly what the operator pattern (Custom Resource Definitions, AKA CRDs, plus custom controllers) were meant to solve -- now you can actually give declarative specifications of what you want your Redis/Postgres/Celery/whatever cluster to look like, and `kubectl apply`, and the platform handles it. There's arguably no difference here between how you'd use this and a tool like heroku. Also, for the record you can trivially extend `kubectl`: https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/ https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plug...

enos_feedler 8y ago

Kubernetes is the Android of the datacenter. Android OEMs are Samsung, HTC, Huawei, etc. Kubernetes OEMs are Google, Amazon, Microsoft etc. With Android, a large % of consumer market gains access to an app ecosystem by choosing Android. Similarly, businesses will _eventually_ gain access to an ecosystem of B2B applications by adopting kubernetes. Kubernetes will be the OS for business.

dfee 8y ago
For a number of reasons, I think this metaphor fails. K8s appears more open than Android, is not largely controlled by one corporation, doesn’t come crippled with vendor services, etc.
- enos_feedler 8y ago
  No metaphor is perfect. I highlighted where I thought it makes sense (in the ecosystem sense of apps and modules being built on top of it).

tootie 8y ago

I think https://draft.sh https://draft.sh is trying to address this but it's still early in development.

hliyan 8y ago

This whole image, to me, represents a big problem with software engineering today: https://twitter.com/dankohn1/status/989956137603747840 https://twitter.com/dankohn1/status/989956137603747840 The industry is full of engineers who are experts in weirdly named "technologies" (which are really just products and libraries) but have no idea how the actual technologies (e.g. TCP/IP, file systems, memory hierarchy etc.) work. I don't know what to think when I meet engineers who know how to setup an ELB on AWS but don't quite understand what a socket is...

ramblerman 8y ago
That you both have your domains of knowledge? I mean should the electrical engineer worry that you know what a socket is but not how it actually travels over the wire?
- Vendan 8y ago
  Manchester encoding? (assuming twisted pair ethernet, at least last time I looked into this stuff). And no, while I don't expect a developer to be able to build a NIC from first principles, I do expect someone writing a distributed application to at least know the difference between TCP and UDP. (cause that's actually HIGHLY relevant).
- hliyan 8y ago
  Actually, the correct analogy here is an electrical engineer who knows the the General Electric Foobar technology and can use its console to design and build an electrical system, while another engineer knows the Alacatel FizzBuzz technology and believes its superior because it can build larger systems. Neither truly understands how resistance or current actually works, because they're all abstracted away under system specific terminologies.
tananaev 8y ago
I have a mixed feeling about this. On one hand I definitely agree that it's good to know lower level technologies and that's something I always ask people in the interviews. I think it's important because I know it. On the other hand there is no end on how low you can go in the technology stack. Do you need to know how sockets works underneath? Low level network protocols? Do you need to know how hardware works because it probably influenced decisions that were made on the low level of the software stack? I know what socket is, but maybe the only reason I know it is because I'm old and I started coding at the time when raw sockets was the only way to implement network communication. When we have modern libraries and frameworks do we really need low level knowledge?

zzbzq 8y ago

The way the author's 'scaffolding' idea should work is that you start but not using kubernetes at all, rather than using a easier version of it. Of course, 'not using it' already exists.. But what doesn't exist is the sweet way to transition. In particular, you can start off with Azure AppService, AWS Elastic Beanstalk, Google AppEngine. You could also go serverless. All these approaches allow rapid development and deployment with low ops overhead, and they'll actually scale and heal well. Ultimately, the services are doing the k8s type of stuff for you. To state that inversely, running kubernetes is like trying to run your own PaaS. (When put that way, it sounds dubious that so many people are trying to jump into k8s, but I'm not an expert on the $$ economics of devops.) The next gen evolution of the cloud platforms could really take this migration from PaaS to IaaS to a whole new level than it's at right now.

spockz 8y ago

Firstly, I had the same experience on at kubecon this year, most people I encountered were ops engineers or infra engineers. Maybe the application developers were there but they were less vocal. I can also imagine the subjects being rather specific and deep for your average application developer. Secondly, aren’t solutions like Helm supposed to take away the need for scaffolding? The problem with scaffolding is staying up to date with the new templates and usually results in the deployments not being updated anymore. Additionally, I have to say that getting started with K8s was quite easy because we already had experience with Docker. OpenShift was similar and has source2build which is very convenient. So I don’t percieve K8s to be hard to start to use. To use it ‘correctly’ and use all potential, yes that is harder but that holds for any product.

hennsen 8y ago

Basically agree that simple things should be simple and complex things possible - and that’s from my experience not yet met with standard k8s. And having no „end users“ (app developers) on a conference about tools that should serve exactly these is an interesting observation to investigate further. Having to install one mire tool to get ready for production apps installed with helm in one command is not asked too much, though. Then, slighty unrelated, but it comes to my mind: i wonder if this happy path thing works with influx, where the author is working. Can i have a simple single command and install everything i need to look at logs from my app and db server, see most important performance stats and http/ip access logs, geaphical as well as with notifications if certain, easily to be entered ( and in case of cou, io, ram and diskspace reasonable defaults like 80% or so) thresholds are met? Can i do that with only the free open source tools as the author expects it from the k8s ecosystem? Or do i get it when buying influx‘s professional service? So maybe it’s the job of, and an opportunity for, commercial companies to develop and sell such simplifying tools. At some place, developers time to develop all these things must be paid for - if millions of developers just use the perfectly polished open source tools - and a high percentage doesn’t even help in development with big reports, not to think of patches, what are the developers going to live from while doing the polishing/ simplifying?

pauldix 8y ago
Author here. We have more work to do to make the happy path with the set of Influx tools (the TICK stack) more turn key and easier to run. The entire feature set is available in the open source versions. The thing we keep commercial is HA and scale out clustering (either we operate for you on AWS or you buy on-premise). Our work on 2.0 of the platform should make the happy path much easier, but software is a process of continuous improvement. So I'd expect 3.0 to be even better and so on.
- hennsen 8y ago
  Thanks! Sounds great and I‘ll look into it for the next project/use case!

yeukhon 8y ago

One major reason OpenStack is a messy project today is the amount of companies involved in thr foundation early on. It was a big problem when RedHat vs HP vs Rackspace vs XYZ.

EngineerBetter 8y ago

I was at KubeCon, and had a similar experience. Lots of engineers excites about all the technical possibilities, and less discussion of developer productivity. It reminds me a little of in the 00's everyone thought their company should write its own CMS. I think we're in danger of every writing their own PaaS. This is why things like Deis and Cloud Foundry exist. Most app developers should not have to understand the full depth and breadth of Kubernetes.

Jyaif 8y ago
Amen. Use PaaS as much as possible and fallback to running VMs as a last resort.

reacharavindh 8y ago

I'm a sysadmin and I attended KubeCon recently. I came back with a similar thought flow in mind. This one anecdote nailed the problem in my opinion - "Kubernetes makes simple things hard, and hard things possible." So, if you dont have things that you think are impossible, just don't pay the complexity tax. Real-life example : https://www.reddit.com/r/devops/comments/8byasq/is_kubernetes_worth_it/ https://www.reddit.com/r/devops/comments/8byasq/is_kubernete... Paraphrasing for discussion: Poster: A rails project with deployed to 6 servers running in production currently. Poster: During the asset compilation process, the servers often freeze. Poster: I need to manually remove servers from the load balancer and deploy one by one. Poster:I looked a lot into kubernetes and production containerization lately, and as far as I read it, it should solve the deployment and uptime issues. I imagine it'd be a lot easier to just switch containers instead of deploying with capistrano. I also really like the self-healing capabilities a lot. So, he/she hopes that Kubernetes will magically solve his problem(asset compilation freezes the server). I suppose in his/her mind, Kubernetes is the snake oil. Things that he/she failed to put thought into (and rather got revved up about Kubernetes): * Could I setup CI with a script that will perform the asset compilation once on one server and just rsync the final result to the prod servers? * Could I spend a couple of hours understanding the asset compilation process and find out why it freezes the server? * Could I learn more about load balancing, rolling deploys? I think this is the real problem in the tech field. People are running after shiny tools and hope to through tools at their problems all the while ignoring the basics. In this particular case, I think if they had a grey beard sysadmin who was grumpy to the devs, and enforced a strict release process, everyone would've been happier.

gant 8y ago

I think people misunderstand why Kubernetes exists. It is the reverse OpenStack. Kubernetes has the potential to be the one unified API of the cloud. A middleware for proprietary cloud APIs. A few resources, like load balancers, are already at a point where you barely have to care about the underlying cloud provider. With operators and aggregated API servers (especially if they'll be offered as a service) provisioning resources could follow one well-known standard. Calling it now, within the next 2 years cloud database providers like Compose will offer a way to CRUD resources via CRD/apimachinery-compatible services. Few more years and we'll have a generic yaml spec for these resources that work out of the box on multiple providers (probably with a bunch of annotations that are vendor specific).

erikb 8y ago

Exactly what I'm always saying. It's also nearly impossible in an Enterprise IT environment to get Kubernetes working on your laptop. Minikube and Docker Edge both seem to fail way too often. As a developer one wants to spin up a system to work on, then work on it, then push results to some repo. And this loop simply isn't possible (yet?). Also what the author didn't mention is that even the vanilla k8s stuff is already super complex. Let's assume you manage to setup a cluster somewhere and it continues to work for more than 2 days (rarely seen in real world work environments). Then you are faced with deploying your hello world app to work on. Just for a simple single-server nginx deployment with no files and no config, you already need to understand mutliple objects: deployments, replicasets, pods, containers, nodes, hosts, services, nodePorts, port-forwards, maybe even ingresses and controllers. That means until you feel well, even in a perfect environment, you need several days or weeks. And the documentation is not really helping you there. Yes, it's better than most enterprise grade documentation out there, but still it assumes a lot of stuff upfront. For instance, why should a developer even know what an ingress is and that it might be something he needs? Combine that huge learning overhead with nearly impossible network debugging, beta level stability, and the low possibility to "just use it", then you have a system that most developers will never touch. Docker itself might survive though, and one of the good things of this CNCF world is that other alternatives to docker also get a chance to improve on the existing system.

tuananh 8y ago
it's been extremely easy with Docker for desktop. Just 1 click a way from "enable kubernetes"... in Docker preferences.
- erikb 8y ago
  That worked for me for 2 days. Then a manager was breathing down my neck to debug his cluster. So due to stress I overwrote the kubeconfig file with my managers kubeconfig. Now I can't find a way to get the docker kubeconfig back, resetting the cluster, reinstalling kubernetes etc does nothing, and now that I have uninstalled and reinstalled docker, kubernetes doesn't even come up anymore. I'm also not allowed to share debug information with the Docker company. No real help from Docker developers without debug info. Because they can't tell me how docker generates a kubeconfig or where it writes its logs without seeing my diagnostics output... Not the vanilla perfect world use case is what makes your software but how well your software manages edge cases.

cmorgan8506 8y ago

As an app developer, I can vouch for finding k8 to feel frustratingly complex. My current client work has recently shifted to using k8. I took the time to get minikube working locally to get a better understanding. It definitely helped, but I find the layers of abstraction hard to grok after not working with it for a while. I can see the value the tool offers, but I get the feeling it's supposed to be reserved for higher degrees of scale than the average 2-8 node app. Black box is how I feel about it sometimes. Hopefully I'll get more one on one time with it in the future. It seems like a really cool technology.

austinshea 8y ago

These articles seem common, recently. I don’t really understand what this sentiment is about. It’s a really useful orchestrator, in some cases. In other circumstances, it’s unnecessary complexity.

mverwijs 8y ago

"The number of things you need to create and worry about are a barrier to starting a project and iterating on it quickly." As an SRE / Ops person it is my take that Kubernetes addresses many of the complexity your startup should be worrying about, but doesn't because "it is not a feature". Yes, it is massively complex. That is because Ops is massively complex.

deleted 8y ago
[deleted]

mancerayder 8y ago

This is my personal experience on the matter as a DevOps consultant who periodically interviews in the traditional manner (as opposed to getting gigs from people that already worked with me). Despite having 15+ years of *nix experience, including internals, having a track record of building large scalable infra and knowing a few different programming languages, what happened to me was this: I was getting filtered out because I didn't have Docker and Kubernetes and even (at one point) Cloudformation and/or Terraform. No problem - I learned those things (minus Kubernetes, so far) quite quickly. Much more quickly than the grueling trial-by-fire years of Unix administration. I like to know how things work, not just how to use them. So if you wonder and worry about the state of enterprise IT some days, look no further than hiring managers themselves, who will pick a 25 year-old who writes YAML for some abstraction-of-an-abstraction system that does infrastructure under the hood, infrastructure that people kind of don't really try very hard to understand. After all, it's disposable thanks to infrastructure-as-code, right? How do I know this? Well, I've seen shop after shop that's suffered a spaghetti infrastructure, using all the latest and greatest, from AWS and Kubernetes and Docker and other abstraction layers above AWS. And what happens is that it gets so complex that no one knows what's really going on, and at the very least two common symptoms arise: people are terrified during releases and they take hours, with many people on a call together very late at night; they spend a fortune on extra instances (in the case of AWS) because they haven't properly worked out environment separations (they had trouble keeping them the same, or one of many other problems). A talented dev manager I used to work with used to complain that they had trouble hiring people who knew Javascript well, but they had expert after expert of some fancy JS framework try to interview, unable to answer the fundamentals-types of questions. I think it's similar with enterprise infrastructure. I don't know what to say. I hope things go full swing and people who know how things work under the hood can charge consulting dollars for fixing the fuckups. It's not enough to know YAML, you also need to have wisdom in maintaining complex infrastructure, understand the delicate balance between change and stability, and be able to troubleshoot when it goes wrong WITHOUT just 'rinsing and repeating' where you learn absolutely no lessons at all. [edit:] One theory for all of this. Some of the big shops (Google, FB, Netflix, etc.) did it right, and now everyone is trying to copy the style of infrastructure management, except doesn't have the talent or wisdom to do it well.

jgr447 8y ago

Not having used it in any meaningful way, after a lot of reading I am sometimes still unclear with the value proposition. The value imho could be in being able to package distributed applications and deploy across cloud providers or on prem, seamlessly. I don't think this is true though short of doing a lot of effort to abstract access to a gcp/aws/azure managed service (say, a db), which is probably a bad idea. If you take that away, then a lot of the replication, autoscaling, load balancing, failover etc. can be implemented using cloud providers without having to manage the complexity of k8s. Hope to be proven wrong here.