12 ms·

Why Kubernetes is winning the container war

269 points by bdburns 10y ago

pawadu 10y ago
From the subtitle: It's all about knowing how to build an open source community -- plus experience running applications in Linux containers, which Google invented
- nullspace 10y ago
  It kinda did, right? Maybe, "invented" is to bold of a term, maybe there were others before. But, I've heard folk tales that Google has been running stuff in containers since the early to mid 2000s.
  - virtualnm 10y ago
    I was where I worked, with technology no one's heard of now, I doubt Google was then.
  - lern_too_spel 10y ago
    Google contributed cgroups to Linux, so saying that it invented Linux containers is a fair statement. Other operating systems had containers before Linux.
    - pawadu 10y ago
      would you consider "chroot" a form of container?
    - SEJeff 10y ago
      Not exactly. Google contributed the namespace support to Linux and IBM contributed much of the initial control group bits to Linux. Together, they form the modern "container" that we have all came to know and love.
- deleted 10y ago
  [deleted]
runako 10y ago
On Safari OS X, the article is blocked by a full-page ad that I can't dismiss.
- the_duke 10y ago
  Don't see any ads on Chrome with Adblock Plus. It shows 32 blocked requests though (ouch).
  - cptskippy 10y ago
    That seems light for most pages...
    - the_duke 10y ago
      About average, I'd say. Quick check: * cnn.com: 30-40 * bbc.com: 10-15 * washingtonpost.com: 10-25 * medium.com: 1-2 * teccrunch.com: 25-40
batmansmk 10y ago
It pretends to compare Kubernetes, Apache Mesos and Docker Swarm. This article says Kubernetes has a lot of stars on github (doesn't compare it to Docker or Mesos, only says Kubernetes has a lot), same for Slack/Stack Overflow and number of CVs mentioning the tech ... I will pass Infoworld opinion from now on.
- aCandidMind 10y ago
  I completely agree that this is an article only intended to praise Kubernetes. Stating Google invented Linux containers alone is wrong as well.
tzaman 10y ago
For a devOps fan like me, k8s has been a godsend, and what I like in particular is their 3 month release schedule. There are still some hiccups like no good documentation (or a tutorial really) on setting up shared writeable storage and how to handle databases, or more importantly replication. The k8s team is very responsive and I'm sure these will be ironed out in the near future so we can all adore each other's cattle :)
- yeahbutbut 10y ago
  How did you handle shared writable storage and databases?
  - spudfkc 10y ago
    I've found this to be a pain when setting up a container-based environment. The easiest approach is to just to avoid it as much as possible - hopefully your cloud provider has some managed services (i.e. AWS RDS) that will handle most things for you. Otherwise you need to separate your available container hosts into clusters: Elasticsearch cluster, Cassandra cluster, etc. and treat those differently from your machines you deploy your other apps to, which to be fair, they are different and need to be treated differently.
  - ex3ndr 10y ago
    We (actor.im) also moved from google cloud to our servers + k8s. Shared persistent storage is a huge pain. We eventually stopped to try to do this, will try again when PetSets will be in Beta and will be able to update it's images. We tried: * gluterfs - cluster can be setup in seconds, really. Just launch daemon sets and manually (but you can automate this) create a cluster, but we hit to that fact that CoreOS can't mount glusterfs shares at all. We tried to mount NFS and then hit next problem. * NFS from k8s are not working at all, mostly this is because kubelet (k8s agent) need to be run directly on a machine and not via rkt/docker. Instead of updating all our nodes we mounted NFS share directly to our nodes. * PostgreSQL we haven't tried yet, but if occasional pod kill will take place and then resyncing database can became huge issue. We ended up in running pods that is dedicated to specific node and doing manual master-slave configuration. We are not tried other solutions yet, but they also questionable in k8s cluster. * RabbitMQ - biggest nightmare of all of them. It needs to have good DNS names for each node and here we have huge problems on k8s side: we don't have static host names at all. Documentation said that it can, but it doesn't. You can open kube-dns code it doesn't have any code at all. For pods we have only domain name that ip-like: "10-0-0-10". We ended up with not clustering rabbitmq at all. This is not very important dataset for us and can be easily lost. * Consul - while working around problems with RabbitMQ in k8s and fighting DNS we found that Consul DNS api works much better than built-in kube-dns. So we installed it and our cluster just goes down when we kill some Consul pods as they changed it's host names and ip. And there are no straightforward way to fix IP or hostnames (they are not working at all, only ip-like that can easily changed on pod deletion). So best way is to have some fast(!) external storage and mount it via network to your pods, this is much much slower than direct access to Node's SSD but it give you flexibility.
_asummers 10y ago
One thing I've found extremely difficult to handle is the Zookeeper cluster model of containers. Where when a thing dies, a thing has to come back and be able to referred to as "zookeeper-1" forever. The way to do this currently is to use a service in front of a replication controller with one pod. This feels wrong all over. Supposedly they have a thing called Pet Sets [1] coming to solve this, but it's been in the works for an eternity. Also we've started to outgrow the load balancing simplicity that the k8 load balancer gives you, and I have not seen a nice migration path to something like HAProxy in Kubernetes. All that said, we like kubernetes a lot. [1] To distinguish from cattle. If you have a red blue and green goldfish, and your red goldfish dies, you can replace with another red fish and not really notice, but if it's purple, the others won't play with it.
- jat850 10y ago
  Re: load balancing. Out of curiosity, why not just use haproxy? Or nginx, or your load balancer of choice. Run haproxy in host networking mode, as a daemonset, and use it as your ingress point?
  - _asummers 10y ago
    So it would basically be daemon set sitting on top of all the other services, with their own load balancing turned off? That should work. I like that
    - olafmol 10y ago
      That's how we do it with Vamp. Interested in your thoughts: http://vamp.io/documentation/installation/kubernetes/ http://vamp.io/documentation/installation/kubernetes/
    - jat850 10y ago
      Email's in profile. I'm happy to discuss specifics offline, as I have some experience with this. Can show some example configs of how we've employed it.
    - ex3ndr 10y ago
0xCMP 10y ago
The setup to get k8s running isn't great, but once it's running and you understand it's config files it makes things so much easier. We're getting ready to deploy k8s at work soon and begin moving more there as we can. From what I understand, and is completely not in the article, Mesos is designed for scale while most start-ups (and even established companies) can't afford or justify. K8s is simpler but still robust. Better than just fleet or compose and clearly still better than swarm (based on posts read here on hn).
- steilpass 10y ago
  Disclaimer: I work for Red Hat. > The setup to get k8s running isn't great, +1 But they are greatly improving. For a local test environment have a look at OpenShifts 'oc cluster up' https://github.com/openshift/origin/blob/master/docs/cluster_up_down.md https://github.com/openshift/origin/blob/master/docs/cluster...
  - yoshuaw 10y ago
    Alternative local cluster for OSX is kube-solo https://github.com/TheNewNormal/kube-solo-osx https://github.com/TheNewNormal/kube-solo-osx - it's pretty good, and runs as a native Mac app
  - brazzledazzle 10y ago
    Someone elsewhere in this thread complained about the NFS experience with k8s. I know OpenShift contributed some or all of that code so does it improve the NFS experience as a layer on top of k8s?
    - joefern1 10y ago
      Disclaimer: I work at Red Hat on OpenShift. All of the code Red Hat contributed around Kubernetes storage plugins went into Kubernetes upstream. If you have any questions or problems, feel free to raise an issue or reach out to us in the community. Red Hat has a large number of contributors on the Kubernetes project and we are big fans of the community and the technology! Red Hat OpenShift provides a an enterprise-ready Kubernetes distribution that also builds on top of Docker, RHEL/Fedora/CentOS and includes integrated networking (OVS), routing (HAProxy), logging (ELK stack), metrics, image registry, automated image builds & deployments, integrated CI/CD (Jenkins), and self-service UX (Web, CLI, IDE). You can check out the free Origin community distro here - https://github.com/openshift/origin https://github.com/openshift/origin or sign up for our free hosted developer preview here: https://www.openshift.com/devpreview/ https://www.openshift.com/devpreview/. We also offer commercially supported solutions - https://www.openshift.com/container-platform/ https://www.openshift.com/container-platform/.
iagooar 10y ago
What is the best resource to learn Kubernetes like a boss? I like ebooks, but will take anything, as long as it's up-to-date and easy to follow without being a long-term sysadmin.
- jat850 10y ago
  You might not like this answer, but... play with it. In our environment, we're compiling and deploying Kubernetes by hand (I mean, not entirely by hand, but you get the idea). It was daunting at first but is getting easier and easier all the time, and we have a better handle on how things work as a result. What's going on under the hood, what the mass of command line parameters actually mean, and so on. The documentation is lacking. Badly. And doesn't seem to keep pace with the features that are added in every point release, or in the alpha/beta versions. A lot of what we've learned has been by experimentation, poking at things, following the k8s user discussions, watching Github, reading code.
  - mdaniel 10y ago
    Can you elaborate on the information you wish was documented but isn't? Also, have you seen the box at the bottom of the docs that says "I wish this page ..."? It goes right into their issue tracker, which increases the likelihood of something getting fixed.
    - jat850 10y ago
      To be fair, I have not seen or used that link, but I will take note of it for the future - thank you, that's very helpful. I should keep notes on specifics, so I apologize that I can't highlight a particular thing that I've been frustrated by in a given moment. I'll certainly say that the documentation is improving. As a general item, I think my biggest struggle has been hitting a wall with the documentation - there are some things that are left almost as exercises to the reader, especially getting into more advanced topics (how does one do rolling upgrades or canary deployments in more esoteric situations, how might one do load balancing properly in a master-slave style configuration versus something more round-robin oriented, etc.) And I don't like to levy a complaint without acknowledging that anything opensource is ripe for improvement by contribution, but my professional situation prevents me from doing so. Trust that I would love nothing more than to help and not just make an empty observation or moan about it.
paimpozhil 10y ago
I wish kubernetes has more examples for example their vsphere volume driver has almost 0 documentation/tutorials on how to set that up. http://kubernetes.io/docs/user-guide/volumes/#vsphere-vmdk-example-configuration http://kubernetes.io/docs/user-guide/volumes/#vsphere-vmdk-e... I believe is inadequate
- smarterclayton 10y ago
  Agree - better doc is a focus for us. We've added providers so fast the doc has not caught up.
virtualnm 10y ago
I can bring up an app on Linux or Windows from bare metal in minutes by hand. But the way it's supposed to be done now is something like this, right: 1) Use one of Chef/Puppet/Salt/Ansible to orchestrate 2) One if those in item 1 will use Vagrant which 3) Uses Docker or Kubernetes to 4) Set up a container which will 5) finally run the app Really?
- colemannerd 10y ago
  Your assessment is correct. It takes much longer to just run the app for the first time. The savings come in when you want to redeploy the app in multiple environments, test it, and develop new features on it. That's when all of these steps make sense - although, hopefully you'll be able to remove steps 1 & 2 when everything is a docker / Kubernetes workload.
- oblio 10y ago
  Most of those technologies overlap. Except for some disaster development scenarios, no one is actually using all those technologies in the same environment (i.e. they're not using Chef and Vagrant and Docker in production, at the same time) Chef/Puppet/Salt/Ansible these days are just used to provision 1 VM/container (if even that, many people have just reverted to shell scripts) and that container is then managed using a container management tool (Swarm/Kubernetes).
  - jimbokun 10y ago
    "(if even that, many people have just reverted to shell scripts)" Isn't Ansible basically just a way of executing shell scripts across multiple hosts in a sensible manner? At least that's how it's been explained to me.
    - brianwawok 10y ago
      Yes. Most of the value of Ansible is the library 100 of tasks like copy a file and replace variables, or restart a service if the config changed. It can do a lot of work in a few lines... But can also be a bit too much if you are already in a single container.
spudfkc 10y ago
I haven't dived into Kubernetes yet, but I set up Rancher for our new application and it has been nothing short of amazing so far. I can't express how happy we've been with it. I previously tried the Mesos/Marathon route (with Mesosphere and then again with Mantl) and that was nothing but a huge waste of time due to all the maintenance that was necessary for all the required servers. With Rancher, it's just spin up a container for each host with a single command and you're done.
- webo 10y ago
  I've been looking at Rancher for a few weeks, haven't put an application yet. How do you perform rolling updates with Rancher+Kubernetes? With Kubernetes itself, currently I just do `kubectl rolling-update <rc> --image <new-image>`
leetrout 10y ago
I know it's young still, but I think Nomad is going to get a share of this market with little effort. I played with Mesos & k8s and I picked Nomad instead. Now I'm not managing a huge fleet of servers I want to abstract away as much as I wanted a simple, consistent framework for allocating resources to containerized tasks across a small group of nodes. And I don't think that use case is anything to sneeze at and for a new user there just isn't anything out there as easy as nomad IMO. https://www.nomadproject.io/ https://www.nomadproject.io/
- lmickh 10y ago
  Nomad is definitely shaping up to be a nice alternative if k8s and mesos are overkill for your use case and you don't want to be vendor locked to a simplified option like ECS.
- smnscu 10y ago
  For a simple solution, also see Rancher. I've used them about 2 years ago and even then it was a very useful and stable tool. http://rancher.com/ http://rancher.com/
  - loopbit 10y ago
    I also use Rancher and quite happy with it. The cool thing is that you can set up and environment using cattle (their own orchestration system), k8s, mesos or swarm. Up to you and your preferences.
- friendly_chap 10y ago
  I wanted to use nomad, but at the time their tutorials (not sure about their current state) were not really good, so I ended up rolling my own solution which serves me well since then (https://github.com/crufter/puller https://github.com/crufter/puller).
  - bogomipz 10y ago
    I agree about lack of tutorial or documentation. Hashicorp documentation always feels like an internal wiki company wiki or a reference guide for someone who is already quite familiar with the APIs. I just checked the Nomads docs and it looks pretty much like it did when I looked at it 4 or 5 months ago.
DubiousPusher 10y ago
Seems like "Why Kubernetes is winning the orchestration war" is a more appropriate title for this?
mr_luc 10y ago
In my experience, I haven't been coming to k8s because I particularly like the developer experience (despite their efforts to focus heavily on it), but because it cleanly supported some things that I need. For instance, with k8s, out of the box every running container in a clustered system is discoverable and has its own IP. If you're writing distributed applications, and you're using containers principally as a tool to make your life easier (and not as part of an internal paas or for handling data pipelines or some other use case), having that sort of discovery available out of the box is great.
applepple 10y ago
It would be nice if open source projects (especially popular databases) came with k8s definition files so that you wouldn't have to write the yaml yourself.
- jondubois 10y ago
  I did that for my open source project SocketCluster. See https://github.com/SocketCluster/socketcluster/tree/master/kubernetes https://github.com/SocketCluster/socketcluster/tree/master/k... Also, if you look at the main Kubernetes repo on GitHub, you can find the yaml files for various popular open source projects in there as well. Some of them are out of date though - Things have been changing so fast. I think a problem that still comes up is that you still need yaml files to account for the interactions between different services too. Each service doesn't live in a vacuum. We're currently working on building an complete auto-scalable stack on top of Kubernetes and a deployment service to go with it. See https://baasil.io https://baasil.io Everything we do is open source except the actual website.
  - hackitself 10y ago
    I use SC for my startup and it works great! Thanks. I'll have a look at the Baasil service, it sounds interesting.
contingencies 10y ago
IMHO nobody is winning anything that matters right now because the current transition is a transition to an additional level of abstraction which is definitely not properly met by any of the tools available. What we now need is tools that allow architectural fences around components, and reliability guarantees around subsystems ... versus not only technical but also business-level risks (including state-level actors)... often across borders, including for example exchange rate risk. This is based on business-level risk models, not some engineer feels X or Y type reasoning which is (often very well, but broad-picture uselessly) based on technical know-how. I prototyped such a system pretty successfully, you can read the design @ http://stani.sh/walter/cims/ http://stani.sh/walter/cims/ .. it's incomplete (critically hard to explain investment utility for non-tech business types) but at least infrastructure agnostic and practically informed. NB. To be non-humble, by way of demonstration I am a guy who has been called to conceive/design/set up/manage from-scratch datacenters for extremely visible businesses with highly motivated attackers with USD$Ms at stake for penetration. Systems I personally designed run millions of USD/month and have many times that in investment. And it's obvious that both Google ("compete with amazon on cloud business... high-level desperate for non-advertising income!") and Docker ("VC! growth! growth! growth!") have their own agendas here... we should trust no-one. It's early days. Bring on the ideas, build the future. Hack, hack, hack. We the little people own the future. It's ideas.
sebringj 10y ago
Just as an outside observer developing on a platform, I see my fellow devOps team members working in Kubernetes and it's been a shit storm for the most part where containers disappear randomly, stuff breaks and they are on call on the weekends not having fun. I have my own clients on other projects using AWS where I just upload and click buttons like a dumb monkey and I end up looking more competent even though I'm completely not. I've consequently not been motivated to dive into these DIY deployments just yet.
stcredzero 10y ago
It's all about knowing how to build an open source community This. Engineering excellence is secondary. You can get away with complete craptitude in your tech if you can build community. (I won't name examples.) Of course, it's better if you also have technical excellence. On the other hand, you can have technical excellence, but it will come to naught if you have community destroying anti-patterns.
amr_abdelrazik 10y ago
Disclaimer: I work for Mesosphere (Champions of Apache Mesos and DC/OS) We have total respect for K8s, but I don't think you can claim winning just based on the community and stars. OpenStack has a huge larger community of developers and advocates, but it is still haven't reached it's potential despite many years and incredible effort, seminars and summits Also most of these next gen infr projects now (DC/OS which is powered by Mesos, K8s, docker swarm ) are converging feature wise, but they also have their strengths and weaknesses, some of these are temporary, some of these are structurally by design Mesos (and DC/OS) were not just designed for scale, but also for extensibility and different workloads, which is why you can run Cassandra, Kafka, Spark, ..etc In production. None of these workloads run as traditional containers, with DC/OS they have their own operational logic such as honoring placement, simple installation, and upgrade, which is a core design function of the two-level scheduler People usually complain that standing up Mesos is hard, which is why we we built DC/OS and open sourced it to give you the power of mesos and its awesome capabilities in managing containers and data workloads without going through all the effort to stitch everything yourself. Check it out at DCOS.io, I am sure you guys will be blown away.
- manojlds 10y ago
  I have been trying to understand advantages of DC/OS over Mesos. Why should I even consider it? All I can understand is that it has a better(?) UI. I understand that even this and the DCOS CLI and packages work with Mesos too? So what's in there in DC/OS?
  - amr_abdelrazik 10y ago
    Tl;DR: DC/OS provides all the components you need on top of Mesos to run in production, all pre-packaged and tested together with easy to install way Mesos provide the core resource management and scheduling, think of it as the Kernel of an (DC) operating system. On top of Mesos you usually need: 1- Container/Workload Scheduling capability which is Marathon, now built in and integrated in DC/OS so you are just up and running in minutes 2- Management (Beautiful DC/OS UI and CLI) to manage all cluster and workload operations. It also includes Installer (of all components including Zookeeper, networking, ..etc) 3- App Store (Universe) that allows you to easily find and deploy latest packages such as Kafka, Cassandra and Spark with a single command, in regular Mesos you needed to find the Framework that worked with a specific version of Mesos, troubleshooting, ...etc. Now it's an app store like experience 4- Core infrastructure components such as very cool service discovery and loadbalancing (Minuteman), Storage drivers, CNI support If you want to run docker containers, DC/OS also gives you the ability to use the docker daemon or the new Universal containerize (technology preview) which we just released. Because mesos is very modular and very powerful, it was built to have modular implementation of all system components, but our team and our founders ran some of the largest implementations of Mesos (Twitter, Airbnb, ..etc) and they complied all thes best practices and the bits and pieces in DC/OS. Hope that explains,
avitzurel 10y ago
I've used both mesosphere and Kube now (in production) and I feel I can safely comment on this. Kube is winning for the same reason React/Redux (and now Mobx) is winning and why Rails was winning at the time. Community. The community for Kube is awesome and the work people are doing in the field is being noticed all over the place. I've seen people (myself included) that moved production clusters from mesos to Kube just because the activity of the development and how secure they feel with the community and the project going forward. React and Rails (at the time) had the same sort of pull towards the community and why a lot of people on-boarded. Golang is most likely a factor here too. I feel most people find Golang friendlier than Scala/Java. That's why Kube has many more contributors, the hurdle for contributing is easier to jump
- dominotw 10y ago
  >I've seen people (myself included) that moved production clusters from mesos to Kube Sorry I am new to this. I saw a presentation where someone was running Kubernetes on DC/OS . Why are people doing that if they can just run pure kubernetes.
  - avitzurel 10y ago
    I don't know why people would. I don't :)
  - amr_abdelrazik 10y ago
    Some people like (the beta) K8s implementation on DC/OS because they like K8s abstraction for container orchestration , but at the same time they also like DC/OS because of distributed stateful (data) services support, While you can run everything in a container on K8s, the advanced data services require some special handling (see the comments about postgress and glusterfs from another commenter), and some people like to get the best of both worlds for example, if you restart a Kafka node on any container orchestration platform, container is destroyed and respawn on another node which is great, but you lose all your data and the broker has to rebuild all that queue again, while Kafka on DC/OS wait and maintains the volume just in case you are having temporary maintenance restart before relocating the workload. another example is Cassandra if you want to add more than 1 node, the traffic to resync the data across 2 nodes can actually degrade the performance on your cluster or even taking it down. I can go on and on, all of these issues and requirements come from running these mission critical data apps in production, with each application having its own logic on how it handles day 2 operations, but you don't know what you don't know unless you tried running them in production. Some people talk about the custom scheduler feature or Petsets in K8s, those features are cool and hats off to the K8s community, but they are still early in development while the framework architecture is a core mesos functionality since day 1. And finally, everyone is free to run whatever they want/like, each company and person might have different preferences on how they want to run workloads and how much effort they want to go through that, I don't think it's productive to go to another VI vs emacs here ;-)
kbredemeier 10y ago
Innovation is what makes this industry so exciting to be a part of. I joined the tech world, via Holberton School[1] just a mere nine months ago, and already so much has changed. It makes it challenging to keep up, but that's the fun part. The future of technology is held in the hands of open source projects. [1] https://www.holbertonschool.com https://www.holbertonschool.com
avitzurel 10y ago
One thing that I think Kube (and dc/os) are missing is what Chef is working on right now. Application definition should live within the app and consumed by the scheduler. Chef's product is called Habitat https://www.habitat.sh/ https://www.habitat.sh/ and it has some VERY interesting concepts that if Kube will implement it will be much more interesting (to a lot of people). Right now, the deployment and configuration of an application is supposed to be separated but I feel they need to be just a bit more coupled. An engineer that develops the application will define a few things like "domain" and how you connect to the application and this will be consumed by the scheduler. Right now, dc/os and mesos are really fine grained around the DevOps people and I feel that the first that will crack the "batteries included" approach will win the fight by a knock-out. Imagine something like Heroku on your own infrastructure, if an engineer can launch and deploy a micro-service with the same ease they deploy and access a Heroku application. That will be awesome.
- kozikow 10y ago
  > Right now, the deployment and configuration of an application is supposed to be separated but I feel they need to be just a bit more coupled. You can dynamically configure kubernetes from within the cluster. You can either use client-go library or talk to the api server directly with rest. For other languages than Golang you may need to set up proxy server, but it's a single kubectl command. So all the basic functionality is there, but IMO configuration should be separate if it is possible - having it in source control is nice. I only use dynamic api for scheduling kubernetes jobs.
  - avitzurel 10y ago
    What I imagine is something like the Procfile for Heroku. The app decides what it runs and how, you can set configuration like domain, whether the application is public/private etc... Once you push the application to the scheduler it will handle the rest.
    - kozikow 10y ago
      Ah. Quite a lot of things you mention are possible by dynamically generating yaml config. I wrote a blog post about using go kubernetes library to generate yaml : https://kozikow.com/2016/09/02/using-go-to-autogenerate-kubernetes-configs/ https://kozikow.com/2016/09/02/using-go-to-autogenerate-kube.... It's quite nice, even for static config IMO, due to type safety. You can easily write golang binary that would ask user series of questions "what database would you like", "do you want public access", etc. and get yaml as output.
shadykiller 10y ago
The author is just gaga over google. He presents no comparison, no benchmarks and no mention of alternatives except docker swarm.
technologia 10y ago
I use Mesos and K8s heavily as well as contribute back to the projects, and while I do agree this is leaning towards being fan-fare there is a bit of truth to this. Community is a big deal, people tend to underestimate this; Putting aside newer companies, when a larger enterprise ventures out to open source they do take community as a major factor as you have to consider the tool you build will have to last 5 years, maybe 10, maybe more. To delve further into the Docker side of things, I personally wish that the company would focus on its core business instead of stretching itself with extra things. I get the need to improve the UX, which they do very well considering how far we have come from LXC. I feel Mesosphere starting to go down the feature creep route as well, but I wish them all the best as I loved Mesos since the beginning all those years ago.
graffitici 10y ago
Sad to see Mesos losing steam. My understanding was that Mesos subsumes the functionality of Kubernetes thanks to its Aurora scheduler. But it has much more customized schedulers, for different purposes, that might make it more efficient to run complicated pieces of software. For instance, it certainly is possible to run a Cassandra cluster by having each instance run in its own Docker container. My understanding is that it would be much more efficient to run this cluster with a dedicated Cassandra cluster instead. Is this right? Or are the performance benefits of running a dedicated Cassandra scheduler on Mesos negligible compared to running them in containers?
huntc 10y ago
Looking through all of these comments we should be glad, as a community, that there are a number players in this burgeoning orchestration space. Nobody has won and nobody should win. They each have their strengths and weaknesses and there is no one size that fits all.
nzoschke 10y ago
These articles never mention the elephant in the room, AWS. How many containers are running on Elastic Beanstalk and ECS? I'd wager magnitudes more are running containers by reading the docs and clicking around than those that mastering the cutting edge challenges getting running on Mesos, Kube or Swarm. Another blind spot is Heroku. Every day new businesses spin up containers with a 'git push Heroku master' and don't even know about it. All these providers, platforms and tools have their place. I simply don't think the "winning the war" talk is accurate or constructive. Disclaimer: I worked on the container systems at Heroku and now manage 100s of production ECS clusters at Convox.
- colemickens 10y ago
  What "cutting edge challenges" are there to get running on Mesos/Kube/Swarm? Are you just referring to getting the cluster booted? There are just so many options for that these days. Just curious, I was skimming through ECS docs last night and it seems like some of the basic building blocks look very similar to their equivalent constructs in Kube/Nomad.
  - nzoschke 10y ago
    Automating VMs with Elastic Beanstalk and deploying to a platform like Heroku are effectively solved problems. Container orchestration brings in new challenges like operating a consensus database, managing data volumes and load balancing. And this is on top of VM management. Yes, ECS, Kube, Nomad are very similar. All relatively hard to use right now compared to their simpler ancestors. Here is more in depth info about the challenges of ECS: https://convox.com/blog/ecs-challenges/ https://convox.com/blog/ecs-challenges/
aCandidMind 10y ago
Why is no one mentioning Docker's response with version 1.12, the new built-in orchestration called swarm mode (different than just swarm): http://blog.nigelpoulton.com/docker-launches-kubernetes-killer/ http://blog.nigelpoulton.com/docker-launches-kubernetes-kill... Granted, the title is fanboyish, but it really seems to be a significant response to Kubernetes.

269 points by bdburns 10y ago

From the subtitle: It's all about knowing how to build an open source community -- plus experience running applications in Linux containers, which Google invented

nullspace 10y ago
It kinda did, right? Maybe, "invented" is to bold of a term, maybe there were others before. But, I've heard folk tales that Google has been running stuff in containers since the early to mid 2000s.
- virtualnm 10y ago
  I was where I worked, with technology no one's heard of now, I doubt Google was then.
- lern_too_spel 10y ago
  Google contributed cgroups to Linux, so saying that it invented Linux containers is a fair statement. Other operating systems had containers before Linux.
  - pawadu 10y ago
    would you consider "chroot" a form of container?
  - SEJeff 10y ago
    Not exactly. Google contributed the namespace support to Linux and IBM contributed much of the initial control group bits to Linux. Together, they form the modern "container" that we have all came to know and love.
deleted 10y ago
[deleted]

runako 10y ago

On Safari OS X, the article is blocked by a full-page ad that I can't dismiss.

the_duke 10y ago
Don't see any ads on Chrome with Adblock Plus. It shows 32 blocked requests though (ouch).
- cptskippy 10y ago
  That seems light for most pages...
  - the_duke 10y ago
    About average, I'd say. Quick check: * cnn.com: 30-40 * bbc.com: 10-15 * washingtonpost.com: 10-25 * medium.com: 1-2 * teccrunch.com: 25-40

batmansmk 10y ago

It pretends to compare Kubernetes, Apache Mesos and Docker Swarm. This article says Kubernetes has a lot of stars on github (doesn't compare it to Docker or Mesos, only says Kubernetes has a lot), same for Slack/Stack Overflow and number of CVs mentioning the tech ... I will pass Infoworld opinion from now on.

aCandidMind 10y ago
I completely agree that this is an article only intended to praise Kubernetes. Stating Google invented Linux containers alone is wrong as well.

tzaman 10y ago

For a devOps fan like me, k8s has been a godsend, and what I like in particular is their 3 month release schedule. There are still some hiccups like no good documentation (or a tutorial really) on setting up shared writeable storage and how to handle databases, or more importantly replication. The k8s team is very responsive and I'm sure these will be ironed out in the near future so we can all adore each other's cattle :)

yeahbutbut 10y ago
How did you handle shared writable storage and databases?
- spudfkc 10y ago
  I've found this to be a pain when setting up a container-based environment. The easiest approach is to just to avoid it as much as possible - hopefully your cloud provider has some managed services (i.e. AWS RDS) that will handle most things for you. Otherwise you need to separate your available container hosts into clusters: Elasticsearch cluster, Cassandra cluster, etc. and treat those differently from your machines you deploy your other apps to, which to be fair, they are different and need to be treated differently.
- ex3ndr 10y ago
  We (actor.im) also moved from google cloud to our servers + k8s. Shared persistent storage is a huge pain. We eventually stopped to try to do this, will try again when PetSets will be in Beta and will be able to update it's images. We tried: * gluterfs - cluster can be setup in seconds, really. Just launch daemon sets and manually (but you can automate this) create a cluster, but we hit to that fact that CoreOS can't mount glusterfs shares at all. We tried to mount NFS and then hit next problem. * NFS from k8s are not working at all, mostly this is because kubelet (k8s agent) need to be run directly on a machine and not via rkt/docker. Instead of updating all our nodes we mounted NFS share directly to our nodes. * PostgreSQL we haven't tried yet, but if occasional pod kill will take place and then resyncing database can became huge issue. We ended up in running pods that is dedicated to specific node and doing manual master-slave configuration. We are not tried other solutions yet, but they also questionable in k8s cluster. * RabbitMQ - biggest nightmare of all of them. It needs to have good DNS names for each node and here we have huge problems on k8s side: we don't have static host names at all. Documentation said that it can, but it doesn't. You can open kube-dns code it doesn't have any code at all. For pods we have only domain name that ip-like: "10-0-0-10". We ended up with not clustering rabbitmq at all. This is not very important dataset for us and can be easily lost. * Consul - while working around problems with RabbitMQ in k8s and fighting DNS we found that Consul DNS api works much better than built-in kube-dns. So we installed it and our cluster just goes down when we kill some Consul pods as they changed it's host names and ip. And there are no straightforward way to fix IP or hostnames (they are not working at all, only ip-like that can easily changed on pod deletion). So best way is to have some fast(!) external storage and mount it via network to your pods, this is much much slower than direct access to Node's SSD but it give you flexibility.

_asummers 10y ago

One thing I've found extremely difficult to handle is the Zookeeper cluster model of containers. Where when a thing dies, a thing has to come back and be able to referred to as "zookeeper-1" forever. The way to do this currently is to use a service in front of a replication controller with one pod. This feels wrong all over. Supposedly they have a thing called Pet Sets [1] coming to solve this, but it's been in the works for an eternity. Also we've started to outgrow the load balancing simplicity that the k8 load balancer gives you, and I have not seen a nice migration path to something like HAProxy in Kubernetes. All that said, we like kubernetes a lot. [1] To distinguish from cattle. If you have a red blue and green goldfish, and your red goldfish dies, you can replace with another red fish and not really notice, but if it's purple, the others won't play with it.

jat850 10y ago
Re: load balancing. Out of curiosity, why not just use haproxy? Or nginx, or your load balancer of choice. Run haproxy in host networking mode, as a daemonset, and use it as your ingress point?
- _asummers 10y ago
  So it would basically be daemon set sitting on top of all the other services, with their own load balancing turned off? That should work. I like that
  - olafmol 10y ago
    That's how we do it with Vamp. Interested in your thoughts: http://vamp.io/documentation/installation/kubernetes/ http://vamp.io/documentation/installation/kubernetes/
  - jat850 10y ago
    Email's in profile. I'm happy to discuss specifics offline, as I have some experience with this. Can show some example configs of how we've employed it.
  - ex3ndr 10y ago

0xCMP 10y ago

The setup to get k8s running isn't great, but once it's running and you understand it's config files it makes things so much easier. We're getting ready to deploy k8s at work soon and begin moving more there as we can. From what I understand, and is completely not in the article, Mesos is designed for scale while most start-ups (and even established companies) can't afford or justify. K8s is simpler but still robust. Better than just fleet or compose and clearly still better than swarm (based on posts read here on hn).

steilpass 10y ago
Disclaimer: I work for Red Hat. > The setup to get k8s running isn't great, +1 But they are greatly improving. For a local test environment have a look at OpenShifts 'oc cluster up' https://github.com/openshift/origin/blob/master/docs/cluster_up_down.md https://github.com/openshift/origin/blob/master/docs/cluster...
- yoshuaw 10y ago
  Alternative local cluster for OSX is kube-solo https://github.com/TheNewNormal/kube-solo-osx https://github.com/TheNewNormal/kube-solo-osx - it's pretty good, and runs as a native Mac app
- brazzledazzle 10y ago
  Someone elsewhere in this thread complained about the NFS experience with k8s. I know OpenShift contributed some or all of that code so does it improve the NFS experience as a layer on top of k8s?
  - joefern1 10y ago
    Disclaimer: I work at Red Hat on OpenShift. All of the code Red Hat contributed around Kubernetes storage plugins went into Kubernetes upstream. If you have any questions or problems, feel free to raise an issue or reach out to us in the community. Red Hat has a large number of contributors on the Kubernetes project and we are big fans of the community and the technology! Red Hat OpenShift provides a an enterprise-ready Kubernetes distribution that also builds on top of Docker, RHEL/Fedora/CentOS and includes integrated networking (OVS), routing (HAProxy), logging (ELK stack), metrics, image registry, automated image builds & deployments, integrated CI/CD (Jenkins), and self-service UX (Web, CLI, IDE). You can check out the free Origin community distro here - https://github.com/openshift/origin https://github.com/openshift/origin or sign up for our free hosted developer preview here: https://www.openshift.com/devpreview/ https://www.openshift.com/devpreview/. We also offer commercially supported solutions - https://www.openshift.com/container-platform/ https://www.openshift.com/container-platform/.

iagooar 10y ago

What is the best resource to learn Kubernetes like a boss? I like ebooks, but will take anything, as long as it's up-to-date and easy to follow without being a long-term sysadmin.

jat850 10y ago
You might not like this answer, but... play with it. In our environment, we're compiling and deploying Kubernetes by hand (I mean, not entirely by hand, but you get the idea). It was daunting at first but is getting easier and easier all the time, and we have a better handle on how things work as a result. What's going on under the hood, what the mass of command line parameters actually mean, and so on. The documentation is lacking. Badly. And doesn't seem to keep pace with the features that are added in every point release, or in the alpha/beta versions. A lot of what we've learned has been by experimentation, poking at things, following the k8s user discussions, watching Github, reading code.
- mdaniel 10y ago
  Can you elaborate on the information you wish was documented but isn't? Also, have you seen the box at the bottom of the docs that says "I wish this page ..."? It goes right into their issue tracker, which increases the likelihood of something getting fixed.
  - jat850 10y ago
    To be fair, I have not seen or used that link, but I will take note of it for the future - thank you, that's very helpful. I should keep notes on specifics, so I apologize that I can't highlight a particular thing that I've been frustrated by in a given moment. I'll certainly say that the documentation is improving. As a general item, I think my biggest struggle has been hitting a wall with the documentation - there are some things that are left almost as exercises to the reader, especially getting into more advanced topics (how does one do rolling upgrades or canary deployments in more esoteric situations, how might one do load balancing properly in a master-slave style configuration versus something more round-robin oriented, etc.) And I don't like to levy a complaint without acknowledging that anything opensource is ripe for improvement by contribution, but my professional situation prevents me from doing so. Trust that I would love nothing more than to help and not just make an empty observation or moan about it.

paimpozhil 10y ago

I wish kubernetes has more examples for example their vsphere volume driver has almost 0 documentation/tutorials on how to set that up. http://kubernetes.io/docs/user-guide/volumes/#vsphere-vmdk-example-configuration http://kubernetes.io/docs/user-guide/volumes/#vsphere-vmdk-e... I believe is inadequate

smarterclayton 10y ago
Agree - better doc is a focus for us. We've added providers so fast the doc has not caught up.

virtualnm 10y ago

I can bring up an app on Linux or Windows from bare metal in minutes by hand. But the way it's supposed to be done now is something like this, right: 1) Use one of Chef/Puppet/Salt/Ansible to orchestrate 2) One if those in item 1 will use Vagrant which 3) Uses Docker or Kubernetes to 4) Set up a container which will 5) finally run the app Really?

colemannerd 10y ago
Your assessment is correct. It takes much longer to just run the app for the first time. The savings come in when you want to redeploy the app in multiple environments, test it, and develop new features on it. That's when all of these steps make sense - although, hopefully you'll be able to remove steps 1 & 2 when everything is a docker / Kubernetes workload.
oblio 10y ago
Most of those technologies overlap. Except for some disaster development scenarios, no one is actually using all those technologies in the same environment (i.e. they're not using Chef and Vagrant and Docker in production, at the same time) Chef/Puppet/Salt/Ansible these days are just used to provision 1 VM/container (if even that, many people have just reverted to shell scripts) and that container is then managed using a container management tool (Swarm/Kubernetes).
- jimbokun 10y ago
  "(if even that, many people have just reverted to shell scripts)" Isn't Ansible basically just a way of executing shell scripts across multiple hosts in a sensible manner? At least that's how it's been explained to me.
  - brianwawok 10y ago
    Yes. Most of the value of Ansible is the library 100 of tasks like copy a file and replace variables, or restart a service if the config changed. It can do a lot of work in a few lines... But can also be a bit too much if you are already in a single container.

spudfkc 10y ago

I haven't dived into Kubernetes yet, but I set up Rancher for our new application and it has been nothing short of amazing so far. I can't express how happy we've been with it. I previously tried the Mesos/Marathon route (with Mesosphere and then again with Mantl) and that was nothing but a huge waste of time due to all the maintenance that was necessary for all the required servers. With Rancher, it's just spin up a container for each host with a single command and you're done.

webo 10y ago
I've been looking at Rancher for a few weeks, haven't put an application yet. How do you perform rolling updates with Rancher+Kubernetes? With Kubernetes itself, currently I just do `kubectl rolling-update <rc> --image <new-image>`

leetrout 10y ago

I know it's young still, but I think Nomad is going to get a share of this market with little effort. I played with Mesos & k8s and I picked Nomad instead. Now I'm not managing a huge fleet of servers I want to abstract away as much as I wanted a simple, consistent framework for allocating resources to containerized tasks across a small group of nodes. And I don't think that use case is anything to sneeze at and for a new user there just isn't anything out there as easy as nomad IMO. https://www.nomadproject.io/ https://www.nomadproject.io/

lmickh 10y ago
Nomad is definitely shaping up to be a nice alternative if k8s and mesos are overkill for your use case and you don't want to be vendor locked to a simplified option like ECS.
smnscu 10y ago
For a simple solution, also see Rancher. I've used them about 2 years ago and even then it was a very useful and stable tool. http://rancher.com/ http://rancher.com/
- loopbit 10y ago
  I also use Rancher and quite happy with it. The cool thing is that you can set up and environment using cattle (their own orchestration system), k8s, mesos or swarm. Up to you and your preferences.
friendly_chap 10y ago
I wanted to use nomad, but at the time their tutorials (not sure about their current state) were not really good, so I ended up rolling my own solution which serves me well since then (https://github.com/crufter/puller https://github.com/crufter/puller).
- bogomipz 10y ago
  I agree about lack of tutorial or documentation. Hashicorp documentation always feels like an internal wiki company wiki or a reference guide for someone who is already quite familiar with the APIs. I just checked the Nomads docs and it looks pretty much like it did when I looked at it 4 or 5 months ago.

DubiousPusher 10y ago

Seems like "Why Kubernetes is winning the orchestration war" is a more appropriate title for this?

mr_luc 10y ago

In my experience, I haven't been coming to k8s because I particularly like the developer experience (despite their efforts to focus heavily on it), but because it cleanly supported some things that I need. For instance, with k8s, out of the box every running container in a clustered system is discoverable and has its own IP. If you're writing distributed applications, and you're using containers principally as a tool to make your life easier (and not as part of an internal paas or for handling data pipelines or some other use case), having that sort of discovery available out of the box is great.

applepple 10y ago

It would be nice if open source projects (especially popular databases) came with k8s definition files so that you wouldn't have to write the yaml yourself.

jondubois 10y ago
I did that for my open source project SocketCluster. See https://github.com/SocketCluster/socketcluster/tree/master/kubernetes https://github.com/SocketCluster/socketcluster/tree/master/k... Also, if you look at the main Kubernetes repo on GitHub, you can find the yaml files for various popular open source projects in there as well. Some of them are out of date though - Things have been changing so fast. I think a problem that still comes up is that you still need yaml files to account for the interactions between different services too. Each service doesn't live in a vacuum. We're currently working on building an complete auto-scalable stack on top of Kubernetes and a deployment service to go with it. See https://baasil.io https://baasil.io Everything we do is open source except the actual website.
- hackitself 10y ago
  I use SC for my startup and it works great! Thanks. I'll have a look at the Baasil service, it sounds interesting.

contingencies 10y ago

IMHO nobody is winning anything that matters right now because the current transition is a transition to an additional level of abstraction which is definitely not properly met by any of the tools available. What we now need is tools that allow architectural fences around components, and reliability guarantees around subsystems ... versus not only technical but also business-level risks (including state-level actors)... often across borders, including for example exchange rate risk. This is based on business-level risk models, not some engineer feels X or Y type reasoning which is (often very well, but broad-picture uselessly) based on technical know-how. I prototyped such a system pretty successfully, you can read the design @ http://stani.sh/walter/cims/ http://stani.sh/walter/cims/ .. it's incomplete (critically hard to explain investment utility for non-tech business types) but at least infrastructure agnostic and practically informed. NB. To be non-humble, by way of demonstration I am a guy who has been called to conceive/design/set up/manage from-scratch datacenters for extremely visible businesses with highly motivated attackers with USD$Ms at stake for penetration. Systems I personally designed run millions of USD/month and have many times that in investment. And it's obvious that both Google ("compete with amazon on cloud business... high-level desperate for non-advertising income!") and Docker ("VC! growth! growth! growth!") have their own agendas here... we should trust no-one. It's early days. Bring on the ideas, build the future. Hack, hack, hack. We the little people own the future. It's ideas.

sebringj 10y ago

Just as an outside observer developing on a platform, I see my fellow devOps team members working in Kubernetes and it's been a shit storm for the most part where containers disappear randomly, stuff breaks and they are on call on the weekends not having fun. I have my own clients on other projects using AWS where I just upload and click buttons like a dumb monkey and I end up looking more competent even though I'm completely not. I've consequently not been motivated to dive into these DIY deployments just yet.

stcredzero 10y ago

It's all about knowing how to build an open source community This. Engineering excellence is secondary. You can get away with complete craptitude in your tech if you can build community. (I won't name examples.) Of course, it's better if you also have technical excellence. On the other hand, you can have technical excellence, but it will come to naught if you have community destroying anti-patterns.

amr_abdelrazik 10y ago

Disclaimer: I work for Mesosphere (Champions of Apache Mesos and DC/OS) We have total respect for K8s, but I don't think you can claim winning just based on the community and stars. OpenStack has a huge larger community of developers and advocates, but it is still haven't reached it's potential despite many years and incredible effort, seminars and summits Also most of these next gen infr projects now (DC/OS which is powered by Mesos, K8s, docker swarm ) are converging feature wise, but they also have their strengths and weaknesses, some of these are temporary, some of these are structurally by design Mesos (and DC/OS) were not just designed for scale, but also for extensibility and different workloads, which is why you can run Cassandra, Kafka, Spark, ..etc In production. None of these workloads run as traditional containers, with DC/OS they have their own operational logic such as honoring placement, simple installation, and upgrade, which is a core design function of the two-level scheduler People usually complain that standing up Mesos is hard, which is why we we built DC/OS and open sourced it to give you the power of mesos and its awesome capabilities in managing containers and data workloads without going through all the effort to stitch everything yourself. Check it out at DCOS.io, I am sure you guys will be blown away.

manojlds 10y ago
I have been trying to understand advantages of DC/OS over Mesos. Why should I even consider it? All I can understand is that it has a better(?) UI. I understand that even this and the DCOS CLI and packages work with Mesos too? So what's in there in DC/OS?
- amr_abdelrazik 10y ago
  Tl;DR: DC/OS provides all the components you need on top of Mesos to run in production, all pre-packaged and tested together with easy to install way Mesos provide the core resource management and scheduling, think of it as the Kernel of an (DC) operating system. On top of Mesos you usually need: 1- Container/Workload Scheduling capability which is Marathon, now built in and integrated in DC/OS so you are just up and running in minutes 2- Management (Beautiful DC/OS UI and CLI) to manage all cluster and workload operations. It also includes Installer (of all components including Zookeeper, networking, ..etc) 3- App Store (Universe) that allows you to easily find and deploy latest packages such as Kafka, Cassandra and Spark with a single command, in regular Mesos you needed to find the Framework that worked with a specific version of Mesos, troubleshooting, ...etc. Now it's an app store like experience 4- Core infrastructure components such as very cool service discovery and loadbalancing (Minuteman), Storage drivers, CNI support If you want to run docker containers, DC/OS also gives you the ability to use the docker daemon or the new Universal containerize (technology preview) which we just released. Because mesos is very modular and very powerful, it was built to have modular implementation of all system components, but our team and our founders ran some of the largest implementations of Mesos (Twitter, Airbnb, ..etc) and they complied all thes best practices and the bits and pieces in DC/OS. Hope that explains,

avitzurel 10y ago

I've used both mesosphere and Kube now (in production) and I feel I can safely comment on this. Kube is winning for the same reason React/Redux (and now Mobx) is winning and why Rails was winning at the time. Community. The community for Kube is awesome and the work people are doing in the field is being noticed all over the place. I've seen people (myself included) that moved production clusters from mesos to Kube just because the activity of the development and how secure they feel with the community and the project going forward. React and Rails (at the time) had the same sort of pull towards the community and why a lot of people on-boarded. Golang is most likely a factor here too. I feel most people find Golang friendlier than Scala/Java. That's why Kube has many more contributors, the hurdle for contributing is easier to jump

dominotw 10y ago
>I've seen people (myself included) that moved production clusters from mesos to Kube Sorry I am new to this. I saw a presentation where someone was running Kubernetes on DC/OS . Why are people doing that if they can just run pure kubernetes.
- avitzurel 10y ago
  I don't know why people would. I don't :)
- amr_abdelrazik 10y ago
  Some people like (the beta) K8s implementation on DC/OS because they like K8s abstraction for container orchestration , but at the same time they also like DC/OS because of distributed stateful (data) services support, While you can run everything in a container on K8s, the advanced data services require some special handling (see the comments about postgress and glusterfs from another commenter), and some people like to get the best of both worlds for example, if you restart a Kafka node on any container orchestration platform, container is destroyed and respawn on another node which is great, but you lose all your data and the broker has to rebuild all that queue again, while Kafka on DC/OS wait and maintains the volume just in case you are having temporary maintenance restart before relocating the workload. another example is Cassandra if you want to add more than 1 node, the traffic to resync the data across 2 nodes can actually degrade the performance on your cluster or even taking it down. I can go on and on, all of these issues and requirements come from running these mission critical data apps in production, with each application having its own logic on how it handles day 2 operations, but you don't know what you don't know unless you tried running them in production. Some people talk about the custom scheduler feature or Petsets in K8s, those features are cool and hats off to the K8s community, but they are still early in development while the framework architecture is a core mesos functionality since day 1. And finally, everyone is free to run whatever they want/like, each company and person might have different preferences on how they want to run workloads and how much effort they want to go through that, I don't think it's productive to go to another VI vs emacs here ;-)

kbredemeier 10y ago

Innovation is what makes this industry so exciting to be a part of. I joined the tech world, via Holberton School[1] just a mere nine months ago, and already so much has changed. It makes it challenging to keep up, but that's the fun part. The future of technology is held in the hands of open source projects. [1] https://www.holbertonschool.com https://www.holbertonschool.com

avitzurel 10y ago

One thing that I think Kube (and dc/os) are missing is what Chef is working on right now. Application definition should live within the app and consumed by the scheduler. Chef's product is called Habitat https://www.habitat.sh/ https://www.habitat.sh/ and it has some VERY interesting concepts that if Kube will implement it will be much more interesting (to a lot of people). Right now, the deployment and configuration of an application is supposed to be separated but I feel they need to be just a bit more coupled. An engineer that develops the application will define a few things like "domain" and how you connect to the application and this will be consumed by the scheduler. Right now, dc/os and mesos are really fine grained around the DevOps people and I feel that the first that will crack the "batteries included" approach will win the fight by a knock-out. Imagine something like Heroku on your own infrastructure, if an engineer can launch and deploy a micro-service with the same ease they deploy and access a Heroku application. That will be awesome.

kozikow 10y ago
> Right now, the deployment and configuration of an application is supposed to be separated but I feel they need to be just a bit more coupled. You can dynamically configure kubernetes from within the cluster. You can either use client-go library or talk to the api server directly with rest. For other languages than Golang you may need to set up proxy server, but it's a single kubectl command. So all the basic functionality is there, but IMO configuration should be separate if it is possible - having it in source control is nice. I only use dynamic api for scheduling kubernetes jobs.
- avitzurel 10y ago
  What I imagine is something like the Procfile for Heroku. The app decides what it runs and how, you can set configuration like domain, whether the application is public/private etc... Once you push the application to the scheduler it will handle the rest.
  - kozikow 10y ago
    Ah. Quite a lot of things you mention are possible by dynamically generating yaml config. I wrote a blog post about using go kubernetes library to generate yaml : https://kozikow.com/2016/09/02/using-go-to-autogenerate-kubernetes-configs/ https://kozikow.com/2016/09/02/using-go-to-autogenerate-kube.... It's quite nice, even for static config IMO, due to type safety. You can easily write golang binary that would ask user series of questions "what database would you like", "do you want public access", etc. and get yaml as output.

shadykiller 10y ago

The author is just gaga over google. He presents no comparison, no benchmarks and no mention of alternatives except docker swarm.

technologia 10y ago

I use Mesos and K8s heavily as well as contribute back to the projects, and while I do agree this is leaning towards being fan-fare there is a bit of truth to this. Community is a big deal, people tend to underestimate this; Putting aside newer companies, when a larger enterprise ventures out to open source they do take community as a major factor as you have to consider the tool you build will have to last 5 years, maybe 10, maybe more. To delve further into the Docker side of things, I personally wish that the company would focus on its core business instead of stretching itself with extra things. I get the need to improve the UX, which they do very well considering how far we have come from LXC. I feel Mesosphere starting to go down the feature creep route as well, but I wish them all the best as I loved Mesos since the beginning all those years ago.

graffitici 10y ago

Sad to see Mesos losing steam. My understanding was that Mesos subsumes the functionality of Kubernetes thanks to its Aurora scheduler. But it has much more customized schedulers, for different purposes, that might make it more efficient to run complicated pieces of software. For instance, it certainly is possible to run a Cassandra cluster by having each instance run in its own Docker container. My understanding is that it would be much more efficient to run this cluster with a dedicated Cassandra cluster instead. Is this right? Or are the performance benefits of running a dedicated Cassandra scheduler on Mesos negligible compared to running them in containers?

huntc 10y ago

Looking through all of these comments we should be glad, as a community, that there are a number players in this burgeoning orchestration space. Nobody has won and nobody should win. They each have their strengths and weaknesses and there is no one size that fits all.

nzoschke 10y ago

These articles never mention the elephant in the room, AWS. How many containers are running on Elastic Beanstalk and ECS? I'd wager magnitudes more are running containers by reading the docs and clicking around than those that mastering the cutting edge challenges getting running on Mesos, Kube or Swarm. Another blind spot is Heroku. Every day new businesses spin up containers with a 'git push Heroku master' and don't even know about it. All these providers, platforms and tools have their place. I simply don't think the "winning the war" talk is accurate or constructive. Disclaimer: I worked on the container systems at Heroku and now manage 100s of production ECS clusters at Convox.

colemickens 10y ago
What "cutting edge challenges" are there to get running on Mesos/Kube/Swarm? Are you just referring to getting the cluster booted? There are just so many options for that these days. Just curious, I was skimming through ECS docs last night and it seems like some of the basic building blocks look very similar to their equivalent constructs in Kube/Nomad.
- nzoschke 10y ago
  Automating VMs with Elastic Beanstalk and deploying to a platform like Heroku are effectively solved problems. Container orchestration brings in new challenges like operating a consensus database, managing data volumes and load balancing. And this is on top of VM management. Yes, ECS, Kube, Nomad are very similar. All relatively hard to use right now compared to their simpler ancestors. Here is more in depth info about the challenges of ECS: https://convox.com/blog/ecs-challenges/ https://convox.com/blog/ecs-challenges/

aCandidMind 10y ago

Why is no one mentioning Docker's response with version 1.12, the new built-in orchestration called swarm mode (different than just swarm): http://blog.nigelpoulton.com/docker-launches-kubernetes-killer/ http://blog.nigelpoulton.com/docker-launches-kubernetes-kill... Granted, the title is fanboyish, but it really seems to be a significant response to Kubernetes.