5 ms·
I too am confuzzled by this. I'm at a Docker shop where we currently run on DC/OS and Marathon. I think the most confusing thing about Docker right now are all
by djsumdog 10y ago
I too am confuzzled by this. I'm at a Docker shop where we currently run on DC/OS and Marathon. I think the most confusing thing about Docker right now are all the different scheduling and networking frameworks (DC/OS which adds a web ui to mesos and marathon, Kubernetes, Nomad, CoreOS/Fleet, etc).
None of these systems scale up from a single 1 node system to a full distributed cluster. Here was my attempt of trying to get DC/OS to run in a minimal cluster on Digital Ocean:
http://penguindreams.org/blog/installing-mesosphere-dcos-on-small-digital-ocean-droplets/ http://penguindreams.org/blog/installing-mesosphere-dcos-on-...
We use DC/OS at work and the web UI frequently transfers over 1MB of json a second and the failed containers tab can max out CPU at 100% O_o
Marathon, Kubernetes, Nomad and Swarm all have their own orchestration json files in totally different formats. It gets more confusing with you take about pluggable network layers (WeaveNet, Flannel).
I'd _like_ to think Docker is trying to bring some standardisation to clusters, networking and scheduling ... but I'm not sure if that's the case.
- manojlds 10y ago> None of these systems scale up from a single 1 node system to a full distributed cluster What do you mean? This is exactly what they do.
- tinco 10y agoNo, most scale up from a 3 or 6 node cluster to an N node cluster.
- endymi0n 10y agohttps://kubernetes.io/docs/getting-started-guides/minikube/ https://kubernetes.io/docs/getting-started-guides/minikube/ > Running Kubernetes Locally via Minikube > Minikube is a tool that makes it easy to run Kubernetes locally. Minikube runs a single-node Kubernetes cluster inside a VM on your laptop for users looking to try out Kubernetes or develop with it day-to-day.
- tinco 10y agoYes, so the problem is I think that for the parent it is hard to scale minikube out to a full cluster.
- hosh 10y agoI don't contribute to Kubernetes, but I am curious about what the parent or what you mean by that. We use 1-node for dev work and then n-node cluster on staging and production. My team's dev workflow spans from 1-node to n-node just fine. Granted, I also wrote Matsuri to consolidate managing dev and production systems. But I also know that K8S 1.5 and 1.6 introduced a lot of resources that makes it easier to manage things like that (such as ConfigSets). I also know that Helm has been doing a lot in this area. (I had never used it because it overlaps with what Matsuri does).
- tinco 10y agoI think they want to start their production environment on one node, and then scale as their product grows. I'm not saying that it's a particularly good approach, but I've seen situations where people frown when you say that your 10 user product needs at least 3 machines to function, just because it's run on cluster software that's designed to build products for millions of users on. It would be easier to sell if you could run it on 2 or even 1 server, and still grow the cluster to proper size when the circumstances demand it. Of course, there are very good reasons for it to require 3 machines, and since machines are so cheap, it should be absolutely a no brainer to spawn a $5 node to get that quorum running, but that's I think the argument for it.
- hosh 10y agoI think maybe people are under the mistaken impression K8S requires forming some sort of quorum. You can run 1 node in production if that is what you really want. You run 3-nodes for availability and resilience. You can spread the 3-nodes across availability zones (isolating failure domains). If there is sufficient capacity, then one out of three nodes can go down, and K8S master will reschedule the pods in the down node. This also allows for controlled upgrades. If those are not a requirement (your prototype doesn't need that kind of uptime), then why not run it in 1-node? The only subsystem I know of that has a quorum mechanic is etcd backing k8s master. The docs will recommended to run the backing etcd in a 3-node configuration-- again, for availability. However, I run k8s worker, k8s master, and 1-node etcd on the same dev box. (The trick is to tell etcd to listen to three ports and tell the single etcd node that those other ports are part of the quorum). There is no reason the 10-user app that can tolerate some downtime cannot run like that either. I do think though, K8S has not yet closed the gap for people who want to prototype quickly and get something out. Heroku is a great platform for that. There is an migration path that goes from Heroku to Deis (which is built on top of Kubernetes and will work with Heroku build-paths).
- hosh 10y agoI run Kubernetes on a 1-node cluster for development on my laptop (as do the rest of my team), and we scale up just fine to our 6-node staging and 9-node production cluster.
- fredsir 10y agoMay I ask how you do it? I've thought about using minikube for local development myself as a way of being able to reuse helm charts so instead of having one setup for dev (perhaps docker compose or simply the stack standard way) and one for staging/production (helm), it's all the same helm chart with just different values. Another reason is to be able to encapsulate dev environments better. Right now, every stack I use have different ways to do it (pythons virtualenv for example). If I were using minikube, everything would be using the same encapsulation. I've tried setting it up, and it works fine, using host mapping so files are still reloaded when files change locally on and so on, but something I haven't solved yet is that in this setup, the only way stack specific packages/modules/whatever is installed, is inside the container, which is a problem because my editor/IDE requires the packages/modules to be available locally for it to be able to do "intellisense". Have you and your team delt with the same problem or other problems? How have you solved it? Any tips and thoughts are much appreciated!
- hosh 10y agoNo problem. I wrote Matsuri as a way to manage this. https://github.com/matsuri-rb/matsuri https://github.com/matsuri-rb/matsuri This was written with K8S 1.1 in mind, and later somewhat updated to 1.4. It was primarily intended to solve: (1) Consolidate interfaces for working with dev and production (or other environments such as staging, demo, etc.) (2) It implements this by being a manifest generator and feeding the manifest into `kubectl`. This was designed and written before ConfigSet, so some of what Matsuri solved is now solvable with ConfigSet. Matsuri is written in Ruby, and so it takes advantages of class inheritance and method override to achieve what it does. The class definitions used generate the manifest is broken down into composable chunks, which can be overridden at any point. In this way, I can have a base template for say, a web pod for dev, staging, and production. In dev, I would then method-override what is returned for volume mounts and volumes. In that way, I can specify source mounting and SSH agent sock mounting for dev ... but not for staging or production. In another example, I can use method override to augment the environment variables passed into the container in order to capture the uid/gid so that the container can reconfigure itself to use those. There are helper methods to generate things like, correct format of ports or volumes. None of those are necessary if you already know how to format it. Likewise, since a manifest is just a class definition, replication controller, replicasets, and deployments all reference a pod and use those to create the pod template spec. I then have an extra resource called an "app" that does not map to K8S. They define a bundle of K8S resources or other apps that can be converged together. That allows someone to quickly setup a dev environment with all the moving pieces. Those also include hooks for you to do things like, define the command for shelling (with correct uid/gid) or console (which depends; redis console is invoked differently than a rails console). This allows a consolidation ... for example, to log into my dev rails console, I would do something like: bin/dev console myapp And to do that on production: bin/production console myapp (Powerful, but I suspect there will be some asks around security here). I had flirted with the idea of allowing plugins or packages ... and then Helm was announced. I figured more people will use that than Matsuri. But one possibility might be to get Matsuri to work with Helm. There are a lot of rough spots with Matsuri. The command-line interface could be done a lot better (right now, it assumes you will always be in the root of your platform repo). There are no documentation or examples for Matsuri. It does not implement all of the resources. It also helps a lot if you know Ruby. Infrastructure is not my full-time role on my team. Being a small team and early stage, we have had to wear multiple hats. I had not had the energy or time to document and promote this. If you want to chat about this further, feel free to email me at hosheng.hsiao at gmail.
- lobster_johnson 10y agoKubernetes runs fine on a single node, and scales to thousands. Lots of developers use a single node in a VM to develop apps locally. You wouldn't want to run a single node in production, of course, because that defeats the point (and introduces a single point of failure). But there's nothing technically preventing you from doing it. As a Kubernetes user, I hope Docker as an application is going away, because it's increasingly in the way. Kubernetes would benefit from managing its containers with a more direct, more lightweight mechanism (like rkt or containerd).
- drdaeman 10y agoTook me some time to figure it out, but Kubernetes actually does, with kubeadm. If you're on a Debian/Ubuntu-based OS, you basically `apt-get install kubeadm kubelet kubectl`, then `kubeadm init` (and you have 1-node cluster) Then you `kubeadm join --token $token $master_node_ip` and you have the second node. Repeat as necessary, throw in some proper automation when you have enough (3+) nodes. There is a disclaimer that it's beta, though. Still, Docker feels much simpler, and I really believe "simpler" means "better". Maybe that's just my prejudice, but Kubernetes has awfully enterprisey aftertaste and feels to be full of magic. If Docker or Swarm goes haywire - and every software has bugs - I can check almost every component, piece by piece, down to what kernel is told to do - thankfully, they're still not too far from the actual OS primitives. If affected node number is tolerably small, I can even "downgrade" to Compose and manual scheduling and networking kludges, with relative ease. I'm unsure what I would do if some day Kubernetes would insists on malfunctioning.
- djsumdog 10y agoI've played with Kubernetes and I wish I had finished my post on it. I got frustrated when looking at logs from a node wasn't supported by nodes joined with kubeadm (that was a few months back .. I really hope they fixed that by now). I agree with you, Kubernetes seems insanely complicated. I haven't used it in production though. I'm in a Marathon shop, and it is really slick once you have a working DC/OS cluster .. but setting up that cluster requires a full time team. It's not trivial and far from simple.