5 ms·

> None of these systems scale up from a single 1 node system to a full distributed cluster What do you mean? This is exactly what they do.

tinco 10y ago
No, most scale up from a 3 or 6 node cluster to an N node cluster.
- endymi0n 10y ago
  https://kubernetes.io/docs/getting-started-guides/minikube/ https://kubernetes.io/docs/getting-started-guides/minikube/ > Running Kubernetes Locally via Minikube > Minikube is a tool that makes it easy to run Kubernetes locally. Minikube runs a single-node Kubernetes cluster inside a VM on your laptop for users looking to try out Kubernetes or develop with it day-to-day.
  - tinco 10y ago
    Yes, so the problem is I think that for the parent it is hard to scale minikube out to a full cluster.
    - hosh 10y ago
      I don't contribute to Kubernetes, but I am curious about what the parent or what you mean by that. We use 1-node for dev work and then n-node cluster on staging and production. My team's dev workflow spans from 1-node to n-node just fine. Granted, I also wrote Matsuri to consolidate managing dev and production systems. But I also know that K8S 1.5 and 1.6 introduced a lot of resources that makes it easier to manage things like that (such as ConfigSets). I also know that Helm has been doing a lot in this area. (I had never used it because it overlaps with what Matsuri does).
      - tinco 10y ago
        I think they want to start their production environment on one node, and then scale as their product grows. I'm not saying that it's a particularly good approach, but I've seen situations where people frown when you say that your 10 user product needs at least 3 machines to function, just because it's run on cluster software that's designed to build products for millions of users on. It would be easier to sell if you could run it on 2 or even 1 server, and still grow the cluster to proper size when the circumstances demand it. Of course, there are very good reasons for it to require 3 machines, and since machines are so cheap, it should be absolutely a no brainer to spawn a $5 node to get that quorum running, but that's I think the argument for it.
        hosh 10y ago
        I think maybe people are under the mistaken impression K8S requires forming some sort of quorum. You can run 1 node in production if that is what you really want. You run 3-nodes for availability and resilience. You can spread the 3-nodes across availability zones (isolating failure domains). If there is sufficient capacity, then one out of three nodes can go down, and K8S master will reschedule the pods in the down node. This also allows for controlled upgrades. If those are not a requirement (your prototype doesn't need that kind of uptime), then why not run it in 1-node? The only subsystem I know of that has a quorum mechanic is etcd backing k8s master. The docs will recommended to run the backing etcd in a 3-node configuration-- again, for availability. However, I run k8s worker, k8s master, and 1-node etcd on the same dev box. (The trick is to tell etcd to listen to three ports and tell the single etcd node that those other ports are part of the quorum). There is no reason the 10-user app that can tolerate some downtime cannot run like that either. I do think though, K8S has not yet closed the gap for people who want to prototype quickly and get something out. Heroku is a great platform for that. There is an migration path that goes from Heroku to Deis (which is built on top of Kubernetes and will work with Heroku build-paths).
  - cookiecaper 10y ago
    Correct, minikube emulates a cluster via virtual machines for local development. It's a hack.
    - lobster_johnson 10y ago
      It's not a hack, it's a product: "A version of Kubernetes that works anywhere, with zero configuration." You can run Kubernetes locally without a VM just fine, but a VM encapsulates everything nicely without needing to configure the host's network and so on. It doesn't "emulate" a cluster, it creates a cluster that happens to contain only one node. Plus, it works on non-Linux machines.
- hosh 10y ago
  I run Kubernetes on a 1-node cluster for development on my laptop (as do the rest of my team), and we scale up just fine to our 6-node staging and 9-node production cluster.
  - fredsir 10y ago
    May I ask how you do it? I've thought about using minikube for local development myself as a way of being able to reuse helm charts so instead of having one setup for dev (perhaps docker compose or simply the stack standard way) and one for staging/production (helm), it's all the same helm chart with just different values. Another reason is to be able to encapsulate dev environments better. Right now, every stack I use have different ways to do it (pythons virtualenv for example). If I were using minikube, everything would be using the same encapsulation. I've tried setting it up, and it works fine, using host mapping so files are still reloaded when files change locally on and so on, but something I haven't solved yet is that in this setup, the only way stack specific packages/modules/whatever is installed, is inside the container, which is a problem because my editor/IDE requires the packages/modules to be available locally for it to be able to do "intellisense". Have you and your team delt with the same problem or other problems? How have you solved it? Any tips and thoughts are much appreciated!
    - hosh 10y ago
      No problem. I wrote Matsuri as a way to manage this. https://github.com/matsuri-rb/matsuri https://github.com/matsuri-rb/matsuri This was written with K8S 1.1 in mind, and later somewhat updated to 1.4. It was primarily intended to solve: (1) Consolidate interfaces for working with dev and production (or other environments such as staging, demo, etc.) (2) It implements this by being a manifest generator and feeding the manifest into `kubectl`. This was designed and written before ConfigSet, so some of what Matsuri solved is now solvable with ConfigSet. Matsuri is written in Ruby, and so it takes advantages of class inheritance and method override to achieve what it does. The class definitions used generate the manifest is broken down into composable chunks, which can be overridden at any point. In this way, I can have a base template for say, a web pod for dev, staging, and production. In dev, I would then method-override what is returned for volume mounts and volumes. In that way, I can specify source mounting and SSH agent sock mounting for dev ... but not for staging or production. In another example, I can use method override to augment the environment variables passed into the container in order to capture the uid/gid so that the container can reconfigure itself to use those. There are helper methods to generate things like, correct format of ports or volumes. None of those are necessary if you already know how to format it. Likewise, since a manifest is just a class definition, replication controller, replicasets, and deployments all reference a pod and use those to create the pod template spec. I then have an extra resource called an "app" that does not map to K8S. They define a bundle of K8S resources or other apps that can be converged together. That allows someone to quickly setup a dev environment with all the moving pieces. Those also include hooks for you to do things like, define the command for shelling (with correct uid/gid) or console (which depends; redis console is invoked differently than a rails console). This allows a consolidation ... for example, to log into my dev rails console, I would do something like: bin/dev console myapp And to do that on production: bin/production console myapp (Powerful, but I suspect there will be some asks around security here). I had flirted with the idea of allowing plugins or packages ... and then Helm was announced. I figured more people will use that than Matsuri. But one possibility might be to get Matsuri to work with Helm. There are a lot of rough spots with Matsuri. The command-line interface could be done a lot better (right now, it assumes you will always be in the root of your platform repo). There are no documentation or examples for Matsuri. It does not implement all of the resources. It also helps a lot if you know Ruby. Infrastructure is not my full-time role on my team. Being a small team and early stage, we have had to wear multiple hats. I had not had the energy or time to document and promote this. If you want to chat about this further, feel free to email me at hosheng.hsiao at gmail.

tinco 10y ago

No, most scale up from a 3 or 6 node cluster to an N node cluster.

endymi0n 10y ago
https://kubernetes.io/docs/getting-started-guides/minikube/ https://kubernetes.io/docs/getting-started-guides/minikube/ > Running Kubernetes Locally via Minikube > Minikube is a tool that makes it easy to run Kubernetes locally. Minikube runs a single-node Kubernetes cluster inside a VM on your laptop for users looking to try out Kubernetes or develop with it day-to-day.
- tinco 10y ago
  Yes, so the problem is I think that for the parent it is hard to scale minikube out to a full cluster.
  - hosh 10y ago
    I don't contribute to Kubernetes, but I am curious about what the parent or what you mean by that. We use 1-node for dev work and then n-node cluster on staging and production. My team's dev workflow spans from 1-node to n-node just fine. Granted, I also wrote Matsuri to consolidate managing dev and production systems. But I also know that K8S 1.5 and 1.6 introduced a lot of resources that makes it easier to manage things like that (such as ConfigSets). I also know that Helm has been doing a lot in this area. (I had never used it because it overlaps with what Matsuri does).
    - tinco 10y ago
      I think they want to start their production environment on one node, and then scale as their product grows. I'm not saying that it's a particularly good approach, but I've seen situations where people frown when you say that your 10 user product needs at least 3 machines to function, just because it's run on cluster software that's designed to build products for millions of users on. It would be easier to sell if you could run it on 2 or even 1 server, and still grow the cluster to proper size when the circumstances demand it. Of course, there are very good reasons for it to require 3 machines, and since machines are so cheap, it should be absolutely a no brainer to spawn a $5 node to get that quorum running, but that's I think the argument for it.
      - hosh 10y ago
        I think maybe people are under the mistaken impression K8S requires forming some sort of quorum. You can run 1 node in production if that is what you really want. You run 3-nodes for availability and resilience. You can spread the 3-nodes across availability zones (isolating failure domains). If there is sufficient capacity, then one out of three nodes can go down, and K8S master will reschedule the pods in the down node. This also allows for controlled upgrades. If those are not a requirement (your prototype doesn't need that kind of uptime), then why not run it in 1-node? The only subsystem I know of that has a quorum mechanic is etcd backing k8s master. The docs will recommended to run the backing etcd in a 3-node configuration-- again, for availability. However, I run k8s worker, k8s master, and 1-node etcd on the same dev box. (The trick is to tell etcd to listen to three ports and tell the single etcd node that those other ports are part of the quorum). There is no reason the 10-user app that can tolerate some downtime cannot run like that either. I do think though, K8S has not yet closed the gap for people who want to prototype quickly and get something out. Heroku is a great platform for that. There is an migration path that goes from Heroku to Deis (which is built on top of Kubernetes and will work with Heroku build-paths).
- cookiecaper 10y ago
  Correct, minikube emulates a cluster via virtual machines for local development. It's a hack.
  - lobster_johnson 10y ago
    It's not a hack, it's a product: "A version of Kubernetes that works anywhere, with zero configuration." You can run Kubernetes locally without a VM just fine, but a VM encapsulates everything nicely without needing to configure the host's network and so on. It doesn't "emulate" a cluster, it creates a cluster that happens to contain only one node. Plus, it works on non-Linux machines.
hosh 10y ago
I run Kubernetes on a 1-node cluster for development on my laptop (as do the rest of my team), and we scale up just fine to our 6-node staging and 9-node production cluster.
- fredsir 10y ago
  May I ask how you do it? I've thought about using minikube for local development myself as a way of being able to reuse helm charts so instead of having one setup for dev (perhaps docker compose or simply the stack standard way) and one for staging/production (helm), it's all the same helm chart with just different values. Another reason is to be able to encapsulate dev environments better. Right now, every stack I use have different ways to do it (pythons virtualenv for example). If I were using minikube, everything would be using the same encapsulation. I've tried setting it up, and it works fine, using host mapping so files are still reloaded when files change locally on and so on, but something I haven't solved yet is that in this setup, the only way stack specific packages/modules/whatever is installed, is inside the container, which is a problem because my editor/IDE requires the packages/modules to be available locally for it to be able to do "intellisense". Have you and your team delt with the same problem or other problems? How have you solved it? Any tips and thoughts are much appreciated!
  - hosh 10y ago
    No problem. I wrote Matsuri as a way to manage this. https://github.com/matsuri-rb/matsuri https://github.com/matsuri-rb/matsuri This was written with K8S 1.1 in mind, and later somewhat updated to 1.4. It was primarily intended to solve: (1) Consolidate interfaces for working with dev and production (or other environments such as staging, demo, etc.) (2) It implements this by being a manifest generator and feeding the manifest into `kubectl`. This was designed and written before ConfigSet, so some of what Matsuri solved is now solvable with ConfigSet. Matsuri is written in Ruby, and so it takes advantages of class inheritance and method override to achieve what it does. The class definitions used generate the manifest is broken down into composable chunks, which can be overridden at any point. In this way, I can have a base template for say, a web pod for dev, staging, and production. In dev, I would then method-override what is returned for volume mounts and volumes. In that way, I can specify source mounting and SSH agent sock mounting for dev ... but not for staging or production. In another example, I can use method override to augment the environment variables passed into the container in order to capture the uid/gid so that the container can reconfigure itself to use those. There are helper methods to generate things like, correct format of ports or volumes. None of those are necessary if you already know how to format it. Likewise, since a manifest is just a class definition, replication controller, replicasets, and deployments all reference a pod and use those to create the pod template spec. I then have an extra resource called an "app" that does not map to K8S. They define a bundle of K8S resources or other apps that can be converged together. That allows someone to quickly setup a dev environment with all the moving pieces. Those also include hooks for you to do things like, define the command for shelling (with correct uid/gid) or console (which depends; redis console is invoked differently than a rails console). This allows a consolidation ... for example, to log into my dev rails console, I would do something like: bin/dev console myapp And to do that on production: bin/production console myapp (Powerful, but I suspect there will be some asks around security here). I had flirted with the idea of allowing plugins or packages ... and then Helm was announced. I figured more people will use that than Matsuri. But one possibility might be to get Matsuri to work with Helm. There are a lot of rough spots with Matsuri. The command-line interface could be done a lot better (right now, it assumes you will always be in the root of your platform repo). There are no documentation or examples for Matsuri. It does not implement all of the resources. It also helps a lot if you know Ruby. Infrastructure is not my full-time role on my team. Being a small team and early stage, we have had to wear multiple hats. I had not had the energy or time to document and promote this. If you want to chat about this further, feel free to email me at hosheng.hsiao at gmail.