5 ms·

provisioning, partial-mesh network K8s cluster

95 points by guifortaine 9y ago

oarsinsync 9y ago
As someone who doesn't know enough about K8s, this looks like an amazingly easy way to get stuck in. Can anyone who has experience with real deployments advise on what pain-points may be encountered by growing something like this further than N nodes (and talk about what N might be?)
- raesene9 9y ago
  If you want to get started with Kubernetes I'd recommend either, Kelsey Hightower's Kubernetes the hard way (https://github.com/kelseyhightower/kubernetes-the-hard-way https://github.com/kelseyhightower/kubernetes-the-hard-way) which is a good way to start getting an understanding of the various components and how they fit together or, if you want to try out running workloads on a cluster, Kubeadm (https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/ https://kubernetes.io/docs/setup/independent/create-cluster-...) If you're interested in books on the topic, I like Kubernetes in Action (https://www.manning.com/books/kubernetes-in-action https://www.manning.com/books/kubernetes-in-action) or Kubernetes: Up and running (http://shop.oreilly.com/product/0636920043874.do http://shop.oreilly.com/product/0636920043874.do) One of the challenges with Kubernetes is that it's pretty fast moving, so it's a good idea to work with resources that are up to date. I know a last commit from May doesn't seem very old but that's going to a least be missing 1.7 and could be missing 1.6 or earlier as well.
  - meddlepal 9y ago
    Just to parrot... the governance and development model for Kubernetes is amazing. There are releases every three months and it is like clockwork. That said I usually wait for the first point release to ship before upgrading. 1.x.0 releases always feel like an RC and there's usually some critical bug fixed between 1.x.0 and 1.x.1. Project plug. I am building a free service and tool for instantaneous cloud-running ephemeral Kubernetes clusters called Kubernaut. If you just want to play around with Kubernetes and start learning it is a great tool. Also useful for CI use via Travis: https://github.com/datawire/kubernaut https://github.com/datawire/kubernaut
- dguaraglia 9y ago
  I've found a few not-so-obvious pain points on my very limited k8s experience. 1) k8s makes a lot of sense for stateless applications (such as your website) but not so much for stateful applications that require a client to connect to the same container every time (there are ways to do it, but they are a pain in the ass.) 2) Tooling is getting better with time, but it's still pretty green. Packages for your usual orchestration tools like Puppet and Ansible are volunteer work so they get easily out of sync, or require more work than you'd expect to get going. Using their suggested YAML format leads to another problem: there's no easy way to keep secrets outside of the configuration files, unless you build your own process around it 3) Some pieces, like a replicated DB, might be easier to be kept outside of the k8s cluster. You can technically run them there, but they weren't designed for running in that kind of environment and sometimes it shows 4) The CI/CD story using Jenkins pipelines is far from solved. There are several packages that provide some solutions, but the documentation tends to be horrible and that leads to days of debugging through trial and error 5) Leaky abstractions everywhere. As an example, the Jenkins plugins to build using your k8s cluster suggests using the "credentials" system, but you need to add them manually after you boot the Jenkins service. Then your slaves stop receiving the credentials and you have to reboot Jenkins (I had to reboot it on average every 3 or 4 builds.) Don't let that discourage you from using k8s as a "run" cluster, if your app is a shared-nothing, stateless app. It's so much easier to setup (specially on Azure and GCP) and it obviates the need for setting up Puppet + Sensu + load balancing just to make sure your service keeps running when a node dies.
  - alexnewman 9y ago
    1) That's true, but why not support shared state on shared storage. If that's not an option. 2) What exactly do you not like tooling wise. 3) Totally agree. I am actually working on replacing etcd with postgresql though.
    - gtirloni 9y ago
      Could you share more about #3 ? How do you view that in regard to this discussion? https://github.com/kubernetes/kubernetes/issues/1957 https://github.com/kubernetes/kubernetes/issues/1957
web007 9y ago
This sounds like buzzword bingo. I can't tell without reading the source, is this any different from minikube? I see a mention of NUCs and netboot in the README, so I'm guessing it's slightly more, but not documented.
- mbryant 9y ago
  There's a better explanation of what's going on over here: http://ocadotechnology.com/blog/creating-a-distributed-data-centre-architecture-using-kubernetes-and-containers/ http://ocadotechnology.com/blog/creating-a-distributed-data-... There's some libvirt stuff, but that's mostly for local testing without having a bunch of physical hardware. The key things I'm trying to do are: - Automatically provision new machines, just by plugging them in. - Don't have a traditional network (i.e. switches)
  - corndoge 9y ago
    From the article, you're using Quagga -- are you aware of the active fork FRRouting[0]? It was forked about 8 months ago and is over 3,000 commits ahead since most of the core developers have stopped developing Quagga. [0] https://github.com/FRRouting/frr https://github.com/FRRouting/frr
    - mbryant 9y ago
      Yes, but thanks :) FRRouting took off after I'd already moved on to other things. I've dropped an issue in the repo so I remember, when I get a chance to work on Kubermesh again
- jzelinskie 9y ago
  NUCs are pretty standard for how many people are testing bare metal installations on the cheap. It's what we use in the labs at CoreOS. This looks like an opinionated installation for an actual bare metal cluster complete with anycast mirrors of popular registries. Minikube is only really good for running in a VM locally.
  - 9y ago
dominotw 9y ago
kubernetes has so much momentum and developer interest. I rarely see anything about DC/OS which is what they choose at my work last month, apparently its DC/OS is more 'battle tested' in enterprise. I don't think i even want to setup dc/os on my laptop and play with it, seems almost impossible to run mesos on osx . All these articles are making me resent DC/OS :D.
- justicezyx 9y ago
  DC/OS was developed with objectives that are not overlapping with K8s'. Namely, they target machine management. While K8s is about container centric job management. Seems subtle, but if you think along these 2 lines, design decisions in 2 systems fall through naturally.
  - dominotw 9y ago
    >DC/OS was developed with objectives that are not overlapping with K8s'. from their home page ' deploys containers, distributed services, and legacy applications into those machines; ' Containers is listed first and its definitely an overlap with kubernetes . I understand that mesos was originally developed with non container centric focus but dc/os now definitely container focused. First example on their site in graphic is 'Containerized Workloads '.
- BlackjackCF 9y ago
  Having worked in DC/OS prior to Kubernetes, I'm going to say that it's much easier to jump into DC/OS development (working with the API etc). However, Kubernetes has been a lot easier for me to get running a lot faster, mostly due to the much more active community. It also feels like the recent feature sets that Mesosphere has been delivering to Marathon (their choice of scheduler to sit on top of Mesos) is playing catchup to a lot of the features that Kubernetes already has.

95 points by guifortaine 9y ago

As someone who doesn't know enough about K8s, this looks like an amazingly easy way to get stuck in. Can anyone who has experience with real deployments advise on what pain-points may be encountered by growing something like this further than N nodes (and talk about what N might be?)

raesene9 9y ago
If you want to get started with Kubernetes I'd recommend either, Kelsey Hightower's Kubernetes the hard way (https://github.com/kelseyhightower/kubernetes-the-hard-way https://github.com/kelseyhightower/kubernetes-the-hard-way) which is a good way to start getting an understanding of the various components and how they fit together or, if you want to try out running workloads on a cluster, Kubeadm (https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/ https://kubernetes.io/docs/setup/independent/create-cluster-...) If you're interested in books on the topic, I like Kubernetes in Action (https://www.manning.com/books/kubernetes-in-action https://www.manning.com/books/kubernetes-in-action) or Kubernetes: Up and running (http://shop.oreilly.com/product/0636920043874.do http://shop.oreilly.com/product/0636920043874.do) One of the challenges with Kubernetes is that it's pretty fast moving, so it's a good idea to work with resources that are up to date. I know a last commit from May doesn't seem very old but that's going to a least be missing 1.7 and could be missing 1.6 or earlier as well.
- meddlepal 9y ago
  Just to parrot... the governance and development model for Kubernetes is amazing. There are releases every three months and it is like clockwork. That said I usually wait for the first point release to ship before upgrading. 1.x.0 releases always feel like an RC and there's usually some critical bug fixed between 1.x.0 and 1.x.1. Project plug. I am building a free service and tool for instantaneous cloud-running ephemeral Kubernetes clusters called Kubernaut. If you just want to play around with Kubernetes and start learning it is a great tool. Also useful for CI use via Travis: https://github.com/datawire/kubernaut https://github.com/datawire/kubernaut
dguaraglia 9y ago
I've found a few not-so-obvious pain points on my very limited k8s experience. 1) k8s makes a lot of sense for stateless applications (such as your website) but not so much for stateful applications that require a client to connect to the same container every time (there are ways to do it, but they are a pain in the ass.) 2) Tooling is getting better with time, but it's still pretty green. Packages for your usual orchestration tools like Puppet and Ansible are volunteer work so they get easily out of sync, or require more work than you'd expect to get going. Using their suggested YAML format leads to another problem: there's no easy way to keep secrets outside of the configuration files, unless you build your own process around it 3) Some pieces, like a replicated DB, might be easier to be kept outside of the k8s cluster. You can technically run them there, but they weren't designed for running in that kind of environment and sometimes it shows 4) The CI/CD story using Jenkins pipelines is far from solved. There are several packages that provide some solutions, but the documentation tends to be horrible and that leads to days of debugging through trial and error 5) Leaky abstractions everywhere. As an example, the Jenkins plugins to build using your k8s cluster suggests using the "credentials" system, but you need to add them manually after you boot the Jenkins service. Then your slaves stop receiving the credentials and you have to reboot Jenkins (I had to reboot it on average every 3 or 4 builds.) Don't let that discourage you from using k8s as a "run" cluster, if your app is a shared-nothing, stateless app. It's so much easier to setup (specially on Azure and GCP) and it obviates the need for setting up Puppet + Sensu + load balancing just to make sure your service keeps running when a node dies.
- alexnewman 9y ago
  1) That's true, but why not support shared state on shared storage. If that's not an option. 2) What exactly do you not like tooling wise. 3) Totally agree. I am actually working on replacing etcd with postgresql though.
  - gtirloni 9y ago
    Could you share more about #3 ? How do you view that in regard to this discussion? https://github.com/kubernetes/kubernetes/issues/1957 https://github.com/kubernetes/kubernetes/issues/1957

web007 9y ago

This sounds like buzzword bingo. I can't tell without reading the source, is this any different from minikube? I see a mention of NUCs and netboot in the README, so I'm guessing it's slightly more, but not documented.

mbryant 9y ago
There's a better explanation of what's going on over here: http://ocadotechnology.com/blog/creating-a-distributed-data-centre-architecture-using-kubernetes-and-containers/ http://ocadotechnology.com/blog/creating-a-distributed-data-... There's some libvirt stuff, but that's mostly for local testing without having a bunch of physical hardware. The key things I'm trying to do are: - Automatically provision new machines, just by plugging them in. - Don't have a traditional network (i.e. switches)
- corndoge 9y ago
  From the article, you're using Quagga -- are you aware of the active fork FRRouting[0]? It was forked about 8 months ago and is over 3,000 commits ahead since most of the core developers have stopped developing Quagga. [0] https://github.com/FRRouting/frr https://github.com/FRRouting/frr
  - mbryant 9y ago
    Yes, but thanks :) FRRouting took off after I'd already moved on to other things. I've dropped an issue in the repo so I remember, when I get a chance to work on Kubermesh again
jzelinskie 9y ago
NUCs are pretty standard for how many people are testing bare metal installations on the cheap. It's what we use in the labs at CoreOS. This looks like an opinionated installation for an actual bare metal cluster complete with anycast mirrors of popular registries. Minikube is only really good for running in a VM locally.
- 9y ago

dominotw 9y ago

kubernetes has so much momentum and developer interest. I rarely see anything about DC/OS which is what they choose at my work last month, apparently its DC/OS is more 'battle tested' in enterprise. I don't think i even want to setup dc/os on my laptop and play with it, seems almost impossible to run mesos on osx . All these articles are making me resent DC/OS :D.

justicezyx 9y ago
DC/OS was developed with objectives that are not overlapping with K8s'. Namely, they target machine management. While K8s is about container centric job management. Seems subtle, but if you think along these 2 lines, design decisions in 2 systems fall through naturally.
- dominotw 9y ago
  >DC/OS was developed with objectives that are not overlapping with K8s'. from their home page ' deploys containers, distributed services, and legacy applications into those machines; ' Containers is listed first and its definitely an overlap with kubernetes . I understand that mesos was originally developed with non container centric focus but dc/os now definitely container focused. First example on their site in graphic is 'Containerized Workloads '.
BlackjackCF 9y ago
Having worked in DC/OS prior to Kubernetes, I'm going to say that it's much easier to jump into DC/OS development (working with the API etc). However, Kubernetes has been a lot easier for me to get running a lot faster, mostly due to the much more active community. It also feels like the recent feature sets that Mesosphere has been delivering to Marathon (their choice of scheduler to sit on top of Mesos) is playing catchup to a lot of the features that Kubernetes already has.