9 ms·

Kubernetes clusters for the hobbyist

321 points by pstadler 9y ago

guiriduro 9y ago
Great timing, I was wondering to myself about the feasibility of a 10€ cluster on scaleway last week.
bryanlarsen 9y ago
Why are you doing all of this stuff manually? There are several providers that will set all of this stuff up automatically for you. I like the Kismatic toolkit (https://github.com/apprenda/kismatic https://github.com/apprenda/kismatic), but there are a bunch of others. Sure, maybe once you go to production you'll want to install manually so that you have everything finely tuned the way you want, but learn it by using it rather than trying to have to figure things up front. Or even better just use GKE for development / learning purposes. Just stop the cluster when you're not using it, and it'll be a lot cheaper than something you won't want to take down because you spent days installing it.
- irontoby 9y ago
  Because Kubernetes is a complex beast with many moving parts, and learning about all those moving parts becomes more and more important as your usage grows. Personally I've used Stackpoint.io to provision some small clusters but I was very excited to see this project because deploying my own cluster from scratch is next on my todo list. Kelsey Hightower's "Kubernetes the Hard Way"[1] is the canonical go-to reference here but it's also very daunting so this looks like a great middle ground. Let's face it, even today the k8s docs can be quite sparse sometimes or gloss over the details, so knowing how all of the pieces work from the ground up can be a big help. Plus, you prevent vendor lock-in when whatever automated tool you're using doesn't solve your use-case or decides to start charging a lot of money. [1] https://github.com/kelseyhightower/kubernetes-the-hard-way https://github.com/kelseyhightower/kubernetes-the-hard-way
  - deleted 9y ago
    [deleted]
  - ben_jones 9y ago
    And if you want to be awed look up Kelsey Hightower's Github profile.
  - fndrplayer13 9y ago
    I agree with this assessment. We decided to do "kubernetes the 'sorta' hard way" by leveraging the Saltbase installer with some level of customization and full control via terraform of how our infrastructure was being allocated. I think its valuable to learn what the tool is doing if you have to maintain it. When something breaks, an upgrade has issues, or you need to better understand the system to make a decision, I feel that you gain a lot in setting up a system yourself. I think you'll be more likely to know precisely where to look to debug things. You also get closer to the tool which makes it easier to contribute back into the community. You also get the benefit of making your own infrastructure decisions. Yes k8s can provision ELBs and EBS volumes (and their equivalents in Google Cloud, Azure, etc) as well as autoscale nodes via a cluster addon, but the big moving pieces, such as instances, VPCs, Networking, etc, remain well-defined in Terraform or some other infra-as-code. That means that you can decide how to deploy that etcd cluster, how it gets backed up, whether or not its encrypted at rest, etc. Generally speaking, we just value the level of control and insight that we get out of controlling the stack definition ourselves. To some extent that may be antithetical to the purpose of k8s, since the goal of the project overall seems to be simplification and centralization of best practices of deployment. With all that being said, kops is an incredible tool (as are others) and we used it to learn about the system and test some of the functionality for ourselves. Can't recommend it enough.
- pstadler 9y ago
  There are indeed many options for automated setups. I did my first steps with Kubernetes on GKE, later followed the CoreOS guide to set it up on bare metal. This guide is for creators, a written form of my lessons learned. It should enable people to run secure clusters wherever they want. In case you missed it, there's a repository[1] in the same org which offers fully automated provisioning using Terraform. [1] https://github.com/hobby-kube/provisioning https://github.com/hobby-kube/provisioning
  - bryanlarsen 9y ago
    Yes, it does seem like an awesome resource for intermediate Kubernetes users like myself. But beginners should be steered away from it because it's too much, too soon. And by talking about choosing a cheap cloud provider at the beginning it implied to me that you were targeting beginners. Please add a disclaimer on the top pointing beginners at some of the better resources for them, and then tell them to come back once they've learned what stuff like "ingress" means.
  - maleck13 9y ago
    Used this recently and found it great. That's for the hard work and effort that clearly went into this.
- tyingq 9y ago
  Stackpoint.io is great for just spinning up k8s to get a feel for how it works. No charge, and all web based. It supports other providers, but I tried it with digital ocean. You pass it a DO API token, like this: https://cdn-images-1.medium.com/max/800/1*tcGINse5on6qnbsRYN4G3Q.png https://cdn-images-1.medium.com/max/800/1*tcGINse5on6qnbsRYN... Then it does the installation and gives you urls for the control panel. I don't think I'd use it for production, as everything is behind stackpoint.io urls, but for experimenting, it saves a ton of time.
  - irontoby 9y ago
    I've been using Stackpoint for a while and they do a great job, but be aware that they'll soon start charging $50/mo, which will remove them as a tool for hobbyists. https://stackpoint.io/#/pricing https://stackpoint.io/#/pricing They haven't announced a date for when they'll start charging, and I'm not sure how that will affect existing clusters.
    - treehau5 9y ago
      > they'll soon start charging $50/mo, which will remove them as a tool for hobbyists. What's wrong with paying for something as a hobby?
      - Nadya 9y ago
        If every tool/thing I relied on costed $50/mo I wouldn't be able to afford my hobby. I would need to seek a cheaper - possibly free - alternative or give up the hobby. Many people don't have $50/mo to spend on unprofitable hobby projects. There's nothing wrong with paying for hobbies, but most people's hobbies fall under (a) infrequent large purchases or (b) really cheap occasional costs. Upgrading my computer falls under (a). It's expensive but I only do it once every 3-4 years. Buying new guitar strings for my guitar every so often falls under (b). It's cheap and while I should do it every-other-month I do it maybe twice a year. If the cost of new guitar strings was equivalent to the cost of upgrading my computer - I'd give up playing guitar as the hobby would become too expensive.
- MattRogish 9y ago
  Building this stuff by hand is... yeah. We open source our Kube bootstrap toolkit (which uses kops). No one should build this by hand. It would be like, I dunno, setting up a MariaDB Galera Cluster by hand when you intend to use RDS (not a perfect example, please no pedantry!). It's a fun learning experience, I guess - if you're into that sorta thing - but is not what you want to use in any production context. https://www.reactiveops.com/blog/kops-102-an-inside-look-at-deploying-kubernetes/ https://www.reactiveops.com/blog/kops-102-an-inside-look-at-... https://www.reactiveops.com/blog/using-k8s-scripts/ https://www.reactiveops.com/blog/using-k8s-scripts/ Use the automation to bootup clusters, please!
  - sagichmal 9y ago
    I strongly disagree. Rolling out a K8s cluster to a production environment without knowing how to "do it by hand", relying instead on helper tools that abstract away the complexity, is deeply irresponsible, even professionally negligent. Like learning shortcuts in math, you only "earn" automation once you've done things the hard way often enough to know how it works at a low level. That knowledge is critical. Otherwise you're just cargo-culting something you don't understand.
- scurvy 9y ago
  Hobbyists like to do things for themselves and better understand how things work. It's the nature of being a hobbyist rather than a consumer.
jug5 9y ago
Good to see the author using wireguard as an additional network security layer
paukiatwee 9y ago
Really great resources! I was working on my own version of k8s setup scripts using Ansible, and I will definitely use this guide to improve mine.
- daviddumenil 9y ago
  Can I ask what you thought of the k8s roles in Ansible Galaxy?
bryanlarsen 9y ago
I found gluster-kubernetes quite simple to install. But the install instructions do assume that you're going to be giving it it's own partition, which you would be doing on any sort of real production deployment.
fndrplayer13 9y ago
Great set of resources -- I just went through the process of defining a terraform cluster in AWS over the past few weeks, though I'm leveraging the k8s Saltbase installer for the master and nodes. I'm curious, why no mention of AWS as a provider for roll-your-own? Is this a cost thing? Also, I get the feeling that Ubuntu is _not_ a first class citizen of the k8s ecosystem, but perhaps my newness to the ecosystem is to blame here. The Saltbase installer, for example, only supports Debian and RHEL distros, `kops` prefers Debian, and the documentation for cluster deployments on kubernetes.io and elsewhere also seems to be somewhat suggestive of Debian and Core OS. Perhaps thats just a mistaken interpretation on my part. I'm curious what other peoples thoughts on this topic are!
- justinsb 9y ago
  kops doesn't necessarily prefer debian - we support Ubuntu, Debian, CentOS/RHEL, CoreOS and Google's Container OS. One of the outputs of the Kubernetes-on-AWS efforts is an AMI that is "Kubernetes Optimized" - a 4.4 kernel, Docker pre-installed, lots of inodes etc. That AMI _is_ based on Debian, hence the suggestion is that if you don't otherwise care (and my hope is that eventually you won't), that you should probably just use that AMI. But if you do have a preference, by all means use your distro of choice.
- dustinkirkland 9y ago
  Ubuntu is absolutely a 1st class citizen in the K8s Ecosystem! The front page of https://kubernetes.io/docs/ https://kubernetes.io/docs/ has a bullet that links to a super simple way to deploy Kubernetes to Ubuntu on any of [localhost, baremetal cluster, public cloud, private cloud]! See: * Installing Kubernetes on Ubuntu: Deploy a Kubernetes cluster on-premise, baremetal, cloud providers, or localhost with Charms and conjure-up.
- rdli 9y ago
  We found that setting up Kube on AWS in a prod-ready way was sufficiently complex enough that we wrote up some docs on it: https://www.datawire.io/guide/infrastructure/setting-kubernetes-aws/ https://www.datawire.io/guide/infrastructure/setting-kuberne..., hoping it would be helpful.
- 9y ago
empath75 9y ago
You can spin up a cluster on gce in a couple of minutes.
- ralmeida 9y ago
  There are a couple reasons to do it manually and/or outsite GKE, notably: 1) Cost. In VPSs like Digital Ocean/Scaleway, there's usually a large network out transfer quota included in the price, which just isn't there with GCP, where you pay for network usage on a metered basis. 2) Learning. Although you can defer most of the heavy work to GKE, it's still good to understand the moving parts so you can make better choices as you grow.
- rhizome 9y ago
  What do you do if something goes wrong with it?
ramshanker 9y ago
I just had my first read of Kubernates. Looks doable. Time to jump on the bandwagon.
oblio 9y ago
It seems that a proper Kubernetes setup is the modern day equivalent to the proper email server setup of the 90's or the 00's :)
- cryptarch 9y ago
  You mean because it is now considered a basic requirement to have failover and horizontal scaling?
  - oblio 9y ago
    I mean because of the high number of working parts for a fully functional system: Docker, Kubernetes, Etcs, Kubeadm, etc., etc.
    - cryptarch 9y ago
      Oh, so it's about the high complexity and many interoperating parts? I'm just asking because I was born in the 90's, I'm not very aware of the state of email server management then.
      - oblio 9y ago
        Well, let's just say that today we have tutorial and bootstrap projects for Javascript frameworks, back then we had tutorials and bootstrap projects for setting up email servers with spam filters :)
- erikb 9y ago
  Yeah, the community follows the wrong approach. It's not so different what two services need in features. There should be an end-to-end solution. But each tool, including kubernetes, only delivers something that is not even 100% of one feature and hopes that someone else comes up with a solution for the other features. Usually in my day job I have to work with these tools, and currently have a stack of 5, that mostly have incomplete documentation, zero explanation of how they actually solve the problem and nearly zero ability to debug (e.g. what value have kubernetes logs and events. Usually when you have a problem is when you have no kubernetes logs yet/anymore and the events only tell you what you already know). Now I'll probably need to learn another one, considering these three options for storage. At the weekend, where I'm mostly trying to relax and have physically only 2/7th of the time to alot I learn the basics behind containerization, e.g. namespaces, cgroups, virtual network adapters, iptables. And I feel in this small time slot I make a lot more progress at getting to that end-to-end solution that people actually need. The example I'm using is wordpress+mysql. It's a simple thing that covers >70% of what anybody wants to deploy. And on-premise with kubernetes+docker it's still not possible without hacks (e.g. for volumeclaims and logging), after, I don't know, 4 years of Kubernetes? I bet in spending 2 years worth of weekends any normal person can come up with something better. --- Re missing features, examples for Kubernetes: A) Why does Kubernetes not solve the networking part? If I have a cluster and containers that may run in different places, of course the tooling I use to maintain that cluster needs to ensure that containers can talk to each other. There can be an abstract API and the option for other people to write plugins, but the core needs to come with one solution, that mostly works and is debuggable when not. B) CrashLoopBackoff. Why did no Kubernetes developer get the idea that this state may require any kind of log/debugging? C) Why does kubernetes assign random ports to services and not provide a simple way to retrieve them? Of course I can get them after I've learned the JSON api. But usually that is not considered a solution but hacking. I really don't care what the service is running on, I just want to use it. D) Why do I need to manually say how many containers should run for each service? There are very distinct options a user may need. E.g., most services should run 1 instance and replace it if it dies. If it needs reliability I want to define how reliable it should be and accordingly the cluster should decide if it needs 2, 3 or 5 instances. And lastly I want to run stuff on all my nodes. Each node with a container. That is not even possible afaik. E) Most kubernetes tools are not working well with environment proxies. For instance kube-proxy shell tool will completely bug out. But surprise, clusters make a lot of sense in enterprise environments, and enterprise environments have proxies. It's also not a hard setup for testing. A raspberry pie can be your home networks proxy. F) on premise storage solutions, considering that a restarted container may not run on the same host. G) Since not much is really running right now we didn't run into that problem yet. But it's totally possible that the whole cluster runs out of resources. I haven't seen any piece of info about the general cluster status and when I need to increase or replace infrastructure. Honestly if I don't have all these things solved, am I really better of using Kubernetes or writing my own scripts? I think it currently is about equal in effort. And if that's the case my own solution has the huge advantage of being under my control and allowing me to learn a lot of new things.
raesene6 9y ago
This guide makes an interesting choice with regards to etcd security, which I'm not sure I'd go with. etcd stores a load of sensitive cluster information, so unauthorised access to it is a bad thing. There's an assumption in the guide that you have a "secure network" and therefore don't have to worry about etcd authentication/encryption. The thing is if you have a compromised container (say) and that container, which has an in-cluster IP address can see your etcd server, then it can easily dump the etcd database and get access to the information held in it... Personally I'd recommend setting up a small CA for etcd and using it's authentication features, there's a good guide to this on the CoreOS site https://coreos.com/etcd/docs/latest/op-guide/security.html https://coreos.com/etcd/docs/latest/op-guide/security.html
- pstadler 9y ago
  This is great input and definitely something worth considering. Related issue on GitHub: https://github.com/hobby-kube/guide/issues/6 https://github.com/hobby-kube/guide/issues/6
  - raesene6 9y ago
    no worries, I've got some more info. on that on my blog https://raesene.github.io/blog/2017/05/01/Kubernetes-Security-etcd/ https://raesene.github.io/blog/2017/05/01/Kubernetes-Securit... which may be of use.
- 013a 9y ago
  Is the issue you outline something an out-of-the-box Kops cluster would suffer from?
  - raesene6 9y ago
    That's a very interesting question. The answer is that it depends on the K8s distribution in question. One of the challenges of K8s security is that there are a wide variety of distributions, each with their own defaults and configuration, so it's hard to make blanket statements about their security. I've definitely seen some with this issue out of the box.
dkarapetyan 9y ago
I like the juxtaposition of the words "hobbyist" and "kubernetes cluster".
- deleted 9y ago
  [deleted]
tuco86 9y ago
exactly what i was looking for! Eureka!
gtirloni 9y ago
I'm surprised a hobbyist K8s administrator is not choosing to use kubeadm instead. https://kubernetes.io/docs/getting-started-guides/kubeadm/ https://kubernetes.io/docs/getting-started-guides/kubeadm/
- pstadler 9y ago
  ¯\_(ツ)_/¯
- cweagans 9y ago
  https://github.com/hobby-kube/guide#installing-kubernetes https://github.com/hobby-kube/guide#installing-kubernetes > There are plenty of ways to set up a Kubernetes cluster from scratch. At this point however, we settle on kubeadm. This dramatically simplifies the setup process by automating the creation of certificates, services and configuration files.
  - deleted 9y ago
    [deleted]
_rp6i 9y ago
The second question: > Choosing a cloud provider This really annoys me about Kubernetes. Essentially all the official documentation is about how to select a cloud and let a cloud-specific tool magically do everything for you. There's no procedure for setting up a single host for development purposes or to have a Dokku-like personal PaaS. This guide is super useful because it avoids all the magic and lets you set things up properly (despite assuming you're doing it on a cloud) and potentially even do it on a single host.
- user5994461 9y ago
  It's for real world distributed systems. The sort of thing that cannot run or be simulated on a single machine. It's not worth bothering to make a single machine setup.
  - subway 9y ago
    Except they provide a official single-machine setup called minikube. Sadly minikube suffers from the same issue GP describes - it is just calling one of the existing automagic bootstrapping scripts against a local VM.
  - Veratyr 9y ago
    > It's for real world distributed systems. The same can be said of Heroku, yet Dokku exists. The same can be said of Docker, yet plenty of hobbyists use it for their projects. > It's not worth bothering to make a single machine setup. Sure it is. Say you're a web developer and you use MySQL, PHP and Apache on Kubernetes in prod. Why would you go to the effort of setting those up individually on your machine when you could just put your prod manifest on Kubernetes on your workstation? There's nothing about Kubernetes (aside from poor documentation) that prevents it from being used on a single host. Actually, according to its own documentation (https://kubernetes.io/docs/getting-started-guides/scratch/ https://kubernetes.io/docs/getting-started-guides/scratch/), it consists of: - Docker/Rkt - Kubelet - Kube-proxy - Etcd - Apiserver - Controller - Scheduler The bottom 6 are actually written in Go and can be run from a single binary. There's no reason Kubernetes should be hard to set up. There's no reason it can't run on a single node. There's no reason you need a cloud. There's no reason you need a "supported" operating system.
deleted 9y ago
[deleted]

321 points by pstadler 9y ago

Great timing, I was wondering to myself about the feasibility of a 10€ cluster on scaleway last week.

Why are you doing all of this stuff manually? There are several providers that will set all of this stuff up automatically for you. I like the Kismatic toolkit (https://github.com/apprenda/kismatic https://github.com/apprenda/kismatic), but there are a bunch of others. Sure, maybe once you go to production you'll want to install manually so that you have everything finely tuned the way you want, but learn it by using it rather than trying to have to figure things up front. Or even better just use GKE for development / learning purposes. Just stop the cluster when you're not using it, and it'll be a lot cheaper than something you won't want to take down because you spent days installing it.

irontoby 9y ago
Because Kubernetes is a complex beast with many moving parts, and learning about all those moving parts becomes more and more important as your usage grows. Personally I've used Stackpoint.io to provision some small clusters but I was very excited to see this project because deploying my own cluster from scratch is next on my todo list. Kelsey Hightower's "Kubernetes the Hard Way"[1] is the canonical go-to reference here but it's also very daunting so this looks like a great middle ground. Let's face it, even today the k8s docs can be quite sparse sometimes or gloss over the details, so knowing how all of the pieces work from the ground up can be a big help. Plus, you prevent vendor lock-in when whatever automated tool you're using doesn't solve your use-case or decides to start charging a lot of money. [1] https://github.com/kelseyhightower/kubernetes-the-hard-way https://github.com/kelseyhightower/kubernetes-the-hard-way
- deleted 9y ago
  [deleted]
- ben_jones 9y ago
  And if you want to be awed look up Kelsey Hightower's Github profile.
- fndrplayer13 9y ago
  I agree with this assessment. We decided to do "kubernetes the 'sorta' hard way" by leveraging the Saltbase installer with some level of customization and full control via terraform of how our infrastructure was being allocated. I think its valuable to learn what the tool is doing if you have to maintain it. When something breaks, an upgrade has issues, or you need to better understand the system to make a decision, I feel that you gain a lot in setting up a system yourself. I think you'll be more likely to know precisely where to look to debug things. You also get closer to the tool which makes it easier to contribute back into the community. You also get the benefit of making your own infrastructure decisions. Yes k8s can provision ELBs and EBS volumes (and their equivalents in Google Cloud, Azure, etc) as well as autoscale nodes via a cluster addon, but the big moving pieces, such as instances, VPCs, Networking, etc, remain well-defined in Terraform or some other infra-as-code. That means that you can decide how to deploy that etcd cluster, how it gets backed up, whether or not its encrypted at rest, etc. Generally speaking, we just value the level of control and insight that we get out of controlling the stack definition ourselves. To some extent that may be antithetical to the purpose of k8s, since the goal of the project overall seems to be simplification and centralization of best practices of deployment. With all that being said, kops is an incredible tool (as are others) and we used it to learn about the system and test some of the functionality for ourselves. Can't recommend it enough.
pstadler 9y ago
There are indeed many options for automated setups. I did my first steps with Kubernetes on GKE, later followed the CoreOS guide to set it up on bare metal. This guide is for creators, a written form of my lessons learned. It should enable people to run secure clusters wherever they want. In case you missed it, there's a repository[1] in the same org which offers fully automated provisioning using Terraform. [1] https://github.com/hobby-kube/provisioning https://github.com/hobby-kube/provisioning
- bryanlarsen 9y ago
  Yes, it does seem like an awesome resource for intermediate Kubernetes users like myself. But beginners should be steered away from it because it's too much, too soon. And by talking about choosing a cheap cloud provider at the beginning it implied to me that you were targeting beginners. Please add a disclaimer on the top pointing beginners at some of the better resources for them, and then tell them to come back once they've learned what stuff like "ingress" means.
- maleck13 9y ago
  Used this recently and found it great. That's for the hard work and effort that clearly went into this.
tyingq 9y ago
Stackpoint.io is great for just spinning up k8s to get a feel for how it works. No charge, and all web based. It supports other providers, but I tried it with digital ocean. You pass it a DO API token, like this: https://cdn-images-1.medium.com/max/800/1*tcGINse5on6qnbsRYN4G3Q.png https://cdn-images-1.medium.com/max/800/1*tcGINse5on6qnbsRYN... Then it does the installation and gives you urls for the control panel. I don't think I'd use it for production, as everything is behind stackpoint.io urls, but for experimenting, it saves a ton of time.
- irontoby 9y ago
  I've been using Stackpoint for a while and they do a great job, but be aware that they'll soon start charging $50/mo, which will remove them as a tool for hobbyists. https://stackpoint.io/#/pricing https://stackpoint.io/#/pricing They haven't announced a date for when they'll start charging, and I'm not sure how that will affect existing clusters.
  - treehau5 9y ago
    > they'll soon start charging $50/mo, which will remove them as a tool for hobbyists. What's wrong with paying for something as a hobby?
    - Nadya 9y ago
      If every tool/thing I relied on costed $50/mo I wouldn't be able to afford my hobby. I would need to seek a cheaper - possibly free - alternative or give up the hobby. Many people don't have $50/mo to spend on unprofitable hobby projects. There's nothing wrong with paying for hobbies, but most people's hobbies fall under (a) infrequent large purchases or (b) really cheap occasional costs. Upgrading my computer falls under (a). It's expensive but I only do it once every 3-4 years. Buying new guitar strings for my guitar every so often falls under (b). It's cheap and while I should do it every-other-month I do it maybe twice a year. If the cost of new guitar strings was equivalent to the cost of upgrading my computer - I'd give up playing guitar as the hobby would become too expensive.
MattRogish 9y ago
Building this stuff by hand is... yeah. We open source our Kube bootstrap toolkit (which uses kops). No one should build this by hand. It would be like, I dunno, setting up a MariaDB Galera Cluster by hand when you intend to use RDS (not a perfect example, please no pedantry!). It's a fun learning experience, I guess - if you're into that sorta thing - but is not what you want to use in any production context. https://www.reactiveops.com/blog/kops-102-an-inside-look-at-deploying-kubernetes/ https://www.reactiveops.com/blog/kops-102-an-inside-look-at-... https://www.reactiveops.com/blog/using-k8s-scripts/ https://www.reactiveops.com/blog/using-k8s-scripts/ Use the automation to bootup clusters, please!
- sagichmal 9y ago
  I strongly disagree. Rolling out a K8s cluster to a production environment without knowing how to "do it by hand", relying instead on helper tools that abstract away the complexity, is deeply irresponsible, even professionally negligent. Like learning shortcuts in math, you only "earn" automation once you've done things the hard way often enough to know how it works at a low level. That knowledge is critical. Otherwise you're just cargo-culting something you don't understand.
scurvy 9y ago
Hobbyists like to do things for themselves and better understand how things work. It's the nature of being a hobbyist rather than a consumer.

jug5 9y ago

Good to see the author using wireguard as an additional network security layer

paukiatwee 9y ago

Really great resources! I was working on my own version of k8s setup scripts using Ansible, and I will definitely use this guide to improve mine.

daviddumenil 9y ago
Can I ask what you thought of the k8s roles in Ansible Galaxy?

bryanlarsen 9y ago

I found gluster-kubernetes quite simple to install. But the install instructions do assume that you're going to be giving it it's own partition, which you would be doing on any sort of real production deployment.

fndrplayer13 9y ago

Great set of resources -- I just went through the process of defining a terraform cluster in AWS over the past few weeks, though I'm leveraging the k8s Saltbase installer for the master and nodes. I'm curious, why no mention of AWS as a provider for roll-your-own? Is this a cost thing? Also, I get the feeling that Ubuntu is _not_ a first class citizen of the k8s ecosystem, but perhaps my newness to the ecosystem is to blame here. The Saltbase installer, for example, only supports Debian and RHEL distros, `kops` prefers Debian, and the documentation for cluster deployments on kubernetes.io and elsewhere also seems to be somewhat suggestive of Debian and Core OS. Perhaps thats just a mistaken interpretation on my part. I'm curious what other peoples thoughts on this topic are!

justinsb 9y ago
kops doesn't necessarily prefer debian - we support Ubuntu, Debian, CentOS/RHEL, CoreOS and Google's Container OS. One of the outputs of the Kubernetes-on-AWS efforts is an AMI that is "Kubernetes Optimized" - a 4.4 kernel, Docker pre-installed, lots of inodes etc. That AMI _is_ based on Debian, hence the suggestion is that if you don't otherwise care (and my hope is that eventually you won't), that you should probably just use that AMI. But if you do have a preference, by all means use your distro of choice.
dustinkirkland 9y ago
Ubuntu is absolutely a 1st class citizen in the K8s Ecosystem! The front page of https://kubernetes.io/docs/ https://kubernetes.io/docs/ has a bullet that links to a super simple way to deploy Kubernetes to Ubuntu on any of [localhost, baremetal cluster, public cloud, private cloud]! See: * Installing Kubernetes on Ubuntu: Deploy a Kubernetes cluster on-premise, baremetal, cloud providers, or localhost with Charms and conjure-up.
rdli 9y ago
We found that setting up Kube on AWS in a prod-ready way was sufficiently complex enough that we wrote up some docs on it: https://www.datawire.io/guide/infrastructure/setting-kubernetes-aws/ https://www.datawire.io/guide/infrastructure/setting-kuberne..., hoping it would be helpful.
9y ago

empath75 9y ago

You can spin up a cluster on gce in a couple of minutes.

ralmeida 9y ago
There are a couple reasons to do it manually and/or outsite GKE, notably: 1) Cost. In VPSs like Digital Ocean/Scaleway, there's usually a large network out transfer quota included in the price, which just isn't there with GCP, where you pay for network usage on a metered basis. 2) Learning. Although you can defer most of the heavy work to GKE, it's still good to understand the moving parts so you can make better choices as you grow.
rhizome 9y ago
What do you do if something goes wrong with it?

ramshanker 9y ago

I just had my first read of Kubernates. Looks doable. Time to jump on the bandwagon.

oblio 9y ago

It seems that a proper Kubernetes setup is the modern day equivalent to the proper email server setup of the 90's or the 00's :)

cryptarch 9y ago
You mean because it is now considered a basic requirement to have failover and horizontal scaling?
- oblio 9y ago
  I mean because of the high number of working parts for a fully functional system: Docker, Kubernetes, Etcs, Kubeadm, etc., etc.
  - cryptarch 9y ago
    Oh, so it's about the high complexity and many interoperating parts? I'm just asking because I was born in the 90's, I'm not very aware of the state of email server management then.
    - oblio 9y ago
      Well, let's just say that today we have tutorial and bootstrap projects for Javascript frameworks, back then we had tutorials and bootstrap projects for setting up email servers with spam filters :)
erikb 9y ago
Yeah, the community follows the wrong approach. It's not so different what two services need in features. There should be an end-to-end solution. But each tool, including kubernetes, only delivers something that is not even 100% of one feature and hopes that someone else comes up with a solution for the other features. Usually in my day job I have to work with these tools, and currently have a stack of 5, that mostly have incomplete documentation, zero explanation of how they actually solve the problem and nearly zero ability to debug (e.g. what value have kubernetes logs and events. Usually when you have a problem is when you have no kubernetes logs yet/anymore and the events only tell you what you already know). Now I'll probably need to learn another one, considering these three options for storage. At the weekend, where I'm mostly trying to relax and have physically only 2/7th of the time to alot I learn the basics behind containerization, e.g. namespaces, cgroups, virtual network adapters, iptables. And I feel in this small time slot I make a lot more progress at getting to that end-to-end solution that people actually need. The example I'm using is wordpress+mysql. It's a simple thing that covers >70% of what anybody wants to deploy. And on-premise with kubernetes+docker it's still not possible without hacks (e.g. for volumeclaims and logging), after, I don't know, 4 years of Kubernetes? I bet in spending 2 years worth of weekends any normal person can come up with something better. --- Re missing features, examples for Kubernetes: A) Why does Kubernetes not solve the networking part? If I have a cluster and containers that may run in different places, of course the tooling I use to maintain that cluster needs to ensure that containers can talk to each other. There can be an abstract API and the option for other people to write plugins, but the core needs to come with one solution, that mostly works and is debuggable when not. B) CrashLoopBackoff. Why did no Kubernetes developer get the idea that this state may require any kind of log/debugging? C) Why does kubernetes assign random ports to services and not provide a simple way to retrieve them? Of course I can get them after I've learned the JSON api. But usually that is not considered a solution but hacking. I really don't care what the service is running on, I just want to use it. D) Why do I need to manually say how many containers should run for each service? There are very distinct options a user may need. E.g., most services should run 1 instance and replace it if it dies. If it needs reliability I want to define how reliable it should be and accordingly the cluster should decide if it needs 2, 3 or 5 instances. And lastly I want to run stuff on all my nodes. Each node with a container. That is not even possible afaik. E) Most kubernetes tools are not working well with environment proxies. For instance kube-proxy shell tool will completely bug out. But surprise, clusters make a lot of sense in enterprise environments, and enterprise environments have proxies. It's also not a hard setup for testing. A raspberry pie can be your home networks proxy. F) on premise storage solutions, considering that a restarted container may not run on the same host. G) Since not much is really running right now we didn't run into that problem yet. But it's totally possible that the whole cluster runs out of resources. I haven't seen any piece of info about the general cluster status and when I need to increase or replace infrastructure. Honestly if I don't have all these things solved, am I really better of using Kubernetes or writing my own scripts? I think it currently is about equal in effort. And if that's the case my own solution has the huge advantage of being under my control and allowing me to learn a lot of new things.

raesene6 9y ago

This guide makes an interesting choice with regards to etcd security, which I'm not sure I'd go with. etcd stores a load of sensitive cluster information, so unauthorised access to it is a bad thing. There's an assumption in the guide that you have a "secure network" and therefore don't have to worry about etcd authentication/encryption. The thing is if you have a compromised container (say) and that container, which has an in-cluster IP address can see your etcd server, then it can easily dump the etcd database and get access to the information held in it... Personally I'd recommend setting up a small CA for etcd and using it's authentication features, there's a good guide to this on the CoreOS site https://coreos.com/etcd/docs/latest/op-guide/security.html https://coreos.com/etcd/docs/latest/op-guide/security.html

pstadler 9y ago
This is great input and definitely something worth considering. Related issue on GitHub: https://github.com/hobby-kube/guide/issues/6 https://github.com/hobby-kube/guide/issues/6
- raesene6 9y ago
  no worries, I've got some more info. on that on my blog https://raesene.github.io/blog/2017/05/01/Kubernetes-Security-etcd/ https://raesene.github.io/blog/2017/05/01/Kubernetes-Securit... which may be of use.
013a 9y ago
Is the issue you outline something an out-of-the-box Kops cluster would suffer from?
- raesene6 9y ago
  That's a very interesting question. The answer is that it depends on the K8s distribution in question. One of the challenges of K8s security is that there are a wide variety of distributions, each with their own defaults and configuration, so it's hard to make blanket statements about their security. I've definitely seen some with this issue out of the box.

dkarapetyan 9y ago

I like the juxtaposition of the words "hobbyist" and "kubernetes cluster".

deleted 9y ago
[deleted]

tuco86 9y ago

exactly what i was looking for! Eureka!

gtirloni 9y ago

I'm surprised a hobbyist K8s administrator is not choosing to use kubeadm instead. https://kubernetes.io/docs/getting-started-guides/kubeadm/ https://kubernetes.io/docs/getting-started-guides/kubeadm/

pstadler 9y ago
¯\_(ツ)_/¯
cweagans 9y ago
https://github.com/hobby-kube/guide#installing-kubernetes https://github.com/hobby-kube/guide#installing-kubernetes > There are plenty of ways to set up a Kubernetes cluster from scratch. At this point however, we settle on kubeadm. This dramatically simplifies the setup process by automating the creation of certificates, services and configuration files.
- deleted 9y ago
  [deleted]

_rp6i 9y ago

The second question: > Choosing a cloud provider This really annoys me about Kubernetes. Essentially all the official documentation is about how to select a cloud and let a cloud-specific tool magically do everything for you. There's no procedure for setting up a single host for development purposes or to have a Dokku-like personal PaaS. This guide is super useful because it avoids all the magic and lets you set things up properly (despite assuming you're doing it on a cloud) and potentially even do it on a single host.

user5994461 9y ago
It's for real world distributed systems. The sort of thing that cannot run or be simulated on a single machine. It's not worth bothering to make a single machine setup.
- subway 9y ago
  Except they provide a official single-machine setup called minikube. Sadly minikube suffers from the same issue GP describes - it is just calling one of the existing automagic bootstrapping scripts against a local VM.
- Veratyr 9y ago
  > It's for real world distributed systems. The same can be said of Heroku, yet Dokku exists. The same can be said of Docker, yet plenty of hobbyists use it for their projects. > It's not worth bothering to make a single machine setup. Sure it is. Say you're a web developer and you use MySQL, PHP and Apache on Kubernetes in prod. Why would you go to the effort of setting those up individually on your machine when you could just put your prod manifest on Kubernetes on your workstation? There's nothing about Kubernetes (aside from poor documentation) that prevents it from being used on a single host. Actually, according to its own documentation (https://kubernetes.io/docs/getting-started-guides/scratch/ https://kubernetes.io/docs/getting-started-guides/scratch/), it consists of: - Docker/Rkt - Kubelet - Kube-proxy - Etcd - Apiserver - Controller - Scheduler The bottom 6 are actually written in Go and can be run from a single binary. There's no reason Kubernetes should be hard to set up. There's no reason it can't run on a single node. There's no reason you need a cloud. There's no reason you need a "supported" operating system.

deleted 9y ago

[deleted]