9 ms·
Kubernetes 1.4: Making it easy to run on Kubernetes anywhere
- TheIronYuppie 10y agoHuge congrats to the team - - Incredible simple setup [1] - Stateful application support with integrated Helm [2] - New cross-cluster federation features (secrets, events and namespaces, and lots more in alpha) [3] And lots more... please let us know if you have any questions! [1] http://kubernetes.io/docs/getting-started-guides/kubeadm/ http://kubernetes.io/docs/getting-started-guides/kubeadm/ [2] https://github.com/kubernetes/helm/blob/master/docs/charts.md https://github.com/kubernetes/helm/blob/master/docs/charts.m... [3] http://kubernetes.io/docs/user-guide/federation/ http://kubernetes.io/docs/user-guide/federation/ Disclosure: I work at Google on Kubernetes
- merb 10y agoWell yeah it looks great. However for ease of use in "small" scale. it would be cool to have a built in Load Balancer (a pretty pretty simple one, just http, maybe https via LetsEncrypt). At the moment adding a Cloud Load Balancer per Project is probably not affordable in the smaller project scale (and it would make any PaaS useless). (I mean that is not a "big" problem, it's ok to build it myself (with less quality than you can ;)) But I wanna say Kubernetes is extremly good designed and the amazing part is, is that you can start with a really really really really small cluster (like 1 master 1 node) and later you can raise that pretty easily
- TheIronYuppie 10y agoWe've heard this quite a lot - there's actually a few ways to tackle this, one of the most common is to either use Ingress (which is a built in L7 load balancer) or run an nginx instance in a pod on the cluster (https://github.com/nginxinc/kubernetes-ingress https://github.com/nginxinc/kubernetes-ingress), and, in either case, direct all traffic to your node port across your cluster. It's really one command to do either - if you have trouble, please ping me (aronchick (at) google)! Disclosure: I work at Google on Kubernetes
- x0rg 10y agoIs there a way to do this properly in AWS without nginx? It would also be great to have features to switch only percentage of the traffic to an app when doing blue/green deployments.
- TheIronYuppie 10y agoThe best way to do blue/green is put them behind a single endpoint. So, if your endpoint is 'app: node-app', then you put the label 'app: node-app' on BOTH your existing version, and your future version, and target all traffic to a service with the selector 'app: node-app'. Then, you slowly start to spin up your new instances from 1 -> 10 -> 100 (or whatever). The traffic will split automatically because both apps have the same label/selector, and you control the amount by how many instances of each you have. Disclosure: I work at Google on Kubernetes
- pdeva1 10y agodoes kubernetes take care of 'connection draining', ie making sure requests in flight are completed before killing traffic to an instance?
- sporkland 10y agoSort of a blend. It will send a sigterm to your process first which should be your signal to start draining and exit when they are done. If you don't finish within a configurable timeout then sigkill is sent.
- x0rg 10y agoI was wondering how to make this automated through some sort of pipeline that needs a human to click "go on with the next X % of the rollout and how I would do it with kubectl without too much pain.
- sporkland 10y agoWe've been doing this with multiple deployments (e.g. 1, half, all) and updating the deployments sequentially when the previous one looked good. (these are all fronted by one service).
- olafmol 10y agoTake a look at Vamp (vamp.io) which also integrates with Kubernetes and is a programmable router based on HAproxy. It does loadbalancing, but also percentage and condition-based canary-releasing: http://vamp.io/documentation/installation/kubernetes/ http://vamp.io/documentation/installation/kubernetes/ (disclaimer: I'm the co-founder, looking forward to hear your thoughts and feedback!)
- bastichelaar 10y agoVamp is awesome! It fits really nice into Kubernetes.
- deleted 10y ago[deleted]
- x0rg 10y agoGreat stuff, happy to see this, especially the part that concerns the setup of the cluster, but still too much in early stages. Currently I'm learning a lot on how to get started on AWS and it is still a bit too painful... after using GKE you just don't want to deal with the manual setup.
- 010a 10y agoYou should check out kops. It seems to me like its one level of abstraction above kubeadm, and makes creating clusters on AWS _ridiculously_ easy (one command).
- DigitalJack 10y agoI'd like to try out kops, it sounded good. But step one was setting up a zone on route53, we don't use route53. Kube-aws from the coreos folks makes the same assumption, but is still useable, however it doesn't have multi-az capability. I love kubernetes and I can't wait for the tooling to mature. I'm about to give 1.4 a spin and see where things stand.
- x0rg 10y agoPlus, kube-aws doeesn't really allow of easy maintenance of the cluster (i.e. cluster updates) as far as I know.
- yissachar 10y agoYou can use Route53 for just a subdomain even if your parent domain is not done through Route53, see https://github.com/kubernetes/kops/blob/master/docs/creating_subdomain.md https://github.com/kubernetes/kops/blob/master/docs/creating...
- x0rg 10y agoI'll definitely do it, too bad for the 50 nodes limitation that can't fit my use case.
- 10y ago
- no_protocol 10y agoI feel that the homepage at kubernetes.io is a poor introduction to the project. All the keywords and short descriptions don't add up to a complete picture of what Kubernetes does and why someone would want to use it. The opening tagline "Kubernetes is an open-source system for..." seems like a complete description of the project, but to a _newcomer_, the sentence is not easy to parse. The "What is Kubernetes?" page [1] gives a very clear overview of Kubernetes that will make sense both to people who already know about containers in general and those who are new to the concept. Can someone help me understand what type of person the homepage is targeted at? It just doesn't do anything for me. I'm mainly interested because I find that a lot of projects have very poor home pages, even if the rest of the project is awesome. [1] http://kubernetes.io/docs/whatisk8s/ http://kubernetes.io/docs/whatisk8s/
- swozey 10y agoI think that it targets people who already know or have an idea as to what Kubernetes is and are looking to compare/contrast various Docker/rkt schedulers for their infrastructure. I really doubt that they're targeting anyone who isn't familiar with the above and with as in flux and complex as kubernetes is right now (especially regarding documentation/the site) they probably shouldn't albeit deployment and management simplification seems to be a very important goal in the end. The kube docs/pages are mostly just from the github repo. I doubt they've had much editorialization unfortunately.
- TheIronYuppie 10y agoI think this is excellent feedback - we should revamp! Disclosure: I work at Google on Kubernetes
- swozey 10y agoOne of my huge headaches in working with Kubernetes daily over the last year is how the documentation is dispersed. The documentation on the site will point you to manfiests in github.com/kubernetes/ which just point you back to the documentation on www.kubernetes.io. If you don't know that https://github.com/kubernetes/kubernetes.github.io/tree/master/docs https://github.com/kubernetes/kubernetes.github.io/tree/mast... exists you're in for a bad time.
- usgroup 10y agoAnyone else lament fleetd and the promise of a simple tool based container orchestration? Yet another complex framework. sigh
- felixgallo 10y agonot particularly! Sometimes problems really are difficult and demand thoughtfully designed solutions, and sometimes complexity in those solutions is both tolerable and worth it.
- puzzle 10y agofleetd doesn't even support rolling updates. It just barely got support for in-place updates. There's a reason why CoreOS folks themselves don't dedicate many cycles to it anymore.
- vidarh 10y agoIt has plenty of flaws, but it also solves problems that aren't well handled with Kubernetes unless I'm missing something, such as being able to schedule arbitrary systemd units, including things like timers, across a cluster. As a low level building block it is/was more flexible in some ways. Of course it is solving only a tiny part of what Kubernetes tries to solve.
- ownagefool 10y agoYeah, sure. Fleet for dnyamic scheduling Consul for k/v service discovery Registrator to actually populate consul Confd to talk to consul so you can configure your containers ... Oh wait I seem to be building a crappy more involved version of kubernetes. p.s. I actually did the above. kubernetes is a better solution.
- invisible 10y agoCan confirm; did similar and kubernetes is a better solution.
- msl09 10y ago"No guys stop complaining that google doesn't give support to kubernetes, we are not Google, we are the Cloud Native Computing Foundation." -- Some kubernetes developer. Release blog of kubernetes brought to you by: -- Aparna Sinha, Product Manager, Google. Seriously though those new features are really neat, and seems to compete with the ease of setup of docker 1.12(though swarm was a bit of a mess when I last tried to make something useful of it (docker 1.12.0)), I hope I can test them at work soon. For now I'm having a great time with rancher.io and cattle.
- TheIronYuppie 10y agoSorry, can you say more about your issue? We've always said we contribute (a ton!) to Kubernetes, but it's definitely not a commercial product, so we don't have a support for on-premises/non-Google Cloud deployments. That said (!!), if paid support is what you're after, please use one of the MANY organizations who DO offer support (http://kubernetes.io/community/ http://kubernetes.io/community/ - about halfway down the page) and are huge contributors to the community. Disclosure: I work at Google on Kubernetes.
- otterley 10y agoThe problem is, if the supporting organization is not the same as the developing organization, there's no obligation for the developing organization to respond to feedback from the supporting organization -- the developers are free to accept or reject feedback as they deem fit. There could even be a conflict of interest: the Kubernetes developers are primarily sponsored by Google, which has an interest in promoting its own cloud offerings over those of their competitors. (Note the relative difficulty of setting up K8S on EC2; even if you can set it up, K8S assumes the ability to create a network topology that's unique to GCE; otherwise you have to use overlay network hacks.)
- TheIronYuppie 10y agoA few items: 1) The people on that page all contribute mightily to the project in various ways - we wouldn't be where we are without them. We take feedback from everyone (we have over 15 special interest groups, many led by non-core team members). 2) K8s developers are most definitely not primarily sponsored by Google - more than 60% of K8s devs are NOT Googlers. 3) Overlay networks are definitely not a hack - many partners set them up with great benefit. The fact is networking is hard (tm), and unless you're just looking for a flat network, then you're going to have to use SOMETHING. Disclosure: I work at Google on Kubernetes
- aseipp 10y agoI've been looking at moving our system to Kubernetes, as it seems mostly cloud agnostic and should work well for our use cases. However, I was put off by the somewhat ad-hoc set of installation methods which mostly seemed to boil down to "Run a big opaque script that does a thing to specifically work with your provider". Really glad to see a dead simple setup - I'm not exactly unfamiliar with operations or containers or Linux or anything, but a turnkey setup so I can play on my DO effortlessly to get an idea is really nice! And I only started looking recently, so having `kubeadm` available now is quite convenient.
- petetnt 10y agoIf you want a completely non-opaque way (or just see the steps in general) of installing Kubernetes, check out Kubernetes The Hard Way [1]. That said, `kubeadm` is a great addition! [1] https://github.com/kelseyhightower/kubernetes-the-hard-way https://github.com/kelseyhightower/kubernetes-the-hard-way
- tazjin 10y agoNote that "Kubernetes The Hard Way" is mostly focused on teaching you what happens during cluster bootstrap. It is NOT a production-ready setup in any form. You should investigate which alternatives exist for your platform (e.g. GKE for Google Cloud, kops for AWS).
- weitzj 10y agoRecently I was at a docker meetup and one speaker really embraced Docker for everything. So his only prequsite was an installed docker daemon. The setup scripts to turn a daemonized docker server into a cluster were all published as a docker image themselves. So installation was mainly a docker run away whereas he used the shell evaluation of a subshell to start the actual docker run, e.g. $(docker run ...imagename) would print the actual docker command which will in turn contain volume mount options to the docker socket to help setting up the whole machine. It was quite fascinating to watch this bootstrap method without relying on any package manager but solely a docker engine deployment. So what I am saying this that this would be an interesting deployment approach for Kubernetes as well.
- gtirloni 10y agoSomething like this? https://coreos.com/blog/self-hosted-kubernetes.html https://coreos.com/blog/self-hosted-kubernetes.html
- weitzj 10y agoThank you
- moondev 10y agoYou can run all the k8s components inside containers with hyperkube
- colemickens 10y agoKubernetes-Anywhere indeed deploys Kubernetes and the only Kubernetes asset is the `hyperkube` container (and even that is just pulled automatically when the node comes up). Makes for fast/easy node provisioning.
- softwarelimits 10y agoWould anybody like to shed some light about how openshift origin relates to K8S in general? I would like to better understand which path to follow. Which advantages has openshift origin over "pure" K8S? Thanks!
- gtirloni 10y agoYou may want to check the section "Kubernetes is not": https://github.com/kubernetes/kubernetes/blob/release-1.4/docs/whatisk8s.md https://github.com/kubernetes/kubernetes/blob/release-1.4/do...
- pat2man 10y agoOpenShift adds a lot of features around building applications from source and authoring those applications as pods in Kubernetes. If you already have a build pipeline set up you may want to stick with pure Kubernetes. They also have an ansible based installation process that makes your cluster a lot more production ready than the basic Kubernetes scripts. For me OpenShift is comparable to Heroku or Elastic Beanstalk where developers can deploy applications from source without knowing a lot about the underlying infrastructure. Of course you still need an ops team to manage OpenShift.
- smarterclayton 10y agoAnother significant scenario is tenancy - if you are just using Kube for a single team / set of apps, most of the security and policy in OpenShift isn't useful for you. But if you want to share access to that cluster, the security and policy and integrated rbac can allow many teams to collaborate on their own applications and self-service. So platform for others vs platform for ops. (I work on OpenShift and Kubernetes)
- SEJeff 10y agoIsn't much / most of that RBAC upstream in k8s 1.3 (code from Openshift written by redhat engineers like yourself)? What is still in Openshift related to AAA that isn't upstream now?
- bogomipz 10y agoI saw the mention of "Curated and pre-tested Helm chart", I am curious if anyone is running databases on Kubernetes?
- ownagefool 10y agoYeah, you mount them on a pvc and they'll get their storage reattached.
- elcct 10y agoI was trying to write a tutorial about how to setup Kubernetes on bare metal. I have created etcd cluster of 5 nodes that checks certificates of clients that communicate with it for security. Sadly I found that at least one command in Kubernetes is either not using provided certificates or I am doing something wrong. I can see that entries get created in etcd, so certificates are definitely correct, but kubectl get cs shows that etcd cluster is not healthy. Given that I see some parts of Kubernetes talk with etcd fine, but I am not able to tell the scope of correctness I am kind of blocked. I don't know whether this is something being worked on, or maybe should I look into it and possibly fix. Tried to ask on Kubernetes Slack, but nobody seemed too interested in this. I am guessing everyone is running etcd without checking certificate validity? Because I was able to run it this way, but that kind of feels wrong. I also wanted to check if problem exists with Kubernetes 1.4, but I couldn't find any migration guides. Maybe I'll just start again... Said article: https://medium.com/@elcct/kubernetes-on-bare-metal-part-5-kubernetes-master-b6a0388fa993 https://medium.com/@elcct/kubernetes-on-bare-metal-part-5-ku... and the ticket: https://github.com/kubernetes/kubernetes/issues/29330 https://github.com/kubernetes/kubernetes/issues/29330
- jacques_chester 10y agoI was under the impression k8s already did these things, but it seems like a lot of these features are just now entering beta.
- kentt 10y agoGreat work. What should I make of that ScheduledJob is in alpha?
- bboreham 10y agoIt means it's new; the API might change before it makes it through beta and into stable.
- deleted 10y ago[deleted]
- nvarsj 10y agoSo you want to be a devops hero? Obtain google borg inspired container scheduling and management? It's a powerful abstraction, and I've enjoyed using K8s over the last year. However, like most freely available software, there is work to be done. Here is my take on the state of things: 1. Still very much in rapid development, with features coming at a breakneck pace. I think Kubernetes is sold as production ready a bit too hard - it takes a good amount of effort to make a cluster production ready for anything non-trivial. Expect to contribute PRs to fix the issues you run in. It's probably easiest to use GCE since this has the most active development, I think. Otherwise, Openshift (Redhat's production fork) for on-premise installation. AWS is in a somewhat working state, with many improvements coming (what I'm using). 2. The maintainers are very open to contributions and discussion. PRs are generally accepted within a couple weeks (and given the volume of issues/PRs this is quite amazing). kubernetes.slack.com is a great way to talk to many of the core developers. I wish they had a slack subscription, so we could search the very rich chat history. A lot of stuff is missed unless you read the chat rooms daily. 3. My impression, which could be totally wrong I admit, is that much of the discussion for big changes seems to happen among the core Google & Redhat engineers offline. I wish the project used an open mailing list for these discussions, or did it in Slack (with a slack subscription, so history is available), or some other recorded text means. I don't think the current mechanisms scale to the size of the project and needs of the users. 3b. A lot of time seems devoted to things like scaling to 1000+ nodes, when fundamental things like kube-proxy are broken for basic use cases, in my opinion (kube-proxy uses an iptables hack for VIPs which leads to problems like the connection tracking table filling up and broken keepalive connections during deployments). 4. The provisioning scripts provided were traditionally pretty poor, kube-addons is a broken shell script mess. Fortunately, this is improving quickly. kops/kubeadm help with the provisioning, coreos is doing lots of work here as well, daemonsets to replace kube-addons, and so on. So this area is improving quickly. But expect to do work here if you're serious about using Kubernetes for prod. I'm using ansible/terraform for this. 5. Ingress (external access to your cluster), is still adhoc. This requires custom dev and testing to setup.
- swozey 10y agoJust a few comments. 1. GKE (Google Container Engine, GCE is compute) lags weeks behind any k8s release so if you're looking for bleeding edge it really isn't the best system, you're better off managing the cluster yourself. GKE being bumped to a recent version seems to be a private matter decided by a few Google employees, so you're never sure at all when you're going to get 1.37 or whatever version that is multiple sem-minors behind. It seems like it's usually about 2-3 weeks but I've never found any discussion or issues about its status so I wind up just checking every few days to see if it's been released. I have a mission critical feature that's being released (I hope) with 1.5 (IP persistence for sockets) and it'd be really nice to be able to follow the decision making process. Or at least have some understanding of what it entails (3 weeks of no major issues? GKE specific bugs? someone comfortable enough to release the hounds?) 5. I've had nothing but headaches with Ingress deployments. I'd wager it's by far the most brought up topic on the k8s slack. The documentation is all over the place and it seems like every example is completely different from the last, albeit there's never any explanation as to why exampleA is different from exampleB but does the same thing. Then you throw in annotations that might be required and IIRC unless you're reading the go docs you wouldn't know anything about them.