10 ms·

Kubernetes 1.5.0 Released

226 points by tylermauthe 10y ago

jstoja 10y ago
Has anyone already successfully created a HA-cluster on a bare metal infrastructure? It seems overly complicated and not that well documented from what I can see.
- hnarayanan 10y ago
  I have attempted this using https://coreos.com/kubernetes/docs/latest/kubernetes-on-baremetal.html https://coreos.com/kubernetes/docs/latest/kubernetes-on-bare... . One can get quite far, but I am not sure it is worth the effort. I understand this is a vague analogy, but it feels like you're trying to setup email infrastructure when what you really want is to send email. It is a lot easier on the public cloud, or easier still on a managed service.
  - iagooar 10y ago
    What about people who cannot go to the cloud? What about people who need more performance? Lower costs? Kubernetes seems to be such a great abstraction for the underlying hardware, why not use it where it is needed the most?
    - jdoliner 10y ago
      People do run Kubernetes on bare metal... but there's no denying it takes effort. That effort should be coming down over time, but unfortunately there's still a ways to go.
      - AlexB138 10y ago
        That's not entirely true. I attend meet-ups with engineers from a fairly large company whose name you would recognize who run K8s on bare metal. I know that's vague, but I don't know how public their infrastructure is, so I'd rather not speak out of turn. You're absolutely right that it takes some effort.
        SEJeff 10y ago
        Bloomberg runs Kubernetes on bare metal and one of my friends is an engineer on the project (but did not speak at KubeCon about it). They've got a relatively sizable install as well.
    - nixgeek 10y ago
      citation required, since there are IaaS offerings that do not involve a hypervisor (e.g. Rackspace OnMetal) and therefore its possible to move "to the cloud" without performance penalties. on costs via Reserved Instances and negotiating with your account team at larger volume, it can be pretty reasonable, and for a smaller outfit, substantially cheaper than hiring experts in running physical infrastructure of your own.
      - manigandham 10y ago
        Citation needed for what? its common knowledge that the major clouds VMs with lower performance and higher cost in exchange for flexibility.
      - vidarh 10y ago
        > and for a smaller outfit, substantially cheaper than hiring experts in running physical infrastructure of your own. You needs to have the skillsets to manage cloud deployments too. For the systems I manage, which range from actual bare metal, via dedicated servers, to VPSs and AWS deployments, the incremental effort spent on managing hardware as you go down towards bare metal tends to be pretty much a rounding error compard to the overall operations effort. Once the systems have been wired up, and PXE booted into a suitable setup, the effort is pretty much the same. And with the cost differential, I'd say once you go over a few hundred dollars a month on servers that stay up 24/7, you're losing money on public cloud deployments vs. managed dedicated hosting. Once you go over $1k to $2k/month, you're losing money vs. colo. For people who actually have a lot of batch jobs where servers stay up for less than 6-8 hours a day, the maths look different, but it's very rare I come across cloud setups that are cheaper than dedicated, with all staff costs etc. accounted for.
      - iagooar 10y ago
        citation required for what? For stating, that it is complicated to install on bare-metal and that it's been a priority for Kubernetes to be deploying on cloud servers?
  - jabzd 10y ago
    I felt very similar. Especially thinking about it from a disaster recovery/re-bootstrapping perspective. That scared me into using docker swarm kit which is stupidly easy to bootstrap. Keep in mind that's not necessarily a like for like replacement but it was enough for my use case. I've been keeping an eye out waiting to feel more comfortable to use and support a kube cluster on bare metal.
  - philips 10y ago
    What would your ideal setup look like on baremetal? The Kubernetes on baremetal setup above is for full automation. You can do simpler manual installs with bootkube[1] or kubeadm[2]; and this will further simplifying over the next few months. [1] https://github.com/kubernetes-incubator/bootkube https://github.com/kubernetes-incubator/bootkube [2] http://kubernetes.io/docs/getting-started-guides/kubeadm/ http://kubernetes.io/docs/getting-started-guides/kubeadm/
- vlad00 10y ago
  I've deployed kubernetes on aws as if it were bare metal. Try to run through https://github.com/kelseyhightower/kubernetes-the-hard-way https://github.com/kelseyhightower/kubernetes-the-hard-way translating what they do there to your particular setup(configure firewall, add routes to router, etc)
  - amatix 10y ago
    I'd suggest looking into kops (https://github.com/kubernetes/kops https://github.com/kubernetes/kops) for Kubernetes on AWS, whether you let it take the actions or you tweak/use the Terraform config it can generate
- appelza 10y ago
  Yeah, quite easily. Look at Openshift, ansible playbook installer for Fedora 24. You can configure 2 masters/ha/etc in the playbook.
- olavgg 10y ago
  Could you use Canonical MAAS (https://www.ubuntu.com/server/maas https://www.ubuntu.com/server/maas) for this?
  - jcastro 10y ago
    Indeed, we fully support Kubernetes on MAAS on Ubuntu. Currently for 1.4 but we'll be updating support for 1.5 over the next week or so. Disclaimer: I'm on the team that works on this.
- ownagefool 10y ago
  Yep. We PXE boot coreos and include a cloud-config that bootstraps a kubelet and pulls master manifests. It's actually not very complicated at all.
- sandGorgon 10y ago
  +1 there has been a discussion of making a sig-on-premise/sig-bare-metal (with a fair bit of opposition from others) [1]. Would help to have your comments there as well. The biggest complication is around Ingress and load balancers. https://github.com/kubernetes/ingress/issues/23 https://github.com/kubernetes/ingress/issues/23 https://github.com/kubernetes/ingress/issues/17 https://github.com/kubernetes/ingress/issues/17 The problem is that many people point towards kube-the-hard way.. which is NOT production ready. For example scaling ingress itself - do you do deployments versus daemonset? How do you set up ingresses to pass through source ip. a lot of this is taken care for the cloud...but NOT for bare metal. [1] https://groups.google.com/forum/m/?utm_medium=email&utm_source=footer#!msg/kubernetes-dev/ztVnvbrpTK8/-R4PlfHVBwAJ https://groups.google.com/forum/m/?utm_medium=email&utm_sour...
  - jcastro 10y ago
    sig-onprem is coming, it was decided at the last community meeting, expect an announcement soonish.
  - elcct 10y ago
    It also has problems with using certificates when reporting etcd health: https://github.com/kubernetes/kubernetes/issues/27343 https://github.com/kubernetes/kubernetes/issues/27343 I was writing a blog about installing it on bare metal, but this issue got me blocked.
    - sandGorgon 10y ago
      could you ping on slack on sig-cluster-ops? thats the defacto meeting ground for bare metal issues. Though honestly i think it is a fundamental issue and not specifically related to BM.
- tyingq 10y ago
  It won't help you understand how to install it, but if you just want to get it done, see http://stackpoint.io http://stackpoint.io They support bare metal providers, like Packet, but they also support inexpensive platforms, like Digital Ocean. Really handy if your goal to experiment with an up and running cluster and don't want the hassle of installation.
  - irontoby 10y ago
    Stackpoint user here; running a very small (3 node) cluster on DigitalOcean currently using their service. It's been relatively painless so far, but I'm pretty new to k8s so my needs are very basic.
- hanikesn 10y ago
  We're using Puppet to deploy our kubernetes cluster on bare-metal as well as on AWS. Achieving HA isn't necessarily complicated. Etcd supports clustering by default and the master components have master election builtin. We also use haproxy locally on every machine to loadbalance between the different api servers. So we don't need a central LB. Setting up HA was only a small part of our overall effort. Making things robust and figuring out the small details are a lot harder.
- fen4o 10y ago
  PXE CoreOS and cloud-config to run kubelet and pull manifests. [0] [0] https://youtu.be/4gyeixJLabo?t=814 https://youtu.be/4gyeixJLabo?t=814
- old-gregg 10y ago
  Regarding HA-on-bare metal, we do this for our customers here: http://gravitational.com/managed-kubernetes/ http://gravitational.com/managed-kubernetes/ In our experience setting Kubernetes itself up for the first time is not that hard, the difficulties come from the fact that k8s is a fast moving target with quickly evolving "best practices". Also, companies are struggling a bit integrating k8s-centric workflow with existing applications, particularly data stores.
- jwelch55 10y ago
  I'd suggest checking out KET (Kismatic Enterprise Toolkit) - https://github.com/apprenda/kismatic https://github.com/apprenda/kismatic There will be a 1.5 compatible release coming soon. Disclosure - I do work with the fine folks who made this. But I have had the opportunity to use it myself, and have successfully brought up some small clusters to play around with. I'd say it's certainly worth a look.
- TheIronYuppie 10y ago
  Tons of great suggestions in the thread, but let me also point you to the kubeadm work - in which in Kubernetes 1.6, folks are trying to make it easier to install HA setups. Please join up at the SIG to help out! https://github.com/orgs/kubernetes/teams/sig-cluster-lifecycle https://github.com/orgs/kubernetes/teams/sig-cluster-lifecyc... Disclosure: I work at Google on Kubernetes.
- robhirschfeld 10y ago
  Yes. We've been doing this w/ Digital Rebar. There are additional steps (like creating load balancer automation) that we are planning to add. FWIW, HA is complicated in general, K8s HA is actually reasonable by comparison. Other tasks, like upgrade and security are harder. We're working on docs in the SIG Cluster Ops and you're welcome to join in discuss with us there.
rcarmo 10y ago
Well, that fixes the problem I had with not having enough time to test 1.4... How's the documentation looking?
samuell 10y ago
Folks wanting an easy road to deploying Kube on (so far) AWS, GCE and OpenStack, with bare metal and local on the roadmap, might want to check out KubeNow: https://github.com/kubenow/KubeNow https://github.com/kubenow/KubeNow Tries to re-use as much as possible from great projects like Terraform, Packer, Ansible and the kubeadm tool, and just add a thin layer on top of that (less risk for bit rot), which is an approach that seems appealing to me.
- machbio 10y ago
  Does KubeNow support AWS Spot Instances ?
  - samuell 10y ago
    Good question ... I did only try the GCE version (colleagues tried AWS etc). Open an issue and ask Marco! :)
  - TheIronYuppie 10y ago
    Not sure on AWS Spot, but on Google Cloud you can use Preemptible VMs (which have similar 50-80% discounts). https://cloud.google.com/container-engine/docs/preemptible-vm https://cloud.google.com/container-engine/docs/preemptible-v... Disclosure: I work at Google on Kubernetes.
  - mcapuccini 10y ago
    Not yet, but you just gave me a good idea for future development!
pyvpx 10y ago
you'd think such a project so heavily supported by Google would have IPv6 support. Hell, you'd think it'd be native/first-class.
- dankohn1 10y ago
  Yes, it's a known issue. Pull requests welcome. https://github.com/kubernetes/kubernetes/issues/1443#issuecomment-264605379 https://github.com/kubernetes/kubernetes/issues/1443#issueco...
  - TheIronYuppie 10y ago
    Yes, please! It's a pretty deep problem, so we'd love the help. Disclosure: I work at Google on Kubernetes.
- runeks 10y ago
  Isn't IPv6 mostly for internet traffic, where 32 bits of address space isn't enough? Why not just route IP6 traffic to a Kubernetes cluster, and have that cluster use IPv4 internally? Is there something more to IPv6 than the increased address space that I'm missing?
  - detaro 10y ago
    Then you need gateways that keep track of and translate every connection, and special ways to tell your backends the client IP (e.g. for logging, fraud detection, ...). Dealing with IPv4 to clients with an IPv6 backend is easier than the other way around. This recent discussion had more comments on that: https://news.ycombinator.com/item?id=13085941 https://news.ycombinator.com/item?id=13085941
jcastro 10y ago
If anyone is interested in giving k8s a go we've been working on fully supported Kubernetes for Ubuntu: https://www.ubuntu.com/cloud/kubernetes https://www.ubuntu.com/cloud/kubernetes This works on GCE, AWS, Azure, VMWare, bare metal (via MAAS) and LXD containers. We're currently supporting 1.4 but will support 1.5 (and upgrades) in the next week or so. Disclaimer: I'm on the team that works on this.
- elcct 10y ago
  How do you deal with storage on bare metal?
  - marcoceppi 10y ago
    Typically ceph for rbd
  - lazypower 10y ago
    As Marco alluded, our only supported storage mechanism to date (Which is represented and managed via the charm) is Ceph backed by RBD storage. We have plans on including other storage vendors and mechanisms, but they aren't on the roadmap for the very near term. If end users start requesting a specific storage solution, it would go in our planning doc and get added to the roadmap. We're quite active with our early adopters that give us feedback and file bugs/requests. To date you could continue to use alternative storage providers such as NFS or Gluster - but we don't have the PV creation+enlistment captured in the charm code just yet, but again, due to priorities. End users pretty much set the priorities for us, and we then circle back with some light weight planning and execution. Hope this helps!
- SEJeff 10y ago
  What does Canonical work on in upstream Kubernetes? I see hundreds+ of commits from Redhat and CoreOS in Kubernetes and the associated projects? In the main kubernetes git repo, I see 39 non-merge from @canonical commits, 99% of which are under cluster/juju/* and 1 doc fix for juju, so not really a value add to Kubernetes at all. How can I, as someone deploying kubernetes on bare metal, trust you to manage a difficult project when Canonical doesn't seem to contribute much at all to the project or ecosystem? It is great that you're making kubernetes easy to manage and deploy, but other than ease of use (which GKE does an excellent job of as well from the authors of k8s), what is Canonical's actual value add for paying you to manage k8s? Sorry for the terseness, but I'm genuinely curious what value add there is here. As a purely tech focused person, I simply don't see it. Even other firms doing exactly what you do (Apcera) have actual code change commits. I don't see a single one from Canonical that does.
iamdeedubs 10y ago
k8s-the-hard-way explicitly says its not production ready. Is there a document that explains what its missing or production topology documented somewhere?
- lazypower 10y ago
  I don't think its documented in any official capacity, but we (sig cluster ops) did generate some visuals that might aid in grokking the topology of Kubernetes as a whole, and we did model this after production-setups. A few things to keep in mind: These maps are service centric, and abstract units as vertical columns in their respective diagrams. Services must be HA to be considered “production ready” Additional concerns that may/may-not be represented here: - TLS Security on all endpoints - TLS Key Rotation in the event of compromise/upgrade/expiration - Durable storage backed workloads - ETCD state snapshots for cluster point-in-time recovery - User/RBAC - this still needs more info before i can outline it (time limited) - Network policy for namespace/application isolation (this is an unspoken requirement for many business units) The diagrams: Kubernetes cluster services https://docs.google.com/drawings/d/1U4GBSg9Sdn7JspoxDyA4qwGMJC9JelWiev5MwYG9zwg/edit https://docs.google.com/drawings/d/1U4GBSg9Sdn7JspoxDyA4qwGM... Kubernetes Binary Services Topology map https://docs.google.com/drawings/d/10sXtgdelUI3GbWjrYh2z5vhFahdqcYqQtOu9fs2dPkk/edit https://docs.google.com/drawings/d/10sXtgdelUI3GbWjrYh2z5vhF... Kubernetes Cluster node Maps (3) https://docs.google.com/drawings/d/1x1PEE0RKvCRnP5JCAjmfbr_7NTpjBvOo04pvXbgztqM/edit https://docs.google.com/drawings/d/1x1PEE0RKvCRnP5JCAjmfbr_7... We left off working on a Network draft diagram, and if you’re interested in contributing/participating in this process, join us in the #sig-cluster-ops slack channel. We meet thursdays (or have, new year schedule dependent)
  - Terretta 10y ago
    Missed these in Slack SIG channel, thanks!
shaklee3 10y ago
Looks like PetSets has been renamed StatefulSets and has moved to beta in this release.
awinter-py 10y ago
kubefed autocorrects to 'cubefield' in G search
chrisgaun 10y ago
DotNETes (Windows container and Hyper-v container) support is here! Disclosure: I worked on DotNETes
- iso-8859-1 10y ago
  Do you have an example application?
boyd 10y ago
One note of caution for those skimming the release notes: v1.5.0 now allows anonymous access to the API server by default. With the default k8s authorization being `AllowAll` this makes it very important to pass `--allow-anonymous=false` if updating from v1.4.x to v1.5.0. It looks like this is going to be set to false for v1.5.1: https://github.com/kubernetes/kubernetes/pull/38708 https://github.com/kubernetes/kubernetes/pull/38708
- TheIronYuppie 10y ago
  This is correct, and has been fixed. If you use kube-up or Google Container Engine, the default is set to false. Disclosure: I work at Google on Kubernetes.

226 points by tylermauthe 10y ago

Has anyone already successfully created a HA-cluster on a bare metal infrastructure? It seems overly complicated and not that well documented from what I can see.

hnarayanan 10y ago
I have attempted this using https://coreos.com/kubernetes/docs/latest/kubernetes-on-baremetal.html https://coreos.com/kubernetes/docs/latest/kubernetes-on-bare... . One can get quite far, but I am not sure it is worth the effort. I understand this is a vague analogy, but it feels like you're trying to setup email infrastructure when what you really want is to send email. It is a lot easier on the public cloud, or easier still on a managed service.
- iagooar 10y ago
  What about people who cannot go to the cloud? What about people who need more performance? Lower costs? Kubernetes seems to be such a great abstraction for the underlying hardware, why not use it where it is needed the most?
  - jdoliner 10y ago
    People do run Kubernetes on bare metal... but there's no denying it takes effort. That effort should be coming down over time, but unfortunately there's still a ways to go.
    - AlexB138 10y ago
      That's not entirely true. I attend meet-ups with engineers from a fairly large company whose name you would recognize who run K8s on bare metal. I know that's vague, but I don't know how public their infrastructure is, so I'd rather not speak out of turn. You're absolutely right that it takes some effort.
      - SEJeff 10y ago
        Bloomberg runs Kubernetes on bare metal and one of my friends is an engineer on the project (but did not speak at KubeCon about it). They've got a relatively sizable install as well.
  - nixgeek 10y ago
    citation required, since there are IaaS offerings that do not involve a hypervisor (e.g. Rackspace OnMetal) and therefore its possible to move "to the cloud" without performance penalties. on costs via Reserved Instances and negotiating with your account team at larger volume, it can be pretty reasonable, and for a smaller outfit, substantially cheaper than hiring experts in running physical infrastructure of your own.
    - manigandham 10y ago
      Citation needed for what? its common knowledge that the major clouds VMs with lower performance and higher cost in exchange for flexibility.
    - vidarh 10y ago
      > and for a smaller outfit, substantially cheaper than hiring experts in running physical infrastructure of your own. You needs to have the skillsets to manage cloud deployments too. For the systems I manage, which range from actual bare metal, via dedicated servers, to VPSs and AWS deployments, the incremental effort spent on managing hardware as you go down towards bare metal tends to be pretty much a rounding error compard to the overall operations effort. Once the systems have been wired up, and PXE booted into a suitable setup, the effort is pretty much the same. And with the cost differential, I'd say once you go over a few hundred dollars a month on servers that stay up 24/7, you're losing money on public cloud deployments vs. managed dedicated hosting. Once you go over $1k to $2k/month, you're losing money vs. colo. For people who actually have a lot of batch jobs where servers stay up for less than 6-8 hours a day, the maths look different, but it's very rare I come across cloud setups that are cheaper than dedicated, with all staff costs etc. accounted for.
    - iagooar 10y ago
      citation required for what? For stating, that it is complicated to install on bare-metal and that it's been a priority for Kubernetes to be deploying on cloud servers?
- jabzd 10y ago
  I felt very similar. Especially thinking about it from a disaster recovery/re-bootstrapping perspective. That scared me into using docker swarm kit which is stupidly easy to bootstrap. Keep in mind that's not necessarily a like for like replacement but it was enough for my use case. I've been keeping an eye out waiting to feel more comfortable to use and support a kube cluster on bare metal.
- philips 10y ago
  What would your ideal setup look like on baremetal? The Kubernetes on baremetal setup above is for full automation. You can do simpler manual installs with bootkube[1] or kubeadm[2]; and this will further simplifying over the next few months. [1] https://github.com/kubernetes-incubator/bootkube https://github.com/kubernetes-incubator/bootkube [2] http://kubernetes.io/docs/getting-started-guides/kubeadm/ http://kubernetes.io/docs/getting-started-guides/kubeadm/
vlad00 10y ago
I've deployed kubernetes on aws as if it were bare metal. Try to run through https://github.com/kelseyhightower/kubernetes-the-hard-way https://github.com/kelseyhightower/kubernetes-the-hard-way translating what they do there to your particular setup(configure firewall, add routes to router, etc)
- amatix 10y ago
  I'd suggest looking into kops (https://github.com/kubernetes/kops https://github.com/kubernetes/kops) for Kubernetes on AWS, whether you let it take the actions or you tweak/use the Terraform config it can generate
appelza 10y ago
Yeah, quite easily. Look at Openshift, ansible playbook installer for Fedora 24. You can configure 2 masters/ha/etc in the playbook.
olavgg 10y ago
Could you use Canonical MAAS (https://www.ubuntu.com/server/maas https://www.ubuntu.com/server/maas) for this?
- jcastro 10y ago
  Indeed, we fully support Kubernetes on MAAS on Ubuntu. Currently for 1.4 but we'll be updating support for 1.5 over the next week or so. Disclaimer: I'm on the team that works on this.
ownagefool 10y ago
Yep. We PXE boot coreos and include a cloud-config that bootstraps a kubelet and pulls master manifests. It's actually not very complicated at all.
sandGorgon 10y ago
+1 there has been a discussion of making a sig-on-premise/sig-bare-metal (with a fair bit of opposition from others) [1]. Would help to have your comments there as well. The biggest complication is around Ingress and load balancers. https://github.com/kubernetes/ingress/issues/23 https://github.com/kubernetes/ingress/issues/23 https://github.com/kubernetes/ingress/issues/17 https://github.com/kubernetes/ingress/issues/17 The problem is that many people point towards kube-the-hard way.. which is NOT production ready. For example scaling ingress itself - do you do deployments versus daemonset? How do you set up ingresses to pass through source ip. a lot of this is taken care for the cloud...but NOT for bare metal. [1] https://groups.google.com/forum/m/?utm_medium=email&utm_source=footer#!msg/kubernetes-dev/ztVnvbrpTK8/-R4PlfHVBwAJ https://groups.google.com/forum/m/?utm_medium=email&utm_sour...
- jcastro 10y ago
  sig-onprem is coming, it was decided at the last community meeting, expect an announcement soonish.
- elcct 10y ago
  It also has problems with using certificates when reporting etcd health: https://github.com/kubernetes/kubernetes/issues/27343 https://github.com/kubernetes/kubernetes/issues/27343 I was writing a blog about installing it on bare metal, but this issue got me blocked.
  - sandGorgon 10y ago
    could you ping on slack on sig-cluster-ops? thats the defacto meeting ground for bare metal issues. Though honestly i think it is a fundamental issue and not specifically related to BM.
tyingq 10y ago
It won't help you understand how to install it, but if you just want to get it done, see http://stackpoint.io http://stackpoint.io They support bare metal providers, like Packet, but they also support inexpensive platforms, like Digital Ocean. Really handy if your goal to experiment with an up and running cluster and don't want the hassle of installation.
- irontoby 10y ago
  Stackpoint user here; running a very small (3 node) cluster on DigitalOcean currently using their service. It's been relatively painless so far, but I'm pretty new to k8s so my needs are very basic.
hanikesn 10y ago
We're using Puppet to deploy our kubernetes cluster on bare-metal as well as on AWS. Achieving HA isn't necessarily complicated. Etcd supports clustering by default and the master components have master election builtin. We also use haproxy locally on every machine to loadbalance between the different api servers. So we don't need a central LB. Setting up HA was only a small part of our overall effort. Making things robust and figuring out the small details are a lot harder.
fen4o 10y ago
PXE CoreOS and cloud-config to run kubelet and pull manifests. [0] [0] https://youtu.be/4gyeixJLabo?t=814 https://youtu.be/4gyeixJLabo?t=814
old-gregg 10y ago
Regarding HA-on-bare metal, we do this for our customers here: http://gravitational.com/managed-kubernetes/ http://gravitational.com/managed-kubernetes/ In our experience setting Kubernetes itself up for the first time is not that hard, the difficulties come from the fact that k8s is a fast moving target with quickly evolving "best practices". Also, companies are struggling a bit integrating k8s-centric workflow with existing applications, particularly data stores.
jwelch55 10y ago
I'd suggest checking out KET (Kismatic Enterprise Toolkit) - https://github.com/apprenda/kismatic https://github.com/apprenda/kismatic There will be a 1.5 compatible release coming soon. Disclosure - I do work with the fine folks who made this. But I have had the opportunity to use it myself, and have successfully brought up some small clusters to play around with. I'd say it's certainly worth a look.
TheIronYuppie 10y ago
Tons of great suggestions in the thread, but let me also point you to the kubeadm work - in which in Kubernetes 1.6, folks are trying to make it easier to install HA setups. Please join up at the SIG to help out! https://github.com/orgs/kubernetes/teams/sig-cluster-lifecycle https://github.com/orgs/kubernetes/teams/sig-cluster-lifecyc... Disclosure: I work at Google on Kubernetes.
robhirschfeld 10y ago
Yes. We've been doing this w/ Digital Rebar. There are additional steps (like creating load balancer automation) that we are planning to add. FWIW, HA is complicated in general, K8s HA is actually reasonable by comparison. Other tasks, like upgrade and security are harder. We're working on docs in the SIG Cluster Ops and you're welcome to join in discuss with us there.

rcarmo 10y ago

Well, that fixes the problem I had with not having enough time to test 1.4... How's the documentation looking?

samuell 10y ago

Folks wanting an easy road to deploying Kube on (so far) AWS, GCE and OpenStack, with bare metal and local on the roadmap, might want to check out KubeNow: https://github.com/kubenow/KubeNow https://github.com/kubenow/KubeNow Tries to re-use as much as possible from great projects like Terraform, Packer, Ansible and the kubeadm tool, and just add a thin layer on top of that (less risk for bit rot), which is an approach that seems appealing to me.

machbio 10y ago
Does KubeNow support AWS Spot Instances ?
- samuell 10y ago
  Good question ... I did only try the GCE version (colleagues tried AWS etc). Open an issue and ask Marco! :)
- TheIronYuppie 10y ago
  Not sure on AWS Spot, but on Google Cloud you can use Preemptible VMs (which have similar 50-80% discounts). https://cloud.google.com/container-engine/docs/preemptible-vm https://cloud.google.com/container-engine/docs/preemptible-v... Disclosure: I work at Google on Kubernetes.
- mcapuccini 10y ago
  Not yet, but you just gave me a good idea for future development!

pyvpx 10y ago

you'd think such a project so heavily supported by Google would have IPv6 support. Hell, you'd think it'd be native/first-class.

dankohn1 10y ago
Yes, it's a known issue. Pull requests welcome. https://github.com/kubernetes/kubernetes/issues/1443#issuecomment-264605379 https://github.com/kubernetes/kubernetes/issues/1443#issueco...
- TheIronYuppie 10y ago
  Yes, please! It's a pretty deep problem, so we'd love the help. Disclosure: I work at Google on Kubernetes.
runeks 10y ago
Isn't IPv6 mostly for internet traffic, where 32 bits of address space isn't enough? Why not just route IP6 traffic to a Kubernetes cluster, and have that cluster use IPv4 internally? Is there something more to IPv6 than the increased address space that I'm missing?
- detaro 10y ago
  Then you need gateways that keep track of and translate every connection, and special ways to tell your backends the client IP (e.g. for logging, fraud detection, ...). Dealing with IPv4 to clients with an IPv6 backend is easier than the other way around. This recent discussion had more comments on that: https://news.ycombinator.com/item?id=13085941 https://news.ycombinator.com/item?id=13085941

jcastro 10y ago

If anyone is interested in giving k8s a go we've been working on fully supported Kubernetes for Ubuntu: https://www.ubuntu.com/cloud/kubernetes https://www.ubuntu.com/cloud/kubernetes This works on GCE, AWS, Azure, VMWare, bare metal (via MAAS) and LXD containers. We're currently supporting 1.4 but will support 1.5 (and upgrades) in the next week or so. Disclaimer: I'm on the team that works on this.

elcct 10y ago
How do you deal with storage on bare metal?
- marcoceppi 10y ago
  Typically ceph for rbd
- lazypower 10y ago
  As Marco alluded, our only supported storage mechanism to date (Which is represented and managed via the charm) is Ceph backed by RBD storage. We have plans on including other storage vendors and mechanisms, but they aren't on the roadmap for the very near term. If end users start requesting a specific storage solution, it would go in our planning doc and get added to the roadmap. We're quite active with our early adopters that give us feedback and file bugs/requests. To date you could continue to use alternative storage providers such as NFS or Gluster - but we don't have the PV creation+enlistment captured in the charm code just yet, but again, due to priorities. End users pretty much set the priorities for us, and we then circle back with some light weight planning and execution. Hope this helps!
SEJeff 10y ago
What does Canonical work on in upstream Kubernetes? I see hundreds+ of commits from Redhat and CoreOS in Kubernetes and the associated projects? In the main kubernetes git repo, I see 39 non-merge from @canonical commits, 99% of which are under cluster/juju/* and 1 doc fix for juju, so not really a value add to Kubernetes at all. How can I, as someone deploying kubernetes on bare metal, trust you to manage a difficult project when Canonical doesn't seem to contribute much at all to the project or ecosystem? It is great that you're making kubernetes easy to manage and deploy, but other than ease of use (which GKE does an excellent job of as well from the authors of k8s), what is Canonical's actual value add for paying you to manage k8s? Sorry for the terseness, but I'm genuinely curious what value add there is here. As a purely tech focused person, I simply don't see it. Even other firms doing exactly what you do (Apcera) have actual code change commits. I don't see a single one from Canonical that does.

iamdeedubs 10y ago

k8s-the-hard-way explicitly says its not production ready. Is there a document that explains what its missing or production topology documented somewhere?

lazypower 10y ago
I don't think its documented in any official capacity, but we (sig cluster ops) did generate some visuals that might aid in grokking the topology of Kubernetes as a whole, and we did model this after production-setups. A few things to keep in mind: These maps are service centric, and abstract units as vertical columns in their respective diagrams. Services must be HA to be considered “production ready” Additional concerns that may/may-not be represented here: - TLS Security on all endpoints - TLS Key Rotation in the event of compromise/upgrade/expiration - Durable storage backed workloads - ETCD state snapshots for cluster point-in-time recovery - User/RBAC - this still needs more info before i can outline it (time limited) - Network policy for namespace/application isolation (this is an unspoken requirement for many business units) The diagrams: Kubernetes cluster services https://docs.google.com/drawings/d/1U4GBSg9Sdn7JspoxDyA4qwGMJC9JelWiev5MwYG9zwg/edit https://docs.google.com/drawings/d/1U4GBSg9Sdn7JspoxDyA4qwGM... Kubernetes Binary Services Topology map https://docs.google.com/drawings/d/10sXtgdelUI3GbWjrYh2z5vhFahdqcYqQtOu9fs2dPkk/edit https://docs.google.com/drawings/d/10sXtgdelUI3GbWjrYh2z5vhF... Kubernetes Cluster node Maps (3) https://docs.google.com/drawings/d/1x1PEE0RKvCRnP5JCAjmfbr_7NTpjBvOo04pvXbgztqM/edit https://docs.google.com/drawings/d/1x1PEE0RKvCRnP5JCAjmfbr_7... We left off working on a Network draft diagram, and if you’re interested in contributing/participating in this process, join us in the #sig-cluster-ops slack channel. We meet thursdays (or have, new year schedule dependent)
- Terretta 10y ago
  Missed these in Slack SIG channel, thanks!

shaklee3 10y ago

Looks like PetSets has been renamed StatefulSets and has moved to beta in this release.

awinter-py 10y ago

kubefed autocorrects to 'cubefield' in G search

chrisgaun 10y ago

DotNETes (Windows container and Hyper-v container) support is here! Disclosure: I worked on DotNETes

iso-8859-1 10y ago
Do you have an example application?

boyd 10y ago

One note of caution for those skimming the release notes: v1.5.0 now allows anonymous access to the API server by default. With the default k8s authorization being `AllowAll` this makes it very important to pass `--allow-anonymous=false` if updating from v1.4.x to v1.5.0. It looks like this is going to be set to false for v1.5.1: https://github.com/kubernetes/kubernetes/pull/38708 https://github.com/kubernetes/kubernetes/pull/38708

TheIronYuppie 10y ago
This is correct, and has been fixed. If you use kube-up or Google Container Engine, the default is set to false. Disclosure: I work at Google on Kubernetes.