5 ms·

Rktnetes brings rkt container engine to Kubernetes

113 points by philips 10y ago

TheIronYuppie 10y ago
We're really excited to broaden customer choice and let people decide what container technology makes sense for their workloads. The CoreOS team has been amazing in helping to ensure Kubernetes is flexible enough to support alternate runtimes starting with not just rkt, but a full scale implementation of all of Kubernetes (running on rkt). Thank you so much! Disclosure: I work at Google on Kubernetes.
- zwischenzug 10y ago
  I am interested in Google's plans for Kubernetes - is the plan to include unikernels in the aegis of Kubernetes' control?
  - TheIronYuppie 10y ago
    Well, two things really: 1) It's definitely not a Google project. CoreOS just gave a great example of that (Redhat, Samsung, Huawei, IBM, Cisco, etc etc etc) 2) We plan on fully supporting OCI and any container technology that supports that interface should run without issue. So, to the extent that a unikernel runtime supported that, then you're good. There are, of course, many items to be sorted out (how do you provide underlying scheduling, how do you handle the functions that kubelet currently provides, how do you provide effective debugging, etc), but I'll leave that to smarter folks than I to work out :) Disclosure: I work at Google on Kubernetes
    - fh973 10y ago
      While CoreOS is not Google you should mention that Google Ventures is the leading investor in CoreOS.
      - TheIronYuppie 10y ago
        That's true, but it is totally and completely separate. While I know Google Venture folks, they do not share anything about investments (performance/strategy/etc) to the main product org.
- gagdad 10y ago
  What about LXC? Is there just no community interest for this feature?
  - colemickens 10y ago
    I would hope/expect to see systemd-nspawn support before LXC directly.
  - themgt 10y ago
    If you've ever looked at the underlying code of "LXC" it's basically a bunch of spaghetti scripts that utilize cgroups. It was awesome for its time, but it's really not something worth supporting in 2016 vs. the new container runtimes.
    - throw2016 10y ago
      This is extremely misinformed. When did you last look at it? Can you post some examples of the spagetti scripts mentioned? What features do the 'new 2016 container run times' have that the LXC project did not provide initially and does not have? Really? It's sad to see this level of dimissive reductive fud being used against the LXC project liberally and consistently especially here on HN to promote other container solutions. A quick visit to linuxcontainers.org or the LXC github repo will quickly and dramatically dispel such notions. The LXC project is written in C and has been in development since 2009 along with cgroup and namespace support in the kernel. It's mature, robust and easy to use. It provides advanced container networking options from veth to macvlan and supports aufs, overlayfs, btrfs, LVM thin, ZFS for snapshots and clones, since at least 2011. It was also the first to support unprivileged containers with its 1.0 release in 2013. It's this work that Docker was based on and its unfortunate the ecosystem far from highlighting the benefits they got many choose to downplay and misrepresent the LXC project leading to comments like the above. LXC is now on 2.0 with multiple improvements from proper display of cgroup controls and container resource usage inside containers with the lxcfs project to initial support for live migration. It is also significantly easier to use than single app containers especially for multiple processes and networking without needing to create custom process managers, run apps in specific ways or network workarounds. LXC runs the container OS's init in the container, Docker and others do not. There are reasons for both approaches. Hope for better informed discussions on Linux containers here.
- otterley 10y ago
  Does this mean k8s will be able to run pods directly via systemd-nspawn? I'm not completely sure what rkt buys us -- please fill me in.
  - robszumski 10y ago
    rkt utilizes systemd-nspawn under the hood, but you need some additional features around that as well. The usage is explained pretty well in the architecture guide: https://coreos.com/rkt/docs/latest/devel/architecture.html https://coreos.com/rkt/docs/latest/devel/architecture.html
jaz46 10y ago
Rkt + Kubernetes = rktnetes! I'd love a little help with the pronunciation? In my head the "t" is silent...rock-a-netes? We're obviously big CoreOS and Kubernetes fans at Pachyderm so great to see more technologies supported in general
- mbreese 10y ago
  rocket-netes?
  - ecnahc515 10y ago
    This is how we say it at CoreOS. Roughly: rocket-net-ease
    - SEJeff 10y ago
      Will it ever be possible to switch between rkt and docker at the pod level instead of at the kubelet level?
- skj 10y ago
  rockaneties is how I said it in my head.
uep 10y ago
Does kubernetes without rkt support pluggable isolation environments? It seems like a pretty cool feature to be able to say "this pod needs to be run under kvm." I'm not well-versed in how good the regular container isolation has become at this point, so maybe it's not as big of an issue these days.
- philips 10y ago
  It is an alpha feature but you can switch the rkt "stage1" using the rkt.alpha.kubernetes.io/stage1-name-override annotation on a pod. So if you wanted to use the virtual machine based pod isolation you could do something like rkt.alpha.kubernetes.io/stage1-name-override=coreos.com/rkt/stage1-kvm:1.10.0. We are working with upstream to come up with a better mechanism but this is a great example of where having an annotation is a great release valve for adding experimental features.
- deleted 10y ago
  [deleted]
- evidencepi 10y ago
  Also there is a node-wide kubelet flag `--rkt-stage1-image1` to set the default isolation environments, which can be overridden by the per pod annotation `rkt.alpha.kubernetes.io/stage1-name-override`
- pas 10y ago
  see also hypernetes: http://blog.kubernetes.io/2016/05/hypernetes-security-and-multi-tenancy-in-kubernetes.html http://blog.kubernetes.io/2016/05/hypernetes-security-and-mu...
lobster_johnson 10y ago
The rkt process model does makes a lot more sense than Docker. The combo of Kubernetes + Docker has always seemed a little awkward to me; K8s just wants to control containers, and something lower-level than Docker would fit its needs better. Looking forward to trying it out!
jdoliner 10y ago
Will it ever be possible to run a heterogeneous cluster with rkt and docker containers running side by side in the same cluster? Or is that a bridge too far?
- philips 10y ago
  This should be possible as it is a kubelet level flag. And the k8s API doesn't care about those details. So, you could have a kubelet join the cluster with docker and a different one join with rkt. That said, I haven't tested it personally. What would your motivation be for doing this?
  - gtaylor 10y ago
    Would certainly make it easier to evaluate and gradually migrate to rkt. I don't have any plans on setting a separate cluster up to test rkt anytime soon.
  - jdoliner 10y ago
    Our motivation would just be supporting both to the end user. I'm imagining a single Kube cluster with Pachyderm running on it and users can specify their jobs as either Docker containers or rkt containers. Ofc this use case is only as real as the users desire to specify their jobs in both formats on the same cluster which I haven't validated. Maybe a more realistic use case is that it would allow us to specify our Pachyderm containers in rkt and still support Docker for users jobs.
- evidencepi 10y ago
  Yes, the container runtime flags are per kubelet wise[1], so you can have a cluster with a mixture of runtimes. I can see the value you mentioned in testing and trying out the rkt+k8s. In fact we did something similar a while ago where we had Docker running in the master node, and rkt running on the minion node to narrow our testing scope. [1] http://kubernetes.io/docs/getting-started-guides/rkt/#spin-up-a-local-kubernetes-cluster-with-the-rkt-runtime http://kubernetes.io/docs/getting-started-guides/rkt/#spin-u...
amitkgupta84 10y ago
What would a user of Kubernetes (as opposed to the person setting up Kubernetes) gain from this? How much of the underlying containerization technology is exposed through the Kubernetes APIs such that one could actually leverage features in rkt that aren't in Docker, or vice versa?
cyphar 10y ago
I'm personally most interested in getting Kubernetes to provide support for the Open Containers Initiative specifications. And it would be really awesome if you could use runC as the executor for Kubernetes. Or at the very least, the rkt executor generated OCI configuration files so you could use other OCI-compliant runtimes.
nikolay 10y ago
This is some terrible branding!

113 points by philips 10y ago

We're really excited to broaden customer choice and let people decide what container technology makes sense for their workloads. The CoreOS team has been amazing in helping to ensure Kubernetes is flexible enough to support alternate runtimes starting with not just rkt, but a full scale implementation of all of Kubernetes (running on rkt). Thank you so much! Disclosure: I work at Google on Kubernetes.

zwischenzug 10y ago
I am interested in Google's plans for Kubernetes - is the plan to include unikernels in the aegis of Kubernetes' control?
- TheIronYuppie 10y ago
  Well, two things really: 1) It's definitely not a Google project. CoreOS just gave a great example of that (Redhat, Samsung, Huawei, IBM, Cisco, etc etc etc) 2) We plan on fully supporting OCI and any container technology that supports that interface should run without issue. So, to the extent that a unikernel runtime supported that, then you're good. There are, of course, many items to be sorted out (how do you provide underlying scheduling, how do you handle the functions that kubelet currently provides, how do you provide effective debugging, etc), but I'll leave that to smarter folks than I to work out :) Disclosure: I work at Google on Kubernetes
  - fh973 10y ago
    While CoreOS is not Google you should mention that Google Ventures is the leading investor in CoreOS.
    - TheIronYuppie 10y ago
      That's true, but it is totally and completely separate. While I know Google Venture folks, they do not share anything about investments (performance/strategy/etc) to the main product org.
gagdad 10y ago
What about LXC? Is there just no community interest for this feature?
- colemickens 10y ago
  I would hope/expect to see systemd-nspawn support before LXC directly.
- themgt 10y ago
  If you've ever looked at the underlying code of "LXC" it's basically a bunch of spaghetti scripts that utilize cgroups. It was awesome for its time, but it's really not something worth supporting in 2016 vs. the new container runtimes.
  - throw2016 10y ago
    This is extremely misinformed. When did you last look at it? Can you post some examples of the spagetti scripts mentioned? What features do the 'new 2016 container run times' have that the LXC project did not provide initially and does not have? Really? It's sad to see this level of dimissive reductive fud being used against the LXC project liberally and consistently especially here on HN to promote other container solutions. A quick visit to linuxcontainers.org or the LXC github repo will quickly and dramatically dispel such notions. The LXC project is written in C and has been in development since 2009 along with cgroup and namespace support in the kernel. It's mature, robust and easy to use. It provides advanced container networking options from veth to macvlan and supports aufs, overlayfs, btrfs, LVM thin, ZFS for snapshots and clones, since at least 2011. It was also the first to support unprivileged containers with its 1.0 release in 2013. It's this work that Docker was based on and its unfortunate the ecosystem far from highlighting the benefits they got many choose to downplay and misrepresent the LXC project leading to comments like the above. LXC is now on 2.0 with multiple improvements from proper display of cgroup controls and container resource usage inside containers with the lxcfs project to initial support for live migration. It is also significantly easier to use than single app containers especially for multiple processes and networking without needing to create custom process managers, run apps in specific ways or network workarounds. LXC runs the container OS's init in the container, Docker and others do not. There are reasons for both approaches. Hope for better informed discussions on Linux containers here.
otterley 10y ago
Does this mean k8s will be able to run pods directly via systemd-nspawn? I'm not completely sure what rkt buys us -- please fill me in.
- robszumski 10y ago
  rkt utilizes systemd-nspawn under the hood, but you need some additional features around that as well. The usage is explained pretty well in the architecture guide: https://coreos.com/rkt/docs/latest/devel/architecture.html https://coreos.com/rkt/docs/latest/devel/architecture.html

jaz46 10y ago

Rkt + Kubernetes = rktnetes! I'd love a little help with the pronunciation? In my head the "t" is silent...rock-a-netes? We're obviously big CoreOS and Kubernetes fans at Pachyderm so great to see more technologies supported in general

mbreese 10y ago
rocket-netes?
- ecnahc515 10y ago
  This is how we say it at CoreOS. Roughly: rocket-net-ease
  - SEJeff 10y ago
    Will it ever be possible to switch between rkt and docker at the pod level instead of at the kubelet level?
skj 10y ago
rockaneties is how I said it in my head.

uep 10y ago

Does kubernetes without rkt support pluggable isolation environments? It seems like a pretty cool feature to be able to say "this pod needs to be run under kvm." I'm not well-versed in how good the regular container isolation has become at this point, so maybe it's not as big of an issue these days.

philips 10y ago
It is an alpha feature but you can switch the rkt "stage1" using the rkt.alpha.kubernetes.io/stage1-name-override annotation on a pod. So if you wanted to use the virtual machine based pod isolation you could do something like rkt.alpha.kubernetes.io/stage1-name-override=coreos.com/rkt/stage1-kvm:1.10.0. We are working with upstream to come up with a better mechanism but this is a great example of where having an annotation is a great release valve for adding experimental features.
deleted 10y ago
[deleted]
evidencepi 10y ago
Also there is a node-wide kubelet flag `--rkt-stage1-image1` to set the default isolation environments, which can be overridden by the per pod annotation `rkt.alpha.kubernetes.io/stage1-name-override`
pas 10y ago
see also hypernetes: http://blog.kubernetes.io/2016/05/hypernetes-security-and-multi-tenancy-in-kubernetes.html http://blog.kubernetes.io/2016/05/hypernetes-security-and-mu...

lobster_johnson 10y ago

The rkt process model does makes a lot more sense than Docker. The combo of Kubernetes + Docker has always seemed a little awkward to me; K8s just wants to control containers, and something lower-level than Docker would fit its needs better. Looking forward to trying it out!

jdoliner 10y ago

Will it ever be possible to run a heterogeneous cluster with rkt and docker containers running side by side in the same cluster? Or is that a bridge too far?

philips 10y ago
This should be possible as it is a kubelet level flag. And the k8s API doesn't care about those details. So, you could have a kubelet join the cluster with docker and a different one join with rkt. That said, I haven't tested it personally. What would your motivation be for doing this?
- gtaylor 10y ago
  Would certainly make it easier to evaluate and gradually migrate to rkt. I don't have any plans on setting a separate cluster up to test rkt anytime soon.
- jdoliner 10y ago
  Our motivation would just be supporting both to the end user. I'm imagining a single Kube cluster with Pachyderm running on it and users can specify their jobs as either Docker containers or rkt containers. Ofc this use case is only as real as the users desire to specify their jobs in both formats on the same cluster which I haven't validated. Maybe a more realistic use case is that it would allow us to specify our Pachyderm containers in rkt and still support Docker for users jobs.
evidencepi 10y ago
Yes, the container runtime flags are per kubelet wise[1], so you can have a cluster with a mixture of runtimes. I can see the value you mentioned in testing and trying out the rkt+k8s. In fact we did something similar a while ago where we had Docker running in the master node, and rkt running on the minion node to narrow our testing scope. [1] http://kubernetes.io/docs/getting-started-guides/rkt/#spin-up-a-local-kubernetes-cluster-with-the-rkt-runtime http://kubernetes.io/docs/getting-started-guides/rkt/#spin-u...

amitkgupta84 10y ago

What would a user of Kubernetes (as opposed to the person setting up Kubernetes) gain from this? How much of the underlying containerization technology is exposed through the Kubernetes APIs such that one could actually leverage features in rkt that aren't in Docker, or vice versa?

cyphar 10y ago

I'm personally most interested in getting Kubernetes to provide support for the Open Containers Initiative specifications. And it would be really awesome if you could use runC as the executor for Kubernetes. Or at the very least, the rkt executor generated OCI configuration files so you could use other OCI-compliant runtimes.

nikolay 10y ago

This is some terrible branding!