6 ms·

Container Runtime Interface (CRI) in Kubernetes

124 points by philips 10y ago

vidarh 10y ago
I can't get over how much of a smell the requirement of an RPC interface in order to interface with tools that may or may not have any good reason to be running on an ongoing basis is to me. I'm sure there may be cases where you interact with the containers frequently enough that spawning a process each time is actually a worthwhile optimization, but more and more of these containerisation systems are becoming an unholy mess of daemons that needs to run in order to run and manage containers that need not depend on anything but the host init/systemd. E.g. one of the really appealing things of rkt for me is the simplicity - depending on the level of isolation everything is running either direclty under systemd, or under an individual isolator like systemd-nspawn. I disliked this tendency towards a herd of daemons intensely when Docker continued as it started and used HTTP for volume/network plugins, and I dislike it just as much now. It's as if someone sat down and thought long and hard about how to add more complexity and more "fun" failure modes.
- wmf 10y ago
  It's the microservice philosophy: Why use a function call or fork/exec when you can use RPC? (At least CRI is binary RPC instead of JSON over HTTP/1.) Also, Go doesn't dlopen AFAIK.
  - bradfitz 10y ago
    https://beta.golang.org/doc/go1.8#plugin https://beta.golang.org/doc/go1.8#plugin
  - deleted 10y ago
    [deleted]
  - vidarh 10y ago
    It gets better. Take a look at rktlet, a CRI implementation for rkt (EDIT: I originally mistakenly wrote Docker). Specifically the runtime [1], which ends up shelling out to the "rkt" binary. So you end up running a new daemon that communicates with Kubernetes via gRPC, that then spawns rkt anyway. So you get to RPC and fork/exec. I'm sure that ends up "optimized away" at some point by e.g. having a rkt CRI implementation that just links in the relevant rkt code. But I'm left wondering why we need this complexity in the first place. [1] https://github.com/kubernetes-incubator/rktlet/tree/master/rktlet/runtime https://github.com/kubernetes-incubator/rktlet/tree/master/r...
    - chrissnell 10y ago
      I don't get your argument. If I'm reading this correctly, you're arguing that system calls and/or a call to a shared library function are cleaner than RPC to another process? The overhead of RPC in an application like this is tiny and the cost of an additional process on 2016 equipment is non-existent.
      - vidarh 10y ago
        It's more things that can fail that now needs monitoring. (EDIT:) And in the specific case of rktlet you still end up forking/execing anyway. The overhead isn't necessarily a big deal (and can easily go the other way - if the request frequency is high enough, it's cheaper to keep the process around), but it does also potentially add up.
- xyzzy123 10y ago
  The usual reasons to choose this pattern are "because versioning" and "location transparency". When the resources you interact with are local OR you have tight over the environment, you're totally right, there's little reason to introduce "middle procs". When there's a risk that different versions of things will manipulate resources in incompatible ways then the rpc layer makes sense.
  - vidarh 10y ago
    The resources are local in this case - it specifically uses a Unix domain socket. And if/when the "different versions of things" are an issue, they can just as well affect the RPC layer.
    - xyzzy123 10y ago
      OK yeah, fair point, sorry, I didn't evaluate the context sufficiently.
- ithkuil 10y ago
  What if the thing you're calling via an RPC abstraction is not a resident deamon but instead it gets loaded on demand via socket activation and then after performing the request it shuts off? Or a process gets forked and a RPC request/response get passed via stdio? Would that smell better? I'm honestly curious. (I'm not saying that this would make sense for the CRI API, I didn't even looked at it)
  - vidarh 10y ago
    Socket activation is a possible workaround, of course, but you then still have the gPRC dependencies. At least that removes the issue of keeping the service running. In general for an API as simple as the CRI API I'd prefer forking and passing data via stdio or the environment any day.
- compsciphd 10y ago
  In practice the GRPC interface can be "statically linked" into the kubelet binary and it then just becomes a plugin API that implementations can build. I believe this is how "dockershim" works today. The advantage of the GRPC framework for developing though are tremendous, I'm building my own CRI implementation and I can iterate extremely quickly due to the fact that it can use RPC.
  - vidarh 10y ago
    If the separate processes can be avoided, that's quite a bit better, though also entirely different from the descriptions on their pages. What advantage do you get from gRPC over e.g. taking a JSON document on stdin or using environment variables? To me the issue isn't that things happens in a separate process, but that I really don't like solutions that ends up adding even more moving wheels without very good reason. Kubernetes is extremely convoluted as it is.
    - compsciphd 10y ago
      The way I see GRPC in the context of the CRI is a very well defined API. i.e. kubelet makes function (or in this case rpc) calls to my implementation of the api and I return results to it. I tend to think of it like the VFS in the Linux kernel. If Linux was a microkernel, it would talk to each FS module over an RPC interface, since its not, they are standard function calls as statically linked together. I'm unsure what your Q really is, how would I easily have function request/response layer with json documents or stdin? It seems more complicated. But as I don't quite understand perhaps I'm missing the point. I also dont think its adding more moving wheels, its just taking one wheel out of kubelet (i.e. dockertools part of kubelet) and moving it elsewhere (i.e. dockershim as a cri implementation). The only increase in "moving wheels" is that even in the non "statically linked" scenario, one has to distribute 2 binaries to each node. kubelet and the cri implementation, but in practice I'm not sure thats really such a big deal.
philips 10y ago
This was posted last week but here is rkt's roadmap around Kubernetes's CRI and use of OCI's runc: https://coreos.com/blog/rkt-and-kubernetes.html https://coreos.com/blog/rkt-and-kubernetes.html
cyphar 10y ago
Currently quite a few people from the OCI community (including myself) are working on implementing a CRI-compliant runtime[1] around runC and the various OCI specifications as well as the containers/image and containers/storage projects. There's a lot of cool design to ocid which means that it doesn't require a daemon to be constantly running. [1]: https://github.com/kubernetes-incubator/cri-o https://github.com/kubernetes-incubator/cri-o
- bogomipz 10y ago
  I am curious how do you manage developing this while the OCI runtime is not yet finalized, do you just make sure they are compliant with the current RC of the spec?
  - mrunalp 10y ago
    The OCI runtime-spec is very close to 1.0. We are just using the rc version for now.
- vidarh 10y ago
  Do you have any more specific pointers regarding using it without a daemon? As the examples seem to start with starting a daemon, unless I misunderstand something. If it doesn't need that, then that's a big plus in my book. Though, I'm getting more and more disillusioned in general with where these specs are heading - the complexity seems to be skyrocketing for sometimes very little benefit. Not necessarily specific to Kubernetes and/or OCI - Docker is a prime offender. E.g. typical example: the highly coupled nature of many of the networking alternatives where routing, fabric and ip allocation gets muddled all up, when there are well developed, stable, well tested independent and orthogonal alternatives for tunnelling and route propagation there's a serious level of Not Invented Here syndrome at work in many of the container projects. I'm sure some people need all the complexity, but I'm getting more and more tempted to ditch many of the higher level tools in favour of composing smaller, simpler tools. (Incidentally I'll make one prediction: one good thing likely to come from CRI is that I suspect it will lead to a new array of Kubernetes "replacements" from simpler composable tools; the APIs don't look all that bad - I just don't like the RPC dependency)

124 points by philips 10y ago

I can't get over how much of a smell the requirement of an RPC interface in order to interface with tools that may or may not have any good reason to be running on an ongoing basis is to me. I'm sure there may be cases where you interact with the containers frequently enough that spawning a process each time is actually a worthwhile optimization, but more and more of these containerisation systems are becoming an unholy mess of daemons that needs to run in order to run and manage containers that need not depend on anything but the host init/systemd. E.g. one of the really appealing things of rkt for me is the simplicity - depending on the level of isolation everything is running either direclty under systemd, or under an individual isolator like systemd-nspawn. I disliked this tendency towards a herd of daemons intensely when Docker continued as it started and used HTTP for volume/network plugins, and I dislike it just as much now. It's as if someone sat down and thought long and hard about how to add more complexity and more "fun" failure modes.

wmf 10y ago
It's the microservice philosophy: Why use a function call or fork/exec when you can use RPC? (At least CRI is binary RPC instead of JSON over HTTP/1.) Also, Go doesn't dlopen AFAIK.
- bradfitz 10y ago
  https://beta.golang.org/doc/go1.8#plugin https://beta.golang.org/doc/go1.8#plugin
- deleted 10y ago
  [deleted]
- vidarh 10y ago
  It gets better. Take a look at rktlet, a CRI implementation for rkt (EDIT: I originally mistakenly wrote Docker). Specifically the runtime [1], which ends up shelling out to the "rkt" binary. So you end up running a new daemon that communicates with Kubernetes via gRPC, that then spawns rkt anyway. So you get to RPC and fork/exec. I'm sure that ends up "optimized away" at some point by e.g. having a rkt CRI implementation that just links in the relevant rkt code. But I'm left wondering why we need this complexity in the first place. [1] https://github.com/kubernetes-incubator/rktlet/tree/master/rktlet/runtime https://github.com/kubernetes-incubator/rktlet/tree/master/r...
  - chrissnell 10y ago
    I don't get your argument. If I'm reading this correctly, you're arguing that system calls and/or a call to a shared library function are cleaner than RPC to another process? The overhead of RPC in an application like this is tiny and the cost of an additional process on 2016 equipment is non-existent.
    - vidarh 10y ago
      It's more things that can fail that now needs monitoring. (EDIT:) And in the specific case of rktlet you still end up forking/execing anyway. The overhead isn't necessarily a big deal (and can easily go the other way - if the request frequency is high enough, it's cheaper to keep the process around), but it does also potentially add up.
xyzzy123 10y ago
The usual reasons to choose this pattern are "because versioning" and "location transparency". When the resources you interact with are local OR you have tight over the environment, you're totally right, there's little reason to introduce "middle procs". When there's a risk that different versions of things will manipulate resources in incompatible ways then the rpc layer makes sense.
- vidarh 10y ago
  The resources are local in this case - it specifically uses a Unix domain socket. And if/when the "different versions of things" are an issue, they can just as well affect the RPC layer.
  - xyzzy123 10y ago
    OK yeah, fair point, sorry, I didn't evaluate the context sufficiently.
ithkuil 10y ago
What if the thing you're calling via an RPC abstraction is not a resident deamon but instead it gets loaded on demand via socket activation and then after performing the request it shuts off? Or a process gets forked and a RPC request/response get passed via stdio? Would that smell better? I'm honestly curious. (I'm not saying that this would make sense for the CRI API, I didn't even looked at it)
- vidarh 10y ago
  Socket activation is a possible workaround, of course, but you then still have the gPRC dependencies. At least that removes the issue of keeping the service running. In general for an API as simple as the CRI API I'd prefer forking and passing data via stdio or the environment any day.
compsciphd 10y ago
In practice the GRPC interface can be "statically linked" into the kubelet binary and it then just becomes a plugin API that implementations can build. I believe this is how "dockershim" works today. The advantage of the GRPC framework for developing though are tremendous, I'm building my own CRI implementation and I can iterate extremely quickly due to the fact that it can use RPC.
- vidarh 10y ago
  If the separate processes can be avoided, that's quite a bit better, though also entirely different from the descriptions on their pages. What advantage do you get from gRPC over e.g. taking a JSON document on stdin or using environment variables? To me the issue isn't that things happens in a separate process, but that I really don't like solutions that ends up adding even more moving wheels without very good reason. Kubernetes is extremely convoluted as it is.
  - compsciphd 10y ago
    The way I see GRPC in the context of the CRI is a very well defined API. i.e. kubelet makes function (or in this case rpc) calls to my implementation of the api and I return results to it. I tend to think of it like the VFS in the Linux kernel. If Linux was a microkernel, it would talk to each FS module over an RPC interface, since its not, they are standard function calls as statically linked together. I'm unsure what your Q really is, how would I easily have function request/response layer with json documents or stdin? It seems more complicated. But as I don't quite understand perhaps I'm missing the point. I also dont think its adding more moving wheels, its just taking one wheel out of kubelet (i.e. dockertools part of kubelet) and moving it elsewhere (i.e. dockershim as a cri implementation). The only increase in "moving wheels" is that even in the non "statically linked" scenario, one has to distribute 2 binaries to each node. kubelet and the cri implementation, but in practice I'm not sure thats really such a big deal.

philips 10y ago

This was posted last week but here is rkt's roadmap around Kubernetes's CRI and use of OCI's runc: https://coreos.com/blog/rkt-and-kubernetes.html https://coreos.com/blog/rkt-and-kubernetes.html

cyphar 10y ago

Currently quite a few people from the OCI community (including myself) are working on implementing a CRI-compliant runtime[1] around runC and the various OCI specifications as well as the containers/image and containers/storage projects. There's a lot of cool design to ocid which means that it doesn't require a daemon to be constantly running. [1]: https://github.com/kubernetes-incubator/cri-o https://github.com/kubernetes-incubator/cri-o

bogomipz 10y ago
I am curious how do you manage developing this while the OCI runtime is not yet finalized, do you just make sure they are compliant with the current RC of the spec?
- mrunalp 10y ago
  The OCI runtime-spec is very close to 1.0. We are just using the rc version for now.
vidarh 10y ago
Do you have any more specific pointers regarding using it without a daemon? As the examples seem to start with starting a daemon, unless I misunderstand something. If it doesn't need that, then that's a big plus in my book. Though, I'm getting more and more disillusioned in general with where these specs are heading - the complexity seems to be skyrocketing for sometimes very little benefit. Not necessarily specific to Kubernetes and/or OCI - Docker is a prime offender. E.g. typical example: the highly coupled nature of many of the networking alternatives where routing, fabric and ip allocation gets muddled all up, when there are well developed, stable, well tested independent and orthogonal alternatives for tunnelling and route propagation there's a serious level of Not Invented Here syndrome at work in many of the container projects. I'm sure some people need all the complexity, but I'm getting more and more tempted to ditch many of the higher level tools in favour of composing smaller, simpler tools. (Incidentally I'll make one prediction: one good thing likely to come from CRI is that I suspect it will lead to a new array of Kubernetes "replacements" from simpler composable tools; the APIs don't look all that bad - I just don't like the RPC dependency)