3 ms·
I can't get over how much of a smell the requirement of an RPC interface in order to interface with tools that may or may not have any good reason to be running
by vidarh 10y ago
I can't get over how much of a smell the requirement of an RPC interface in order to interface with tools that may or may not have any good reason to be running on an ongoing basis is to me.
I'm sure there may be cases where you interact with the containers frequently enough that spawning a process each time is actually a worthwhile optimization, but more and more of these containerisation systems are becoming an unholy mess of daemons that needs to run in order to run and manage containers that need not depend on anything but the host init/systemd.
E.g. one of the really appealing things of rkt for me is the simplicity - depending on the level of isolation everything is running either direclty under systemd, or under an individual isolator like systemd-nspawn.
I disliked this tendency towards a herd of daemons intensely when Docker continued as it started and used HTTP for volume/network plugins, and I dislike it just as much now.
It's as if someone sat down and thought long and hard about how to add more complexity and more "fun" failure modes.
- wmf 10y agoIt's the microservice philosophy: Why use a function call or fork/exec when you can use RPC? (At least CRI is binary RPC instead of JSON over HTTP/1.) Also, Go doesn't dlopen AFAIK.
- bradfitz 10y agohttps://beta.golang.org/doc/go1.8#plugin https://beta.golang.org/doc/go1.8#plugin
- deleted 10y ago[deleted]
- vidarh 10y agoIt gets better. Take a look at rktlet, a CRI implementation for rkt (EDIT: I originally mistakenly wrote Docker). Specifically the runtime [1], which ends up shelling out to the "rkt" binary. So you end up running a new daemon that communicates with Kubernetes via gRPC, that then spawns rkt anyway. So you get to RPC and fork/exec. I'm sure that ends up "optimized away" at some point by e.g. having a rkt CRI implementation that just links in the relevant rkt code. But I'm left wondering why we need this complexity in the first place. [1] https://github.com/kubernetes-incubator/rktlet/tree/master/rktlet/runtime https://github.com/kubernetes-incubator/rktlet/tree/master/r...
- chrissnell 10y agoI don't get your argument. If I'm reading this correctly, you're arguing that system calls and/or a call to a shared library function are cleaner than RPC to another process? The overhead of RPC in an application like this is tiny and the cost of an additional process on 2016 equipment is non-existent.
- vidarh 10y agoIt's more things that can fail that now needs monitoring. (EDIT:) And in the specific case of rktlet you still end up forking/execing anyway. The overhead isn't necessarily a big deal (and can easily go the other way - if the request frequency is high enough, it's cheaper to keep the process around), but it does also potentially add up.
- xyzzy123 10y agoThe usual reasons to choose this pattern are "because versioning" and "location transparency". When the resources you interact with are local OR you have tight over the environment, you're totally right, there's little reason to introduce "middle procs". When there's a risk that different versions of things will manipulate resources in incompatible ways then the rpc layer makes sense.
- ithkuil 10y agoWhat if the thing you're calling via an RPC abstraction is not a resident deamon but instead it gets loaded on demand via socket activation and then after performing the request it shuts off? Or a process gets forked and a RPC request/response get passed via stdio? Would that smell better? I'm honestly curious. (I'm not saying that this would make sense for the CRI API, I didn't even looked at it)
- vidarh 10y agoSocket activation is a possible workaround, of course, but you then still have the gPRC dependencies. At least that removes the issue of keeping the service running. In general for an API as simple as the CRI API I'd prefer forking and passing data via stdio or the environment any day.
- compsciphd 10y agoIn practice the GRPC interface can be "statically linked" into the kubelet binary and it then just becomes a plugin API that implementations can build. I believe this is how "dockershim" works today. The advantage of the GRPC framework for developing though are tremendous, I'm building my own CRI implementation and I can iterate extremely quickly due to the fact that it can use RPC.
- vidarh 10y agoIf the separate processes can be avoided, that's quite a bit better, though also entirely different from the descriptions on their pages. What advantage do you get from gRPC over e.g. taking a JSON document on stdin or using environment variables? To me the issue isn't that things happens in a separate process, but that I really don't like solutions that ends up adding even more moving wheels without very good reason. Kubernetes is extremely convoluted as it is.
- compsciphd 10y agoThe way I see GRPC in the context of the CRI is a very well defined API. i.e. kubelet makes function (or in this case rpc) calls to my implementation of the api and I return results to it. I tend to think of it like the VFS in the Linux kernel. If Linux was a microkernel, it would talk to each FS module over an RPC interface, since its not, they are standard function calls as statically linked together. I'm unsure what your Q really is, how would I easily have function request/response layer with json documents or stdin? It seems more complicated. But as I don't quite understand perhaps I'm missing the point. I also dont think its adding more moving wheels, its just taking one wheel out of kubelet (i.e. dockertools part of kubelet) and moving it elsewhere (i.e. dockershim as a cri implementation). The only increase in "moving wheels" is that even in the non "statically linked" scenario, one has to distribute 2 binaries to each node. kubelet and the cri implementation, but in practice I'm not sure thats really such a big deal.