6 ms·

I created this (I'm the Chief Architect and Co-founder of Rancher Labs, I experiment with stuff like this all the time). The purpose of this project is really t

I created this (I'm the Chief Architect and Co-founder of Rancher Labs, I experiment with stuff like this all the time). The purpose of this project is really to embed in another project (https://github.com/rancher/rio https://github.com/rancher/rio is an example) but as a side effect it's a nice little standalone k8s package. I never had any intention to support it as a standalone project, but hey if somebody is interested who knows. All of the features removed are based off of two basic ideas 1) If there are X ways to do something, I choose the best one (obviously subjective). 2) The features are not commonly used or really shouldn't be in k8s to begin with. The end goal of what I'm working on is basically to use k8s code as a library to do orchestration. Step one was to reduce the footprint of k8s (I'm 80% there with this project), step two is to completely rework the state management such that you no longer need to persist state in k8s. This is a much larger goal. I've fooled around a lot with the persistence layer in k8s (this project is running on sqlite3) so I'm basically doing a lot of work of figuring what state I can throw away. The theory is that all desired state comes from your yaml files. Actual state is actual (what really exists). Everything else in k8s should be purely derived and not important.

nwmcsween 8y ago
OT but why do you want to push containers? In my opinion it's a misadventure from the real issue of a proper package manager (which in turn is a misadventure from a proper build system) The issue with current containers is it's the wrong level of abstraction, containment should be per binary and ideally inline with code (see pledge(2)).
- barrkel 8y ago
  K8s is about cluster orchestration. It's for building apps that allocate memory and compute from a pool of machines almost like malloc() and free(). It's also an abstraction for allocating compute away from the specifics of a provider, so you theoretically don't care if you deploy to gcloud or Azure or AWS. Containers and images merely define the interface with the agnostically allocated compute. They're not the essential bit though, not the magic.
  - djsumdog 8y ago
    Right, and there are some scheduling systems like Maraton/mesos that can be used to control jobs or VMs as well (although they're more often used with containers than anything else these days). Can K8s run VMs or is its only abstraction containers/pods?
    - dharmab 8y ago
      Containers in pods only. In practice we run all of our Mesos tasks and Marathon services in containers anyway, as you mentioned.
    - peteridah 8y ago
      The pod abstraction can be implemented as a vm. Take a look at clear containers
  - nwmcsween 8y ago
    I never said K8s wasn't about orchestration, I said why push containers in their current horribly broken form
- deleted 8y ago
  [deleted]
- cyphar 8y ago
  As one of the maintainers of runc, I cannot agree with your general thrust more (I am obviously biased since I work for a distribution company -- SUSE). I don't agree it should be containment-per-binary, but I think that containment should be transparent to the operating system so it can be managed. Currently the idea with k8s is that you just destroy anything that has broken state and rebuild it somewhere else. While this is the easiest solution to the problem, I think that containers should not always be treated like disposable objects. I think in many ways, Solaris Zones (and FreeBSD Jails to a lesser extent) did this right. Every administrative command in Solaris is zone-aware and you can manage your zones like you manage your host. On Linux, containers are completely opaque root filesystems which contain who-knows-what and are completely unmanagable or introspectable by the host. Everyone is cargo-culting and creating hacks around this fundamentally opaque system to try to make it less opaque (or just giving up and making it easy to purge them). I'm working on some improvements to the OCI image-spec that _might_ help with introspection of images[1], but runtime introspection is going to be quite hard (needs runtime agreement). I get that it's no longer sexy to work on container runtimes, now that everyone is losing their mind over orchestrators (and I imagine soon they'll be losing their mind over the next big thing). But I think that improving these fundamental blocks is pretty interesting and might result in better systems. [1]: https://github.com/openSUSE/umoci/issues/256 https://github.com/openSUSE/umoci/issues/256
  - pm90 8y ago
    It was never sexy to work on container runtimes, and nobody is losing their minds over orchestrators either. There was a real need for a platform to deploy to without being tied in to {aws, gcp, etc}. First it was OpenStack, now its kubernetes.
  - imhoguy 8y ago
    There is always systemd-nspawn to do it the simple way.
    - cyphar 8y ago
      systemd-nspawn doesn't really help do any of what I complained about. Besides, LXC is a much better "simple" container runtime in my opinion.
- Carpetsmoker 8y ago
  I think pledge (and unveil) are really good ideas, but that doesn't mean it's a substitute for containers. They're kind of different: pledge mostly just enforces correctness of a program, whereas containers enforce separation. I think containers are more analogous to privsep (used extensively in OpenBSD) than pledge. My biggest issue with the current state is that the common tools (e.g. Docker, k8s) are pretty terrible. Docker overcomplicates a great many things, which is one reason for the many bugs. And k8s ... well, let's not even go there. I don't think much more is needed than just `run-container /usr/containers/cool-container` and presto, a container with the files in that directory. No need for buggy/slow platform-dependent FS abstraction layers that change every year because they never quite work right in a bazillion edge cases. This is how e.g. FreeBSD jails work. This won't prevent implementing tools like `docker pull`; at it simplest it could be `curl ... | tar xf - -C /usr/containers/cool-container`. Running multiple containers based of a single image? `cp -r`. Maybe I'm just an old Unix guy, but I think that composition from generic tools is a much better approach for a lot of this stuff. It's easier, less platform-dependent, has less code (meaning less bugs), etc.
  - nwmcsween 8y ago
    Containers are basically a few syscalls wrapped in a ball of mud, pledge is containment and the Linux somewhat broken and ugly alternative seccomp bpf is used in most containers _but_ not properly as most containers are bundles of software (multiple binaries) which makes it hard to contain a container (ha) as you have to allow more syscalls than needed. A better way would just use .preinit_array and inject a .seccomp section, .unshare, etc to create per binary containment or just chain-load everything in one big line.
    - Carpetsmoker 8y ago
      > Containers are basically a few syscalls wrapped in a ball of mud I don't think it has to be; I mean, the current Linux/Docker situation is kind of that, but that's just a single implementation. > most containers are bundles of software (multiple binaries) Yeah, exactly. Containers aren't just about security, also convenience. Something like "docker pull someimage" can be a handy cross-platform way to run stuff. I think "docker pull" is the reason for Docker's popularity. You can build a self-contained container containing Ruby, all your Ruby on Rails stuff and JS stuff, and whatnot, and then send that to any server and just "docker run" it. Getting some of this dynamic stuff to run can be tricky and error-prone: need to get the correct Ruby version, Rails version, muck about with bundle, etc. Want to add a new app-server or migrate to somewhere else? Need to spend a day setting everything up again and hope you didn't make a mistake that will bite you later :-( A self-contained container is a lot easier, even with good OS package management or automation tools like Puppet. Many people also use Docker as "Vagrant" (e.g. to run dev versions or tests and whatnot). Again, "docker pull" is convenient here, especially when there are certain dependencies/requirements, or for running testing against different versions of some dependency, etc. Again, I don't think Docker is very good as such, but I do think there are some ideas worth keeping, both for security and plain convenience (not often those two go hand-in-hand!)
jacques_chester 8y ago
How hard was the etcdectomy surgery? I can't see that situation getting prettier as the world of CRDs explodes.
- darren0 8y ago
  Easier than you would think. I basically replaced the etcd KV API with a SQL based one. But that means I kept the majority of the k8s etcd3 driver in place which has all of the complicated k8s specific code.
djsumdog 8y ago
Striped down and it still looks pretty complex. I really like containers, but I feel the orchestrations systems all leave something to be desired. I've heard there are some big changes in Racher and I've been meaning to look at it again. I've setup DC/OS and played with Nomad and wrote this post about my frustration dealing with orchestration systems: https://penguindreams.org/blog/my-love-hate-relationship-with-docker-and-container-orchestration-systems/ https://penguindreams.org/blog/my-love-hate-relationship-wit...
- darren0 8y ago
  I completely agree. The current trend of orchestration all sells a story of hyperscale like GIFEE. That is not how most people do or should run there deployments (I'm talking about the vast majority outside SV). The operational complexity today is too high. Kubernetes is a big business system, it caters to enterprise. If you aren't an enterprise I would caution you against k8s.
  - mwcampbell 8y ago
    What do you think most people should use instead of k8s? Or is there not a good answer for that yet? Do you think containers themselves are overkill, or just k8s?
    - darren0 8y ago
      Containers have a lot of value. My rule of thumb is basically use containers (docker) and as little as possible of anything else in the container ecosystem. So basically put your app in a container try to not change too much from however you currently do things. There is no shame in running --net=host or just doing "-v /mnt/someebsvolume:/data." I personally like Hashicorp because their tools are simple and grow with you. Besides Hashicorp, I don't have great recommendations for tools. In my opinion the current container ecosystem is a disaster for users. Big business ideas took over and killed innovation. Simple lightweight modular systems all got annihilated by Kubernetes and now it's pretty much a monoculture.
audi0slave 8y ago
How do you get the watch API's without etcd? or does k3s doesn't have watch API support.
- darren0 8y ago
  It's a combination of pulling and notification. The sqlite DB table is structured like a log, so clients basically just need to keep grabbing the latest id. In practice it's has proven to work pretty well. The code I have public right now does not support HA, I have a private branch that I'm still testing.
sandGorgon 8y ago
In context of complexity vs power tradeoff, where do you see this versus Docker Swarm ?
- darren0 8y ago
  k3s is not intended to be a competitor, but let's say I wanted to build it out to compete with systems. Swarm excels in UX and cluster management. It is easy to run and use. But it is incomplete in features and functionality. So k3s addresses the cluster management issues and has tons of features. It makes k8s easy to run, but it still has Kubernetes UX which is quite difficult. So in the end I think it is a wash between k3s and Swarm. You either get hard to use (k3s) or incomplete (Swarm). I have a different project Rio (https://github.com/rancher/rio https://github.com/rancher/rio) that attempts to address UX (and it's built on k3s). So Rio could possibly be a system that is easy to use, powerful, and simple to run. I think Rio could be something that is better than both raw k8s and Swarm for regular users, but it's a very early.
  - ofrzeta 8y ago
    Looks interesting. Why are you (Rancher) building your own distributed block storage (Longhorn)? What about Rook? OpenEBS?
Rapzid 8y ago
I really enjoyed your initial rancher orchestration architecture write up and diagram. In some ways the centralized state machine and federated transition handling reminds me of the omega system now that I think about it.. It would be cool to read what your orchestration plans are for k8s.
- darren0 8y ago
  I'm working on some pretty massive things right now. I should have something to show December (DockerCon EU) time frame. It's all in the area of state management. This is something I've been working on for quite some time. The hardest part of an orchestration systems is the persistent store. I'm working on some patterns to effectively remove the need for database but use only object storage (which is significantly more reliable, cheaper, and easier)
Serow225 8y ago
I know pretty much zero about k8s, but was it really important to remove Swagger? Seems like relatively useful functionality and not a huge complexity point? I really like the idea of a simple opinionated K8s BTW good luck!
- felixfbecker 8y ago
  Yeah this seems weird to me. If your API schema is not introspectable, where do your API clients come from? They don't grow on trees. I guess it's easy to cheat and just use an existing client, but that just shows that this is not a sustainable project
- flurdy 8y ago
  You can probably argue the same about every part removed, but if you (him) are not going to use it now, remove it. Just dead weight otherwise. You can always add it later if people really need it. True, Swagger docs can be helpful, but also if present but rubbish ignored swagger docs with missing features or side effects is more confusing. Just talk.
  - zeptomu 8y ago
    Agreed. I don't really get the love for swagger, everything about it seems so meta. People say you get documentation for free, but auto generating some fancy webpage out of code (or the swagger spec) is not really documentation. You have to write your documentation in a real human language. Maybe I just don't get the point of swagger.
kaivi 8y ago
Hey, I've been using Rancher for a year, and I love it. Just wanted to bring your attention to a YouTube channel from Rancher Labs, where "Learning Rancher" videos are posted regularly. But the content is all the same: boring 10min intro to every video, after which developers go through the same stuff, painfully slowly. Is it meaningful to produce this kind of content over and over again? IMO there are already way better Rancher tutorials by indian dudes on YouTube. I've started with k8s thanks to Rancher, and I presume that the vast majority of Rancher users are doing the same. Rancher without k8s does not make sense, and it seems to me that k8s stuff is actively avoided in docs and tutorials. Maybe it would be better to make a series of tutorials on Rancher+k8s, and go through concrete topics like networking and CI?

nwmcsween 8y ago

OT but why do you want to push containers? In my opinion it's a misadventure from the real issue of a proper package manager (which in turn is a misadventure from a proper build system) The issue with current containers is it's the wrong level of abstraction, containment should be per binary and ideally inline with code (see pledge(2)).

barrkel 8y ago
K8s is about cluster orchestration. It's for building apps that allocate memory and compute from a pool of machines almost like malloc() and free(). It's also an abstraction for allocating compute away from the specifics of a provider, so you theoretically don't care if you deploy to gcloud or Azure or AWS. Containers and images merely define the interface with the agnostically allocated compute. They're not the essential bit though, not the magic.
- djsumdog 8y ago
  Right, and there are some scheduling systems like Maraton/mesos that can be used to control jobs or VMs as well (although they're more often used with containers than anything else these days). Can K8s run VMs or is its only abstraction containers/pods?
  - dharmab 8y ago
    Containers in pods only. In practice we run all of our Mesos tasks and Marathon services in containers anyway, as you mentioned.
  - peteridah 8y ago
    The pod abstraction can be implemented as a vm. Take a look at clear containers
- nwmcsween 8y ago
  I never said K8s wasn't about orchestration, I said why push containers in their current horribly broken form
deleted 8y ago
[deleted]
cyphar 8y ago
As one of the maintainers of runc, I cannot agree with your general thrust more (I am obviously biased since I work for a distribution company -- SUSE). I don't agree it should be containment-per-binary, but I think that containment should be transparent to the operating system so it can be managed. Currently the idea with k8s is that you just destroy anything that has broken state and rebuild it somewhere else. While this is the easiest solution to the problem, I think that containers should not always be treated like disposable objects. I think in many ways, Solaris Zones (and FreeBSD Jails to a lesser extent) did this right. Every administrative command in Solaris is zone-aware and you can manage your zones like you manage your host. On Linux, containers are completely opaque root filesystems which contain who-knows-what and are completely unmanagable or introspectable by the host. Everyone is cargo-culting and creating hacks around this fundamentally opaque system to try to make it less opaque (or just giving up and making it easy to purge them). I'm working on some improvements to the OCI image-spec that _might_ help with introspection of images[1], but runtime introspection is going to be quite hard (needs runtime agreement). I get that it's no longer sexy to work on container runtimes, now that everyone is losing their mind over orchestrators (and I imagine soon they'll be losing their mind over the next big thing). But I think that improving these fundamental blocks is pretty interesting and might result in better systems. [1]: https://github.com/openSUSE/umoci/issues/256 https://github.com/openSUSE/umoci/issues/256
- pm90 8y ago
  It was never sexy to work on container runtimes, and nobody is losing their minds over orchestrators either. There was a real need for a platform to deploy to without being tied in to {aws, gcp, etc}. First it was OpenStack, now its kubernetes.
- imhoguy 8y ago
  There is always systemd-nspawn to do it the simple way.
  - cyphar 8y ago
    systemd-nspawn doesn't really help do any of what I complained about. Besides, LXC is a much better "simple" container runtime in my opinion.
Carpetsmoker 8y ago
I think pledge (and unveil) are really good ideas, but that doesn't mean it's a substitute for containers. They're kind of different: pledge mostly just enforces correctness of a program, whereas containers enforce separation. I think containers are more analogous to privsep (used extensively in OpenBSD) than pledge. My biggest issue with the current state is that the common tools (e.g. Docker, k8s) are pretty terrible. Docker overcomplicates a great many things, which is one reason for the many bugs. And k8s ... well, let's not even go there. I don't think much more is needed than just `run-container /usr/containers/cool-container` and presto, a container with the files in that directory. No need for buggy/slow platform-dependent FS abstraction layers that change every year because they never quite work right in a bazillion edge cases. This is how e.g. FreeBSD jails work. This won't prevent implementing tools like `docker pull`; at it simplest it could be `curl ... | tar xf - -C /usr/containers/cool-container`. Running multiple containers based of a single image? `cp -r`. Maybe I'm just an old Unix guy, but I think that composition from generic tools is a much better approach for a lot of this stuff. It's easier, less platform-dependent, has less code (meaning less bugs), etc.
- nwmcsween 8y ago
  Containers are basically a few syscalls wrapped in a ball of mud, pledge is containment and the Linux somewhat broken and ugly alternative seccomp bpf is used in most containers _but_ not properly as most containers are bundles of software (multiple binaries) which makes it hard to contain a container (ha) as you have to allow more syscalls than needed. A better way would just use .preinit_array and inject a .seccomp section, .unshare, etc to create per binary containment or just chain-load everything in one big line.
  - Carpetsmoker 8y ago
    > Containers are basically a few syscalls wrapped in a ball of mud I don't think it has to be; I mean, the current Linux/Docker situation is kind of that, but that's just a single implementation. > most containers are bundles of software (multiple binaries) Yeah, exactly. Containers aren't just about security, also convenience. Something like "docker pull someimage" can be a handy cross-platform way to run stuff. I think "docker pull" is the reason for Docker's popularity. You can build a self-contained container containing Ruby, all your Ruby on Rails stuff and JS stuff, and whatnot, and then send that to any server and just "docker run" it. Getting some of this dynamic stuff to run can be tricky and error-prone: need to get the correct Ruby version, Rails version, muck about with bundle, etc. Want to add a new app-server or migrate to somewhere else? Need to spend a day setting everything up again and hope you didn't make a mistake that will bite you later :-( A self-contained container is a lot easier, even with good OS package management or automation tools like Puppet. Many people also use Docker as "Vagrant" (e.g. to run dev versions or tests and whatnot). Again, "docker pull" is convenient here, especially when there are certain dependencies/requirements, or for running testing against different versions of some dependency, etc. Again, I don't think Docker is very good as such, but I do think there are some ideas worth keeping, both for security and plain convenience (not often those two go hand-in-hand!)

jacques_chester 8y ago

How hard was the etcdectomy surgery? I can't see that situation getting prettier as the world of CRDs explodes.

darren0 8y ago
Easier than you would think. I basically replaced the etcd KV API with a SQL based one. But that means I kept the majority of the k8s etcd3 driver in place which has all of the complicated k8s specific code.

djsumdog 8y ago

Striped down and it still looks pretty complex. I really like containers, but I feel the orchestrations systems all leave something to be desired. I've heard there are some big changes in Racher and I've been meaning to look at it again. I've setup DC/OS and played with Nomad and wrote this post about my frustration dealing with orchestration systems: https://penguindreams.org/blog/my-love-hate-relationship-with-docker-and-container-orchestration-systems/ https://penguindreams.org/blog/my-love-hate-relationship-wit...

darren0 8y ago
I completely agree. The current trend of orchestration all sells a story of hyperscale like GIFEE. That is not how most people do or should run there deployments (I'm talking about the vast majority outside SV). The operational complexity today is too high. Kubernetes is a big business system, it caters to enterprise. If you aren't an enterprise I would caution you against k8s.
- mwcampbell 8y ago
  What do you think most people should use instead of k8s? Or is there not a good answer for that yet? Do you think containers themselves are overkill, or just k8s?
  - darren0 8y ago
    Containers have a lot of value. My rule of thumb is basically use containers (docker) and as little as possible of anything else in the container ecosystem. So basically put your app in a container try to not change too much from however you currently do things. There is no shame in running --net=host or just doing "-v /mnt/someebsvolume:/data." I personally like Hashicorp because their tools are simple and grow with you. Besides Hashicorp, I don't have great recommendations for tools. In my opinion the current container ecosystem is a disaster for users. Big business ideas took over and killed innovation. Simple lightweight modular systems all got annihilated by Kubernetes and now it's pretty much a monoculture.

audi0slave 8y ago

How do you get the watch API's without etcd? or does k3s doesn't have watch API support.

darren0 8y ago
It's a combination of pulling and notification. The sqlite DB table is structured like a log, so clients basically just need to keep grabbing the latest id. In practice it's has proven to work pretty well. The code I have public right now does not support HA, I have a private branch that I'm still testing.

sandGorgon 8y ago

In context of complexity vs power tradeoff, where do you see this versus Docker Swarm ?

darren0 8y ago
k3s is not intended to be a competitor, but let's say I wanted to build it out to compete with systems. Swarm excels in UX and cluster management. It is easy to run and use. But it is incomplete in features and functionality. So k3s addresses the cluster management issues and has tons of features. It makes k8s easy to run, but it still has Kubernetes UX which is quite difficult. So in the end I think it is a wash between k3s and Swarm. You either get hard to use (k3s) or incomplete (Swarm). I have a different project Rio (https://github.com/rancher/rio https://github.com/rancher/rio) that attempts to address UX (and it's built on k3s). So Rio could possibly be a system that is easy to use, powerful, and simple to run. I think Rio could be something that is better than both raw k8s and Swarm for regular users, but it's a very early.
- ofrzeta 8y ago
  Looks interesting. Why are you (Rancher) building your own distributed block storage (Longhorn)? What about Rook? OpenEBS?

Rapzid 8y ago

I really enjoyed your initial rancher orchestration architecture write up and diagram. In some ways the centralized state machine and federated transition handling reminds me of the omega system now that I think about it.. It would be cool to read what your orchestration plans are for k8s.

darren0 8y ago
I'm working on some pretty massive things right now. I should have something to show December (DockerCon EU) time frame. It's all in the area of state management. This is something I've been working on for quite some time. The hardest part of an orchestration systems is the persistent store. I'm working on some patterns to effectively remove the need for database but use only object storage (which is significantly more reliable, cheaper, and easier)

Serow225 8y ago

I know pretty much zero about k8s, but was it really important to remove Swagger? Seems like relatively useful functionality and not a huge complexity point? I really like the idea of a simple opinionated K8s BTW good luck!

felixfbecker 8y ago
Yeah this seems weird to me. If your API schema is not introspectable, where do your API clients come from? They don't grow on trees. I guess it's easy to cheat and just use an existing client, but that just shows that this is not a sustainable project
flurdy 8y ago
You can probably argue the same about every part removed, but if you (him) are not going to use it now, remove it. Just dead weight otherwise. You can always add it later if people really need it. True, Swagger docs can be helpful, but also if present but rubbish ignored swagger docs with missing features or side effects is more confusing. Just talk.
- zeptomu 8y ago
  Agreed. I don't really get the love for swagger, everything about it seems so meta. People say you get documentation for free, but auto generating some fancy webpage out of code (or the swagger spec) is not really documentation. You have to write your documentation in a real human language. Maybe I just don't get the point of swagger.

kaivi 8y ago

Hey, I've been using Rancher for a year, and I love it. Just wanted to bring your attention to a YouTube channel from Rancher Labs, where "Learning Rancher" videos are posted regularly. But the content is all the same: boring 10min intro to every video, after which developers go through the same stuff, painfully slowly. Is it meaningful to produce this kind of content over and over again? IMO there are already way better Rancher tutorials by indian dudes on YouTube. I've started with k8s thanks to Rancher, and I presume that the vast majority of Rancher users are doing the same. Rancher without k8s does not make sense, and it seems to me that k8s stuff is actively avoided in docs and tutorials. Maybe it would be better to make a series of tutorials on Rancher+k8s, and go through concrete topics like networking and CI?