10 ms·

Minikube now supports rootless podman driver for running Kubernetes

225 points by encryptluks2 4y ago

anticristi 4y ago
Just to make sure: "rootless" is really misleading. As far as I researched, podman either relies on suid binaries or privileged capabilities or both to do its magic. You might as well call it "capabilitiesful podman driver".
- mishafb 4y ago
  You do need an suid binary to e.g. set a new user id map, since this requires comparing the user id range owned by you to what you're mapping, but you only do it once and it's a simple, secure operation.
- encryptluks2 4y ago
  I don't think it is misleading. Just because you need root privileges to enable "rootless" doesn't mean it isn't rootless once configured.
  - RandomBK 4y ago
    It's somewhere in between. You definitely need to enable features that are normally out-of-reach of regular users (i.e. user namespaces, network namespace, unprivileged ping, etc.) However it's still a far cry from full root access, and arguably a smaller surface area than regular run-everything-as-root mode.
- wronglyprepaid 4y ago
  > podman either relies on suid binaries I'm fairly sure this is only the case on older systems, if your system is up to date then podman should not rely on suid binaries.
- moody5bundle 4y ago
  Maybe you should include this into your "research": - https://opensource.com/article/19/2/how-does-rootless-podman-work https://opensource.com/article/19/2/how-does-rootless-podman... - https://github.com/containers/podman/blob/main/docs/tutorials/rootless_tutorial.md https://github.com/containers/podman/blob/main/docs/tutorial... TL;DR cgroup V2 support Installing Podman Install slirp4netns Ensure fuse-overlayfs is installed
- znpy 4y ago
  you can run containers without root, suid bits or special capabilities with podman. of course, withouth any of that your containers will be able to do very little (eg: no networking).
  - AkihiroSuda 4y ago
    Slirp networking does not need any suid bit or special capability.
Linda703 4y ago
[dead]
jpswade 4y ago
Does this mean it’ll be faster?
- encryptluks2 4y ago
  No I don't think so and in fact rootless containers can be slower due to user-level networking and overlay storage, but the goal is more isolation and security.
cyberpunk 4y ago
Please please please let rke follow suit :)
lunfard000 4y ago
I wish distros would stop making "docker" an alias for "podman", they are not the same thing and breaks all light-k8s implementations.(looking at you redhat)
- candiddevmike 4y ago
  I've encountered cases where podman CLI does not match Docker, specifically for network creation with IPv6--the commands are different. What are you experiencing?
  - lunfard000 4y ago
    CLI is not my issue, k3s and kind wont work with podman (or any rootless container for the matter) out of the box, in both you need to do some non-trivial cgroups configuration on the OS to make it work (in k3s this mode is experimental)
- rubyist5eva 4y ago
  It's an optional package...just don't install it.
baobob 4y ago
While rootless is a curious technical trick I don't understand why the implementation ever left someone's laptop, both file and networking performance are utterly abysmal, which is completely at odds with one of the primary benefits of containers (near zero overhead).
- awoimbee 4y ago
  On servers, yes, rootless doesn't make much sense. But on on my dev laptop, "sudo docker" is tiring and adding docker to the sudoers group is a big security hole (why does everyone seem to think that "docker run" giving root privileges is ok ?!).
  - candiddevmike 4y ago
    You should add yourself to the docker group...
    - rcxdude 4y ago
      which has the same effect, the docker group effectively has root access.
  - Bayart 4y ago
    sudo usermod -aG docker $USER
    - prmoustache 4y ago
      this is not safer.
    - moody5bundle 4y ago
      this is the same as: %wheel ALL=(ALL) NOPASSWD: ALL effectively disabling sudo completely.
  - snorremd 4y ago
    This indeed. The Docker team should not include the "adding your user to the docker group"-section in the install documentation. It is very unsafe and even though they link to a document on security implications I don't think all users will truly grasp the implications. Better to hide this feature and promote the rootless docker mode for local use. On servers you won't be adding any unprivileged user to the docker group in any case.
srvmshr 4y ago
Please excuse my ignorance. I am aware of how Docker generally operates (at a noob level i.e. containerizing an application & uploading to container public registry for general availability). Given that understanding, could someone please explain what "rootless" would mean? I want to understand these in simpler terms:) (Thank you in advance)
- rythie 4y ago
  It means you don't need to be root to run it.
  - coffeekid 4y ago
    You can also call docker commands by being part of the docker group IIRC. Doesn't this have more to do with the daemon that the user executing commands ?
    - q3k 4y ago
      > You can also call docker commands by being part of the docker group IIRC. Which effectively gives you root on the host.
    - prmoustache 4y ago
      Which is an horrible practice and has roughly the same attack surface as login as root all the time.
    - rythie 4y ago
      With podman there is no daemon, everything is running as you. The standard setup for docker has a daemon running as root, which means when you start a container it has root privileges.
- coffeekid 4y ago
  Not an expert at all, but here's how I would simplify it. All corrections are welcome! Docker has two main components. The daemon (you can think of it somewhat like a server) and the client (application you use to run commands). When you install docker on your machine, it generally installs both. The daemon is a process that runs on your local machine and runs as root. Rootless refers to the alternative method (used by podman for instance) to run the daemon as a standard user, and delegate root-level tasks to something else, like systemd for instance.
candiddevmike 4y ago
Are there security issues with user namespaces? For instance, Arch disables them in their hardened Linux kernel: https://wiki.archlinux.org/title/Linux_Containers#Unprivileged_containers_on_linux-hardened_and_custom_kernels https://wiki.archlinux.org/title/Linux_Containers#Unprivileg...
- moody5bundle 4y ago
  cgroups v1 had some issues: https://nvd.nist.gov/vuln/detail/CVE-2022-0492 https://nvd.nist.gov/vuln/detail/CVE-2022-0492 and: https://nvd.nist.gov/vuln/detail/CVE-2022-0185 https://nvd.nist.gov/vuln/detail/CVE-2022-0185
gigatexal 4y ago
A ton of failed tests in that PR and yet it was merged anyway?
technerder 4y ago
Tangential, but are there any easy ways to run server applications on bare metal in a way that removes the need for an underlying OS in order to decreases the overall attack surface an attacker can look for exploits in? (Mainly talking about applications written in Go(TinyGo), Rust, and C++ that can be easily compiled to run on bare metal)
- qbasic_forever 4y ago
  Unikernel is what you're interested in, but it's not as easy as taking some Linux-based server software and spitting out a bootable image for baremetal. If you strip the kernel and OS out you lose the network stack and all kinds of system services that most software depends on directly. I think Google's distroless container images are worth checking out as a quasi-alternative: https://github.com/GoogleContainerTools/distroless https://github.com/GoogleContainerTools/distroless You use them as a base for a docker image and copy in your server code. These images are tailor made to strip out _everything_ that's not necessary to run the software--there's no shell for example. So you're still running a Linux kernel, libc, etc. but there's nothing there for an attacker to use other than your app code. You yourself can't even get into a shell to debug or examine what the state of your app is (which can actually be kind of aggravating in development).
  - mroche 4y ago
    "Distroless" containers are pretty cool for making deployment images. I feel like a better name could have been chosen, because ultimately you are relying on a distribution and how they operate unless you're building an image from scratch and copying in your self-compiled dependencies. I build my own distroless-like images for personal use using Fedora and RHEL, though I do follow the ubi-micro[0] build steps and include a tiny bit of user space components to enable debugging. [0] https://catalog.redhat.com/software/containers/ubi9-micro/61832b36dd607bfc82e66399?container-tabs=dockerfile https://catalog.redhat.com/software/containers/ubi9-micro/61...
- rvdca 4y ago
  From what I gather a unikernel is what you are searching for. Many exists - https://github.com/unikraft/unikraft https://github.com/unikraft/unikraft - https://github.com/hermitcore/rusty-hermit https://github.com/hermitcore/rusty-hermit are the one that comes to my quick search.
gaius_baltar 4y ago
I getting lost in this orchestrator world. Can somebody explain the use case for Minikube? Or Microk8s? All claim to be certified as "perfectly like kubernetes" and show they can be used in production, but are people using them? Why?
- 120photo 4y ago
  Local development of k8s apps without having to deploy to a k8s cluster that you may or may not need to setup.
  - ghaff 4y ago
    Also edge of the network deployments where you want consistency with datacenter deployments but don't have a lot of local compute resources or are otherwise limited.
- liotier 4y ago
  Recent comparison of Minikube, K3s and MicroK8s: https://goalz.online/pros-cons-of-minikube-k3s-microk8s-lightweight-kubernetes-distro/ https://goalz.online/pros-cons-of-minikube-k3s-microk8s-ligh...
  - sascha_sl 4y ago
    Minikube is in a different category, alongside kind. These clusters are meant to be disposabale for development, and at least for kind, can't be updated easily.
- dekhn 4y ago
  I am adding another executor to a workflow engine. Minikube is a huge help for my dev work (I always test against a Local Real Instance, which is what minikube is). It's helped on more than one occasion to show that a prod k8s instance lacked a feature or was misconfigured.
- vbezhenar 4y ago
  Kubernetes is considered too complex to deploy for mere mortals. Here's rough list of things that I had to do, because I'm doing right now exactly that: trying to deploy kubernetes. 1. Prepare server by installing and configuring containerd. Few simple steps. 2. Load one kernel module. Configure ip forward sysctl. 3. Install kubernetes binaries from apt repository. 4. Run kubeadm init 5. Install helm (this part is optional, but makes things simpler): download binary or install it from snap. 6. Install network plugin like flannel or calico. 7. Install nginx ingress plugin. 8. Install storage provider plugin. Also you should have some storage service like NFS server, so Kubernetes can ask for storage. If you want single-node Kubernetes, I think that should be enough. May be #6 is not even necessary. If you want real cluster, you would need to tinker with load balancer for which I don't have clear picture right now. I'm using external load balancer. If you're using docker, my understanding is that containerd is already installed. I actually spent few weeks trying to understand those parts and I have only shallow understanding so far. With that in mind, simple kubernetes solutions probably have its place among those who can't or don't want to use managed kubernetes from popular clouds. I have no idea about those simple kuberneteses though. My opinion is that vanilla kubernetes is not that hard and you should have some understanding about its moving parts anyway. But if you want easy path, I guess it's something worth considering.
aposm 4y ago
This is awesome, was looking around trying to figure out if this was available yet just last night... and here it is!

225 points by encryptluks2 4y ago

Just to make sure: "rootless" is really misleading. As far as I researched, podman either relies on suid binaries or privileged capabilities or both to do its magic. You might as well call it "capabilitiesful podman driver".

mishafb 4y ago
You do need an suid binary to e.g. set a new user id map, since this requires comparing the user id range owned by you to what you're mapping, but you only do it once and it's a simple, secure operation.
encryptluks2 4y ago
I don't think it is misleading. Just because you need root privileges to enable "rootless" doesn't mean it isn't rootless once configured.
- RandomBK 4y ago
  It's somewhere in between. You definitely need to enable features that are normally out-of-reach of regular users (i.e. user namespaces, network namespace, unprivileged ping, etc.) However it's still a far cry from full root access, and arguably a smaller surface area than regular run-everything-as-root mode.
wronglyprepaid 4y ago
> podman either relies on suid binaries I'm fairly sure this is only the case on older systems, if your system is up to date then podman should not rely on suid binaries.
moody5bundle 4y ago
Maybe you should include this into your "research": - https://opensource.com/article/19/2/how-does-rootless-podman-work https://opensource.com/article/19/2/how-does-rootless-podman... - https://github.com/containers/podman/blob/main/docs/tutorials/rootless_tutorial.md https://github.com/containers/podman/blob/main/docs/tutorial... TL;DR cgroup V2 support Installing Podman Install slirp4netns Ensure fuse-overlayfs is installed
znpy 4y ago
you can run containers without root, suid bits or special capabilities with podman. of course, withouth any of that your containers will be able to do very little (eg: no networking).
- AkihiroSuda 4y ago
  Slirp networking does not need any suid bit or special capability.

Linda703 4y ago

[dead]

jpswade 4y ago

Does this mean it’ll be faster?

encryptluks2 4y ago
No I don't think so and in fact rootless containers can be slower due to user-level networking and overlay storage, but the goal is more isolation and security.

cyberpunk 4y ago

Please please please let rke follow suit :)

lunfard000 4y ago

I wish distros would stop making "docker" an alias for "podman", they are not the same thing and breaks all light-k8s implementations.(looking at you redhat)

candiddevmike 4y ago
I've encountered cases where podman CLI does not match Docker, specifically for network creation with IPv6--the commands are different. What are you experiencing?
- lunfard000 4y ago
  CLI is not my issue, k3s and kind wont work with podman (or any rootless container for the matter) out of the box, in both you need to do some non-trivial cgroups configuration on the OS to make it work (in k3s this mode is experimental)
rubyist5eva 4y ago
It's an optional package...just don't install it.

baobob 4y ago

While rootless is a curious technical trick I don't understand why the implementation ever left someone's laptop, both file and networking performance are utterly abysmal, which is completely at odds with one of the primary benefits of containers (near zero overhead).

awoimbee 4y ago
On servers, yes, rootless doesn't make much sense. But on on my dev laptop, "sudo docker" is tiring and adding docker to the sudoers group is a big security hole (why does everyone seem to think that "docker run" giving root privileges is ok ?!).
- candiddevmike 4y ago
  You should add yourself to the docker group...
  - rcxdude 4y ago
    which has the same effect, the docker group effectively has root access.
- Bayart 4y ago
  sudo usermod -aG docker $USER
  - prmoustache 4y ago
    this is not safer.
  - moody5bundle 4y ago
    this is the same as: %wheel ALL=(ALL) NOPASSWD: ALL effectively disabling sudo completely.
- snorremd 4y ago
  This indeed. The Docker team should not include the "adding your user to the docker group"-section in the install documentation. It is very unsafe and even though they link to a document on security implications I don't think all users will truly grasp the implications. Better to hide this feature and promote the rootless docker mode for local use. On servers you won't be adding any unprivileged user to the docker group in any case.

srvmshr 4y ago

Please excuse my ignorance. I am aware of how Docker generally operates (at a noob level i.e. containerizing an application & uploading to container public registry for general availability). Given that understanding, could someone please explain what "rootless" would mean? I want to understand these in simpler terms:) (Thank you in advance)

rythie 4y ago
It means you don't need to be root to run it.
- coffeekid 4y ago
  You can also call docker commands by being part of the docker group IIRC. Doesn't this have more to do with the daemon that the user executing commands ?
  - q3k 4y ago
    > You can also call docker commands by being part of the docker group IIRC. Which effectively gives you root on the host.
  - prmoustache 4y ago
    Which is an horrible practice and has roughly the same attack surface as login as root all the time.
  - rythie 4y ago
    With podman there is no daemon, everything is running as you. The standard setup for docker has a daemon running as root, which means when you start a container it has root privileges.
coffeekid 4y ago
Not an expert at all, but here's how I would simplify it. All corrections are welcome! Docker has two main components. The daemon (you can think of it somewhat like a server) and the client (application you use to run commands). When you install docker on your machine, it generally installs both. The daemon is a process that runs on your local machine and runs as root. Rootless refers to the alternative method (used by podman for instance) to run the daemon as a standard user, and delegate root-level tasks to something else, like systemd for instance.

candiddevmike 4y ago

Are there security issues with user namespaces? For instance, Arch disables them in their hardened Linux kernel: https://wiki.archlinux.org/title/Linux_Containers#Unprivileged_containers_on_linux-hardened_and_custom_kernels https://wiki.archlinux.org/title/Linux_Containers#Unprivileg...

moody5bundle 4y ago
cgroups v1 had some issues: https://nvd.nist.gov/vuln/detail/CVE-2022-0492 https://nvd.nist.gov/vuln/detail/CVE-2022-0492 and: https://nvd.nist.gov/vuln/detail/CVE-2022-0185 https://nvd.nist.gov/vuln/detail/CVE-2022-0185

gigatexal 4y ago

A ton of failed tests in that PR and yet it was merged anyway?

technerder 4y ago

Tangential, but are there any easy ways to run server applications on bare metal in a way that removes the need for an underlying OS in order to decreases the overall attack surface an attacker can look for exploits in? (Mainly talking about applications written in Go(TinyGo), Rust, and C++ that can be easily compiled to run on bare metal)

qbasic_forever 4y ago
Unikernel is what you're interested in, but it's not as easy as taking some Linux-based server software and spitting out a bootable image for baremetal. If you strip the kernel and OS out you lose the network stack and all kinds of system services that most software depends on directly. I think Google's distroless container images are worth checking out as a quasi-alternative: https://github.com/GoogleContainerTools/distroless https://github.com/GoogleContainerTools/distroless You use them as a base for a docker image and copy in your server code. These images are tailor made to strip out _everything_ that's not necessary to run the software--there's no shell for example. So you're still running a Linux kernel, libc, etc. but there's nothing there for an attacker to use other than your app code. You yourself can't even get into a shell to debug or examine what the state of your app is (which can actually be kind of aggravating in development).
- mroche 4y ago
  "Distroless" containers are pretty cool for making deployment images. I feel like a better name could have been chosen, because ultimately you are relying on a distribution and how they operate unless you're building an image from scratch and copying in your self-compiled dependencies. I build my own distroless-like images for personal use using Fedora and RHEL, though I do follow the ubi-micro[0] build steps and include a tiny bit of user space components to enable debugging. [0] https://catalog.redhat.com/software/containers/ubi9-micro/61832b36dd607bfc82e66399?container-tabs=dockerfile https://catalog.redhat.com/software/containers/ubi9-micro/61...
rvdca 4y ago
From what I gather a unikernel is what you are searching for. Many exists - https://github.com/unikraft/unikraft https://github.com/unikraft/unikraft - https://github.com/hermitcore/rusty-hermit https://github.com/hermitcore/rusty-hermit are the one that comes to my quick search.

gaius_baltar 4y ago

I getting lost in this orchestrator world. Can somebody explain the use case for Minikube? Or Microk8s? All claim to be certified as "perfectly like kubernetes" and show they can be used in production, but are people using them? Why?

120photo 4y ago
Local development of k8s apps without having to deploy to a k8s cluster that you may or may not need to setup.
- ghaff 4y ago
  Also edge of the network deployments where you want consistency with datacenter deployments but don't have a lot of local compute resources or are otherwise limited.
liotier 4y ago
Recent comparison of Minikube, K3s and MicroK8s: https://goalz.online/pros-cons-of-minikube-k3s-microk8s-lightweight-kubernetes-distro/ https://goalz.online/pros-cons-of-minikube-k3s-microk8s-ligh...
- sascha_sl 4y ago
  Minikube is in a different category, alongside kind. These clusters are meant to be disposabale for development, and at least for kind, can't be updated easily.
dekhn 4y ago
I am adding another executor to a workflow engine. Minikube is a huge help for my dev work (I always test against a Local Real Instance, which is what minikube is). It's helped on more than one occasion to show that a prod k8s instance lacked a feature or was misconfigured.
vbezhenar 4y ago
Kubernetes is considered too complex to deploy for mere mortals. Here's rough list of things that I had to do, because I'm doing right now exactly that: trying to deploy kubernetes. 1. Prepare server by installing and configuring containerd. Few simple steps. 2. Load one kernel module. Configure ip forward sysctl. 3. Install kubernetes binaries from apt repository. 4. Run kubeadm init 5. Install helm (this part is optional, but makes things simpler): download binary or install it from snap. 6. Install network plugin like flannel or calico. 7. Install nginx ingress plugin. 8. Install storage provider plugin. Also you should have some storage service like NFS server, so Kubernetes can ask for storage. If you want single-node Kubernetes, I think that should be enough. May be #6 is not even necessary. If you want real cluster, you would need to tinker with load balancer for which I don't have clear picture right now. I'm using external load balancer. If you're using docker, my understanding is that containerd is already installed. I actually spent few weeks trying to understand those parts and I have only shallow understanding so far. With that in mind, simple kubernetes solutions probably have its place among those who can't or don't want to use managed kubernetes from popular clouds. I have no idea about those simple kuberneteses though. My opinion is that vanilla kubernetes is not that hard and you should have some understanding about its moving parts anyway. But if you want easy path, I guess it's something worth considering.

aposm 4y ago

This is awesome, was looking around trying to figure out if this was available yet just last night... and here it is!