8 ms·

How to use Podman inside of a container (2021)

136 points by teleforce 4y ago

debarshri 4y ago
If you thought container in container was cool. How about k8s in container? Do checkout kind [1] [1] https://kind.sigs.k8s.io/ https://kind.sigs.k8s.io/
- spyremeown 4y ago
  Kind is super cool! I used it for my thesis, as I needed many nodes without getting super into k8s configuration. I love, love kind. If you're into learning Kubernetes, it's probably the easiest way to get yourself a cluster quickly.
deleted 4y ago
[deleted]
devjab 4y ago
Yo dawg, I heard you liked containers so I put a container inside your container! In all seriousness though, I wonder if this could solve one of our issues of running mutli-container apps in Azure, which is currently causing us some trouble in regards to VNETs and Private Endpoints. I’m sure it’s likely because we should be using Kubernetes or similar, or at the very least have someone who knows how to actually set the Ops part of DevOps up better than we currently do. Long story short, those things won’t realistically happen, and if we could solve them by running multiple podmans inside a docker docker-container deployed as an azure app then we could likely fix our security issues of not being able to setup VNETs and PEs on multi-container apps because it’s been a “up-coming” feature for at least a year. I’d be interested to know your thoughts on it. The efficiency doesn’t matter as much, our workloads are around 1-5% of our Azure budget which is basically a blip on the bigger budgets.
- hnlmorg 4y ago
  I’ve used Consul for instances where I’ve needed some of the benefits of k8s but without all the overhead of running k8s. I’m fact currently using Consul for one such problem as it happens: managing Docker from inside Docker.
  - sofixa 4y ago
    Are you sure you mean Consul and not Nomad (or are you talking about some other Consul?)? In either case, could you elaborate how you do it?
    - stanac 4y ago
      Maybe they just need service discovery and load balancing (as in replacement for kubernetes service). I may be wrong and would like to here more too.
- mhio 4y ago
  Are you running the multi-container apps under App Service?
  - devjab 4y ago
    Yes, with the docker compose option.
    - mhio 4y ago
      App Service won't allow the level of privilege needed to start the "head" container that enables container-in-container to work. Options like `--privileged` and `--cap SYS_ADMIN` or mounting special volumes from the host are the first step on "how to escape from a container" so shared container services don't allow them. The options might be more allowable in your own App Service Environment but I don't think the API is even there to set the options in the first place. Container-in-container also pushes a number of responsibilities from the the App Service down into the head container, like ingress, service management, health, logging etc. You might as well run compose/swarm on your own VM(s) than reinvent those wheels. AKS + kompose might be a not too bad option. AKS can deploy on the private vnet and can do private endpoints. The cluster will probably fall over once a year for <reasons> but it will mostly manage itself. If you leave k8s to auto upgrade, run a microk8s instance as a test env somewhere and it will hit the upgrade issues before AKS releases a k8s version. - https://github.com/kubernetes/kompose https://github.com/kubernetes/kompose - https://learn.microsoft.com/en-us/azure/aks/private-clusters#use-a-private-endpoint-connection https://learn.microsoft.com/en-us/azure/aks/private-clusters...
- evntdrvn 4y ago
  are you using Azure Container Apps?
  - devjab 4y ago
    Do you mean their vendor-lock-in but much easier to use kubernetes? We can’t because of EU laws prohibiting us from that kind of vendor lock-in.
LeoPanthera 4y ago
It’s containers all the way down.
- BossingAround 4y ago
  It's Linux all the way down. Or, alternatively, it's processes all the way down :).
rexsteroxxy 4y ago
> Most of this has historically been related to Docker in Docker (DIND), but now, people also want to run Podman in Podman (PINP) or Podman in Docker (PIND). So close to some acronym puns :D
RandomBK 4y ago
This is very useful for running containers in lightweight untrusted environments. I personally use this setup for my GitLab CI runners, using a mix of podman and buildah to build containers. I can't 100% trust the container build script, so avoiding having to expose a Docker socket to that untrusted code is a critical safety layer.
- storrgie 4y ago
  Do you have a writeup on this configuration?
sgsag33 4y ago
What is the use case for a container in a container?
- movedx 4y ago
  Over complicating the engineering (Docker) involved in getting some x86 code into RAM and on a CPU stack, by over complicating further (DIND.) It's essentially another way to abstract and abstraction, and engineers are finding ways to justify it.
  - spyremeown 4y ago
    It's not really over complicating anything, though? It makes perfect sense, it's not hard to understand. Nested VMs have been a thing for a looong time. I honestly don't understand where the criticism for Docker and k8s comes from. Some of HN comments sound like old men shouting at the clouds because they don't understand modern technology and its purposes. If you want, go right ahead with your managed, on-site VMs where you copy a .php file to an Apache server using a thumb drive and restart the service. I don't care. But that's not how modern teams are working.
    - movedx 4y ago
      > But that's not how modern teams are working. I'd suggest you do more research. Plenty of teams in (very) big companies, handling (very) large workloads not doing what you'd want them to. Docker? Docker is ideal for teams deploying quickly evolving (possible stateless) applications across multiple platforms. If you have no need to deploy across multiple platforms, simply deploy directly to your OS using the best primitives possible. Kubernetes? Excellent for when you need to do multiple, rapid deployments per day of small (ideally stateless) services. A good example of this is small services that serve up adverts in response to some "profile" of a visitor sent to it from a browser - these things evolve a lot, and often. Abstracting your abstractions because you don't want to learn how-to make the OS better or use what's already there isn't clever. Learning to use the tools is the important thing, and Docker and K8s are just tools.
- 4y ago
onesphere 4y ago
Here's how I package my system database as a user-access object: https://thetaplane.com/runtime/synthetic-database https://thetaplane.com/runtime/synthetic-database
codethief 4y ago
I realize that Linux wasn't made for containerization originally and that's why it's so so painful to nest containers, run them without root etc. But I wish we'd already live in a world where all binaries (maybe with the exception of command line tools) ran containerized by default and I, as the user, had to grant explicit access to files, network, hardware/devices, etc. Not just for security reasons (though it'd be nice if `npm install` and `pip install` couldn't mess up my entire system) but also to make external dependencies more explicit.
- arcanemachiner 4y ago
  `pipx` seems to be a good solution for containing Python dependency clutter.
- dilyevsky 4y ago
  It's not so painful and most pain comes from Linux insistence on /proc-based APIs as opposed to something similar to Solaris-descendants where it's a json spec (I think xml originally)
- PhilippGille 4y ago
  Isn't that what Snap and Flatpak are doing? 1) Include dependencies (there can be shared runtimes but they don't mess up the host), 2) sandboxed by default and you need to grant permissions. E.g. Snap sandboxing: https://snapcraft.io/docs/security-sandboxing https://snapcraft.io/docs/security-sandboxing
  - ElectricalUnion 4y ago
    It mostly works except for the part 2 failing miserably; so many apps require full filesystem access to have sane behaviour it's not even funny. It doesn't help a lot that instead of having a explicit "can I have a host file" API, the apps are supposed to use a framework that invokes "portals".
- kytazo 4y ago
  Sounds like android or what a modern linux distro should look like. Sadly both most used userspace distros and the kernel they come with are borderline dangerous and shouldn't be any trusted by people using them, even if they're technical enough to feel comfortable within their environment.
user3939382 4y ago
Until podman could be used with AWS ECR/ECS it's pretty much moot in my case: https://github.com/aws/containers-roadmap/issues/626 https://github.com/aws/containers-roadmap/issues/626
eriksjolund 4y ago
I just tried this out. The new systemd directive OpenFile= opens up the possibility to pass the file descriptor of a file from the host to a container running in a container. (using rootless Podman running rootless Podman) sudo systemd-run --property User=test --property OpenFile=/etc/secretfile.txt --collect --pipe --wait --quiet podman run --security-opt label=disable --user podman --device /dev/fuse quay.io/podman/stable podman run -q alpine sh -c "cat <&3"

136 points by teleforce 4y ago

If you thought container in container was cool. How about k8s in container? Do checkout kind [1] [1] https://kind.sigs.k8s.io/ https://kind.sigs.k8s.io/

spyremeown 4y ago
Kind is super cool! I used it for my thesis, as I needed many nodes without getting super into k8s configuration. I love, love kind. If you're into learning Kubernetes, it's probably the easiest way to get yourself a cluster quickly.

deleted 4y ago

[deleted]

devjab 4y ago

Yo dawg, I heard you liked containers so I put a container inside your container! In all seriousness though, I wonder if this could solve one of our issues of running mutli-container apps in Azure, which is currently causing us some trouble in regards to VNETs and Private Endpoints. I’m sure it’s likely because we should be using Kubernetes or similar, or at the very least have someone who knows how to actually set the Ops part of DevOps up better than we currently do. Long story short, those things won’t realistically happen, and if we could solve them by running multiple podmans inside a docker docker-container deployed as an azure app then we could likely fix our security issues of not being able to setup VNETs and PEs on multi-container apps because it’s been a “up-coming” feature for at least a year. I’d be interested to know your thoughts on it. The efficiency doesn’t matter as much, our workloads are around 1-5% of our Azure budget which is basically a blip on the bigger budgets.

hnlmorg 4y ago
I’ve used Consul for instances where I’ve needed some of the benefits of k8s but without all the overhead of running k8s. I’m fact currently using Consul for one such problem as it happens: managing Docker from inside Docker.
- sofixa 4y ago
  Are you sure you mean Consul and not Nomad (or are you talking about some other Consul?)? In either case, could you elaborate how you do it?
  - stanac 4y ago
    Maybe they just need service discovery and load balancing (as in replacement for kubernetes service). I may be wrong and would like to here more too.
mhio 4y ago
Are you running the multi-container apps under App Service?
- devjab 4y ago
  Yes, with the docker compose option.
  - mhio 4y ago
    App Service won't allow the level of privilege needed to start the "head" container that enables container-in-container to work. Options like `--privileged` and `--cap SYS_ADMIN` or mounting special volumes from the host are the first step on "how to escape from a container" so shared container services don't allow them. The options might be more allowable in your own App Service Environment but I don't think the API is even there to set the options in the first place. Container-in-container also pushes a number of responsibilities from the the App Service down into the head container, like ingress, service management, health, logging etc. You might as well run compose/swarm on your own VM(s) than reinvent those wheels. AKS + kompose might be a not too bad option. AKS can deploy on the private vnet and can do private endpoints. The cluster will probably fall over once a year for <reasons> but it will mostly manage itself. If you leave k8s to auto upgrade, run a microk8s instance as a test env somewhere and it will hit the upgrade issues before AKS releases a k8s version. - https://github.com/kubernetes/kompose https://github.com/kubernetes/kompose - https://learn.microsoft.com/en-us/azure/aks/private-clusters#use-a-private-endpoint-connection https://learn.microsoft.com/en-us/azure/aks/private-clusters...
evntdrvn 4y ago
are you using Azure Container Apps?
- devjab 4y ago
  Do you mean their vendor-lock-in but much easier to use kubernetes? We can’t because of EU laws prohibiting us from that kind of vendor lock-in.

LeoPanthera 4y ago

It’s containers all the way down.

BossingAround 4y ago
It's Linux all the way down. Or, alternatively, it's processes all the way down :).

rexsteroxxy 4y ago

> Most of this has historically been related to Docker in Docker (DIND), but now, people also want to run Podman in Podman (PINP) or Podman in Docker (PIND). So close to some acronym puns :D

RandomBK 4y ago

This is very useful for running containers in lightweight untrusted environments. I personally use this setup for my GitLab CI runners, using a mix of podman and buildah to build containers. I can't 100% trust the container build script, so avoiding having to expose a Docker socket to that untrusted code is a critical safety layer.

storrgie 4y ago
Do you have a writeup on this configuration?

sgsag33 4y ago

What is the use case for a container in a container?

movedx 4y ago
Over complicating the engineering (Docker) involved in getting some x86 code into RAM and on a CPU stack, by over complicating further (DIND.) It's essentially another way to abstract and abstraction, and engineers are finding ways to justify it.
- spyremeown 4y ago
  It's not really over complicating anything, though? It makes perfect sense, it's not hard to understand. Nested VMs have been a thing for a looong time. I honestly don't understand where the criticism for Docker and k8s comes from. Some of HN comments sound like old men shouting at the clouds because they don't understand modern technology and its purposes. If you want, go right ahead with your managed, on-site VMs where you copy a .php file to an Apache server using a thumb drive and restart the service. I don't care. But that's not how modern teams are working.
  - movedx 4y ago
    > But that's not how modern teams are working. I'd suggest you do more research. Plenty of teams in (very) big companies, handling (very) large workloads not doing what you'd want them to. Docker? Docker is ideal for teams deploying quickly evolving (possible stateless) applications across multiple platforms. If you have no need to deploy across multiple platforms, simply deploy directly to your OS using the best primitives possible. Kubernetes? Excellent for when you need to do multiple, rapid deployments per day of small (ideally stateless) services. A good example of this is small services that serve up adverts in response to some "profile" of a visitor sent to it from a browser - these things evolve a lot, and often. Abstracting your abstractions because you don't want to learn how-to make the OS better or use what's already there isn't clever. Learning to use the tools is the important thing, and Docker and K8s are just tools.
4y ago

onesphere 4y ago

Here's how I package my system database as a user-access object: https://thetaplane.com/runtime/synthetic-database https://thetaplane.com/runtime/synthetic-database

codethief 4y ago

I realize that Linux wasn't made for containerization originally and that's why it's so so painful to nest containers, run them without root etc. But I wish we'd already live in a world where all binaries (maybe with the exception of command line tools) ran containerized by default and I, as the user, had to grant explicit access to files, network, hardware/devices, etc. Not just for security reasons (though it'd be nice if `npm install` and `pip install` couldn't mess up my entire system) but also to make external dependencies more explicit.

arcanemachiner 4y ago
`pipx` seems to be a good solution for containing Python dependency clutter.
dilyevsky 4y ago
It's not so painful and most pain comes from Linux insistence on /proc-based APIs as opposed to something similar to Solaris-descendants where it's a json spec (I think xml originally)
PhilippGille 4y ago
Isn't that what Snap and Flatpak are doing? 1) Include dependencies (there can be shared runtimes but they don't mess up the host), 2) sandboxed by default and you need to grant permissions. E.g. Snap sandboxing: https://snapcraft.io/docs/security-sandboxing https://snapcraft.io/docs/security-sandboxing
- ElectricalUnion 4y ago
  It mostly works except for the part 2 failing miserably; so many apps require full filesystem access to have sane behaviour it's not even funny. It doesn't help a lot that instead of having a explicit "can I have a host file" API, the apps are supposed to use a framework that invokes "portals".
kytazo 4y ago
Sounds like android or what a modern linux distro should look like. Sadly both most used userspace distros and the kernel they come with are borderline dangerous and shouldn't be any trusted by people using them, even if they're technical enough to feel comfortable within their environment.

user3939382 4y ago

Until podman could be used with AWS ECR/ECS it's pretty much moot in my case: https://github.com/aws/containers-roadmap/issues/626 https://github.com/aws/containers-roadmap/issues/626

eriksjolund 4y ago

I just tried this out. The new systemd directive OpenFile= opens up the possibility to pass the file descriptor of a file from the host to a container running in a container. (using rootless Podman running rootless Podman) sudo systemd-run --property User=test --property OpenFile=/etc/secretfile.txt --collect --pipe --wait --quiet podman run --security-opt label=disable --user podman --device /dev/fuse quay.io/podman/stable podman run -q alpine sh -c "cat <&3"