5 ms·

Run Docker containers natively in Proxmox 9.1 (OCI images)

147 points by jandeboevrie 11mo ago

k__ 11mo ago
Is this similar to what FlyIO is doing? Running containers as microVMs?
- indigodaddy 11mo ago
  Perhaps in spirit? But I don't think you can term LXC a microVM, and I doubt they start close to as fast as Firecracker or smolbsd, and similar ilk. EDIT - appears I am probably wrong about firecracker being faster than LXC as LXC is kernel based virtualization and likely has faster startup than microVMs?
  - esseph 11mo ago
    Firecracker would start faster, lxc would perform better. Firecracker should have better actual isolation... I think.
nirav72 11mo ago
I played with this a bit today. Only downside is, no easy way to update containers yet. But on the other hand, no more dealing with macvlan or custom docker networks.
- dijit 11mo ago
  “update”, I assume you mean “recreate with new image”? I think docker itself doesn’t support that.
  - doubled112 11mo ago
    I use Docker compose to recreate containers with a new image regularly. I'm sure you could be creative with volumes in Proxmox and build a new LXC container from a new OCI image with the old volumes attached.
    - dijit 11mo ago
      > I use Docker compose to recreate containers with a new image regularly. try doing so without the compose file though.
      - doubled112 11mo ago
        That's true, isn't it? It was one of those features you'd think they would have had figured out, but no.
        prmoustache 11mo ago
        Isn't the ability to do blue/green deployments, canary releases and easy rollbacks huge incentives to use containers? I think virtually nobody cares about being able to change the image of a container when you can so easily start a new one.
        formerly_proven 11mo ago
        People figuring out how to use containers as pets.
        esseph 11mo ago
        * blue/green deployments * canary releases * easy rollback Have never needed containers to do any of these things.
        prmoustache 11mo ago
        Has anyone said that?
        esseph 11mo ago
        If we could already do it with some loadbalancer changes, I don't understand your comment that it was an incentive to move to containers. Containers are separate from their deployment method. To be able to do those things with containers, some will go to docker, docker swarm, hashicorp nomad, or kubernetes. So if people could already do these deployment methods, and given the HUGE organizational lift in training and platform investment for Ops to do that shift, your comment about those reasons being incentives to move to containers doesn't make sense.
_ache_ 11mo ago
I have an "error" "I am not a teapot" 719 - I am not a teapot Espresso Web (Red Hat Enterprise Linux) at raymii.org Looks suspicious, ... not 418, 719.
- radiator 11mo ago
  I think 418 is 'I am a teapot' so it would not be correct to use it in your case. 719 must be a typo though, perhaps it should be 419.
- squigz 11mo ago
  Haha, this was funny. https://datatracker.ietf.org/doc/html/rfc2324 https://datatracker.ietf.org/doc/html/rfc2324
dizhn 11mo ago
They are converted to LXC images then run. No compose file either. Still pretty neat.
caymanjim 11mo ago
It's unclear to me why running Docker directly in Proxmox (it's just Debian) and using it like any other Docker host is a bad idea, and why this extra layer of abstractions is preferable. Docker has security issues if you're not careful, and it's frankly kind of a shitshow out of the box with defaults. Maybe that's part of the reason. But I struggle to see how a bespoke solution like this is the right answer.
- simcop2387 11mo ago
  Largely management, observability, and then the way that docker mucks with firewalls. Running them this way will allow proxmox to handle all that in the same way {I assume) as the LXC and VMS so automation, and all the rest can be consistent
- zatarc 11mo ago
  I've been running Docker natively on the host since Proxmox 7. The only major problem was an iptables rule that I had to add so that the containers are accessible from outside. Besides that, it runs smoothly.
- tristanj 11mo ago
  Proxmox is a hypervisor OS, and its value comes from its virtualization and container-management features. These features include being able to pause, resume, snapshot, backup/restore from snapshot, and live-migrate VMs or LXCs to another server in just a couple hundred milliseconds of downtime. Once you run docker on the hypervisor itself, you lose these features, which defeats the purpose of running Proxmox in the first place. There's also the security angle. Containers managed by Proxmox are strongly isolated from the host, but containers running on Docker sidestep this isolation model. Docker is not insecure by design, but it greatly increases the attack surface. If the hypervisor gets compromised, the entire cluster of servers will also get compromised. In general, as little software as possible should be installed on the host.
  - nicman23 11mo ago
    still then it would have been just a process in a namespace. there are ways to dump a process and then resume it
haunter 11mo ago
This is something I've always loved about Unraid. The whole apps/containers ecosystem is so well done.
vladgur 11mo ago
#TIL Proxmox 9.1 is out. Im still on 8.x -- it was a fun way to consolidate my different hacky projects -- home assistant, frigate, wireguard, qbittorrent etc Kinda scared to think of what it would take to upgrade to 9.1 :) https://news.ycombinator.com/item?id=45980005 https://news.ycombinator.com/item?id=45980005
- HolyLampshade 11mo ago
  For what it’s worth I went through the upgrade last weekend. There is a compatibility check script and, frankly, the whole process proxmox had described on their site worked precisely as advertised. 5 host cluster; rebooted them all at completion and all of the containers came back up without issue (combination of VMs and LXC)
- ehaughee 11mo ago
  My homelab upgrade from 8.x to 9.x was pretty smooth from following their upgrade guide[1]. I just upgraded from 9.0 to 9.1 this morning without any issues. [1] https://pve.proxmox.com/wiki/Upgrade_from_8_to_9 https://pve.proxmox.com/wiki/Upgrade_from_8_to_9
  - gregoryw3 11mo ago
    Do you happen to know if they fixed the memory ballooning issue?
    - polski-g 11mo ago
      What issue?
- gerdesj 11mo ago
  Do this: https://pve.proxmox.com/wiki/Upgrade_from_8_to_9 https://pve.proxmox.com/wiki/Upgrade_from_8_to_9 No need to overthink it too much. I've done several so far. Run the pve8to9 script first to do some sanity checks (it should already be installed if the system is up to date). Update the box to latest 8.x with apt update etc. Change the package sources to the new ones and update the system. The packages databases can be a bit confusing: You have two lots - stock Debian and Proxmox (enterprise OR no-subscription). Stock Debian is in the single file /etc/apt/sources.list - change "bookworm" to "trixie". Proxmox sources is in a file in /etc/apt/sources.list.d/ Remove all of the Proxmox related ones you have there and run this (or do it yourself with an editor). This example is no-sub - the official doc notes the enterprise equivalent: cat > /etc/apt/sources.list.d/proxmox.sources << EOF Types: deb URIs: http://download.proxmox.com/debian/pve Suites: trixie Components: pve-no-subscription Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg EOF Run apt dist-upgrade then the pve8to9 script again and then reboot. If in doubt choose Y for install the maintainer's version when prompted. There are notes in the doc about several packages. Job done.
js2 11mo ago
I've been running docker apps via podman inside alpine containers on Proxmox. Works for me with homebridge, scrypted, and plex. I created an alpine container per app with "nesting=1". Inside each alpine container I installed podman (`apk install podman`). Then it's a couple init scripts inside each alpine container, one to work-around a cgroups issue, the other to start the app via podman: https://pastebin.com/raw/W8YPKGjG https://pastebin.com/raw/W8YPKGjG
- Havoc 11mo ago
  Yeah same here except arch. Bit fiddly with bind mount permissions but otherwise works pretty well
OCTAGRAM 11mo ago
I have updated and tried to run something. There are apparently three-chunk ids and two-chunk ids. two-chunk ids work. Three-chunk ids don't. atdr.meo.ws/archiveteam/reddit-grab I cannot install this one to Proxmox VE, for instance UPD. Query tag fails, but fetch is successful if I write "latest" tag
btreecat 11mo ago
I run a kube cluster instead, because I want to build transferrable skills, using the standard open tools. I don't think I've ever seen anyone mention using proxmox in a professional context. None the less, nice progress I reckon.
tracker1 11mo ago
I never much minded dedicating a (or some) VM to it anyway, let me assign a dedicated IP. My email stack (mailu) runs best with a dedicated IP to the system, similar for my BBS setup (though broken atm)... For web apps, I run them all in a single Ubuntu Server VM with docker, ufw and caddy on the host OS. I run a similar setup with a few VMs on a mini pc at home as well... It all works well enough for what I need. Lets me somewhat isolate the containers VM from other purpose-specific VMs.

147 points by jandeboevrie 11mo ago

k__ 11mo ago

Is this similar to what FlyIO is doing? Running containers as microVMs?

indigodaddy 11mo ago
Perhaps in spirit? But I don't think you can term LXC a microVM, and I doubt they start close to as fast as Firecracker or smolbsd, and similar ilk. EDIT - appears I am probably wrong about firecracker being faster than LXC as LXC is kernel based virtualization and likely has faster startup than microVMs?
- esseph 11mo ago
  Firecracker would start faster, lxc would perform better. Firecracker should have better actual isolation... I think.

nirav72 11mo ago

I played with this a bit today. Only downside is, no easy way to update containers yet. But on the other hand, no more dealing with macvlan or custom docker networks.

dijit 11mo ago
“update”, I assume you mean “recreate with new image”? I think docker itself doesn’t support that.
- doubled112 11mo ago
  I use Docker compose to recreate containers with a new image regularly. I'm sure you could be creative with volumes in Proxmox and build a new LXC container from a new OCI image with the old volumes attached.
  - dijit 11mo ago
    > I use Docker compose to recreate containers with a new image regularly. try doing so without the compose file though.
    - doubled112 11mo ago
      That's true, isn't it? It was one of those features you'd think they would have had figured out, but no.
      - prmoustache 11mo ago
        Isn't the ability to do blue/green deployments, canary releases and easy rollbacks huge incentives to use containers? I think virtually nobody cares about being able to change the image of a container when you can so easily start a new one.
        formerly_proven 11mo ago
        People figuring out how to use containers as pets.
        esseph 11mo ago
        * blue/green deployments * canary releases * easy rollback Have never needed containers to do any of these things.
        prmoustache 11mo ago
        Has anyone said that?
        esseph 11mo ago
        If we could already do it with some loadbalancer changes, I don't understand your comment that it was an incentive to move to containers. Containers are separate from their deployment method. To be able to do those things with containers, some will go to docker, docker swarm, hashicorp nomad, or kubernetes. So if people could already do these deployment methods, and given the HUGE organizational lift in training and platform investment for Ops to do that shift, your comment about those reasons being incentives to move to containers doesn't make sense.

_ache_ 11mo ago

I have an "error" "I am not a teapot" 719 - I am not a teapot Espresso Web (Red Hat Enterprise Linux) at raymii.org Looks suspicious, ... not 418, 719.

radiator 11mo ago
I think 418 is 'I am a teapot' so it would not be correct to use it in your case. 719 must be a typo though, perhaps it should be 419.
squigz 11mo ago
Haha, this was funny. https://datatracker.ietf.org/doc/html/rfc2324 https://datatracker.ietf.org/doc/html/rfc2324

dizhn 11mo ago

They are converted to LXC images then run. No compose file either. Still pretty neat.

caymanjim 11mo ago

It's unclear to me why running Docker directly in Proxmox (it's just Debian) and using it like any other Docker host is a bad idea, and why this extra layer of abstractions is preferable. Docker has security issues if you're not careful, and it's frankly kind of a shitshow out of the box with defaults. Maybe that's part of the reason. But I struggle to see how a bespoke solution like this is the right answer.

simcop2387 11mo ago
Largely management, observability, and then the way that docker mucks with firewalls. Running them this way will allow proxmox to handle all that in the same way {I assume) as the LXC and VMS so automation, and all the rest can be consistent
zatarc 11mo ago
I've been running Docker natively on the host since Proxmox 7. The only major problem was an iptables rule that I had to add so that the containers are accessible from outside. Besides that, it runs smoothly.
tristanj 11mo ago
Proxmox is a hypervisor OS, and its value comes from its virtualization and container-management features. These features include being able to pause, resume, snapshot, backup/restore from snapshot, and live-migrate VMs or LXCs to another server in just a couple hundred milliseconds of downtime. Once you run docker on the hypervisor itself, you lose these features, which defeats the purpose of running Proxmox in the first place. There's also the security angle. Containers managed by Proxmox are strongly isolated from the host, but containers running on Docker sidestep this isolation model. Docker is not insecure by design, but it greatly increases the attack surface. If the hypervisor gets compromised, the entire cluster of servers will also get compromised. In general, as little software as possible should be installed on the host.
- nicman23 11mo ago
  still then it would have been just a process in a namespace. there are ways to dump a process and then resume it

haunter 11mo ago

This is something I've always loved about Unraid. The whole apps/containers ecosystem is so well done.

vladgur 11mo ago

#TIL Proxmox 9.1 is out. Im still on 8.x -- it was a fun way to consolidate my different hacky projects -- home assistant, frigate, wireguard, qbittorrent etc Kinda scared to think of what it would take to upgrade to 9.1 :) https://news.ycombinator.com/item?id=45980005 https://news.ycombinator.com/item?id=45980005

HolyLampshade 11mo ago
For what it’s worth I went through the upgrade last weekend. There is a compatibility check script and, frankly, the whole process proxmox had described on their site worked precisely as advertised. 5 host cluster; rebooted them all at completion and all of the containers came back up without issue (combination of VMs and LXC)
ehaughee 11mo ago
My homelab upgrade from 8.x to 9.x was pretty smooth from following their upgrade guide[1]. I just upgraded from 9.0 to 9.1 this morning without any issues. [1] https://pve.proxmox.com/wiki/Upgrade_from_8_to_9 https://pve.proxmox.com/wiki/Upgrade_from_8_to_9
- gregoryw3 11mo ago
  Do you happen to know if they fixed the memory ballooning issue?
  - polski-g 11mo ago
    What issue?
gerdesj 11mo ago
Do this: https://pve.proxmox.com/wiki/Upgrade_from_8_to_9 https://pve.proxmox.com/wiki/Upgrade_from_8_to_9 No need to overthink it too much. I've done several so far. Run the pve8to9 script first to do some sanity checks (it should already be installed if the system is up to date). Update the box to latest 8.x with apt update etc. Change the package sources to the new ones and update the system. The packages databases can be a bit confusing: You have two lots - stock Debian and Proxmox (enterprise OR no-subscription). Stock Debian is in the single file /etc/apt/sources.list - change "bookworm" to "trixie". Proxmox sources is in a file in /etc/apt/sources.list.d/ Remove all of the Proxmox related ones you have there and run this (or do it yourself with an editor). This example is no-sub - the official doc notes the enterprise equivalent: cat > /etc/apt/sources.list.d/proxmox.sources << EOF Types: deb URIs: http://download.proxmox.com/debian/pve Suites: trixie Components: pve-no-subscription Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg EOF Run apt dist-upgrade then the pve8to9 script again and then reboot. If in doubt choose Y for install the maintainer's version when prompted. There are notes in the doc about several packages. Job done.

js2 11mo ago

I've been running docker apps via podman inside alpine containers on Proxmox. Works for me with homebridge, scrypted, and plex. I created an alpine container per app with "nesting=1". Inside each alpine container I installed podman (`apk install podman`). Then it's a couple init scripts inside each alpine container, one to work-around a cgroups issue, the other to start the app via podman: https://pastebin.com/raw/W8YPKGjG https://pastebin.com/raw/W8YPKGjG

Havoc 11mo ago
Yeah same here except arch. Bit fiddly with bind mount permissions but otherwise works pretty well

OCTAGRAM 11mo ago

I have updated and tried to run something. There are apparently three-chunk ids and two-chunk ids. two-chunk ids work. Three-chunk ids don't. atdr.meo.ws/archiveteam/reddit-grab I cannot install this one to Proxmox VE, for instance UPD. Query tag fails, but fetch is successful if I write "latest" tag

btreecat 11mo ago

I run a kube cluster instead, because I want to build transferrable skills, using the standard open tools. I don't think I've ever seen anyone mention using proxmox in a professional context. None the less, nice progress I reckon.

tracker1 11mo ago

I never much minded dedicating a (or some) VM to it anyway, let me assign a dedicated IP. My email stack (mailu) runs best with a dedicated IP to the system, similar for my BBS setup (though broken atm)... For web apps, I run them all in a single Ubuntu Server VM with docker, ufw and caddy on the host OS. I run a similar setup with a few VMs on a mini pc at home as well... It all works well enough for what I need. Lets me somewhat isolate the containers VM from other purpose-specific VMs.