4 ms·

> Running docker inside LXC is weird. Knowing when to use a vm and when to use a container is sometimes an opaque problem. This is one of those cases where a

by zer00eyz 11mo ago

> Running docker inside LXC is weird. Knowing when to use a vm and when to use a container is sometimes an opaque problem. This is one of those cases where a VM is a much better choice.

poisonborz 11mo ago
This seems like a niche issue, running Docker in LXC for years with dozens of images without a problem.
- SirMaster 11mo ago
  Guessing you are only running a single node though, not a cluster with HA and live migration and all that.
selectodude 11mo ago
Am I crazy or is converting a dockerfile into LXC something that should be possible?
- mzsl 11mo ago
  In the new Proxmox VE 9.1 release this should be possible, from the changelog: > OCI images can now be uploaded manually or downloaded from image registries, and then be used as templates for LXC containers.
- deleted 11mo ago
  [deleted]
- tharos47 11mo ago
  It should in an ideal world but docker is a very leaky abstraction imho and you will run into a number of problems. It has improved as of newer kernel and docker versions but they were problems (overlayfs/zfs incompatibilities/ uid mapping problems in docker images/ capabilities requested by docker not available in LXC, rootless docker problems,...)
0x1ch 11mo ago
The way I understand it is that Docker with LXC allows for compute / resource sharing, where as dedicated VMs will will require passing through the entire discrete GPU. So, the VMs require a total passthrough of those Zigbees, container wouldn't? I'm not exactly sure how the outcome would have changed here though.
evanjrowley 11mo ago
I can't speak for the author, but they said they have a Coral TPU passed into the LXC & container, which I also have on my Proxmox setup for Frigate NVR. Depending on your hardware platform, there could be valid reasons why you wouldn't want to run Frigate NVR in a VM. Frigate NVR it works best when it can leverage the GPU for video transcoding and TPU for object detection. If you pass the GPU to the VM, then the Proxmox host no longer has video output (without a secondary GPU). Unless you have an Intel Arc iGPU, Intel Arc B50/B60, or fancy server GPU, you won't have SR-IOV on your system, and that means you have to pass the entire GPU into the VM. This is a non-starter for systems where there is no extra PCIe slot for a graphics card, such as the many power-efficient Intel N100 systems that do a good job running Frigate. The reason why you'd put Docker into LXC is that's the best supported way to get docker engine working on Proxmox without a VM. You'd want to do it on Proxmox because it brings other benefits like a familiar interface, clustering, Proxmox Backup Server, and a great community. You'd want to run Frigate NVR within Docker because it is the best supported way to run it. At least, this was the case in Proxmox 8. I haven't checked what advancements in Proxmox 9 may have changed this.
- roger_ 11mo ago
  I have Frigate and a Coral USB running happily in a VM on an N97. GPU pass through is slightly annoying (need to use a custom ROM from here: https://github.com/LongQT-sea/intel-igpu-passthru https://github.com/LongQT-sea/intel-igpu-passthru). I think SRIOV works but haven’t tried. And Coral only works in USB3 mode if you pass the whole PCIe controller.
  - nirav72 11mo ago
    I've been debating if I should move my frigate off an aging Unraid server to spare mini PC with Proxmox. The mini has a N97 with 16gb ram. How cameras do you have in your frigate instance on that N97? Just wondering if a N97 is capable of handling 4+ cameras. I do have a Coral TPU for inference & detection.
    - roger_ 11mo ago
itopaloglu83 11mo ago
Proxmox FAQ calls running Docker on LXC a tech preview and “kind of” recommends VMs. At the very bottom of the page. https://pve.proxmox.com/wiki/FAQ https://pve.proxmox.com/wiki/FAQ > While it can be convenient to run “Application Containers” directly as Proxmox Containers, doing so is currently a tech preview. For use cases requiring container orchestration or live migration, it is still recommended to run them inside a Proxmox QEMU virtual machine.
szszrk 11mo ago
It's not always better. Docker on lxc has a lot of advantages. I would rather use plain lxc on production systems, but I've been homelabbing on lxc+docker for years. It's blazing fast and I cut down around 60% of my RAM consumption. It's easy to manage, boots instantly, allows for more elastic separation while still using docker and/or k8s. I love that it allows me to keep using Proxmox Backup Server. I'm postponing homelab upgrade for a few years thanks to that.

> Running docker inside LXC is weird. Knowing when to use a vm and when to use a container is sometimes an opaque problem. This is one of those cases where a VM is a much better choice.

poisonborz 11mo ago

This seems like a niche issue, running Docker in LXC for years with dozens of images without a problem.

SirMaster 11mo ago
Guessing you are only running a single node though, not a cluster with HA and live migration and all that.

selectodude 11mo ago

Am I crazy or is converting a dockerfile into LXC something that should be possible?

mzsl 11mo ago
In the new Proxmox VE 9.1 release this should be possible, from the changelog: > OCI images can now be uploaded manually or downloaded from image registries, and then be used as templates for LXC containers.
deleted 11mo ago
[deleted]
tharos47 11mo ago
It should in an ideal world but docker is a very leaky abstraction imho and you will run into a number of problems. It has improved as of newer kernel and docker versions but they were problems (overlayfs/zfs incompatibilities/ uid mapping problems in docker images/ capabilities requested by docker not available in LXC, rootless docker problems,...)

0x1ch 11mo ago

The way I understand it is that Docker with LXC allows for compute / resource sharing, where as dedicated VMs will will require passing through the entire discrete GPU. So, the VMs require a total passthrough of those Zigbees, container wouldn't? I'm not exactly sure how the outcome would have changed here though.

evanjrowley 11mo ago

I can't speak for the author, but they said they have a Coral TPU passed into the LXC & container, which I also have on my Proxmox setup for Frigate NVR. Depending on your hardware platform, there could be valid reasons why you wouldn't want to run Frigate NVR in a VM. Frigate NVR it works best when it can leverage the GPU for video transcoding and TPU for object detection. If you pass the GPU to the VM, then the Proxmox host no longer has video output (without a secondary GPU). Unless you have an Intel Arc iGPU, Intel Arc B50/B60, or fancy server GPU, you won't have SR-IOV on your system, and that means you have to pass the entire GPU into the VM. This is a non-starter for systems where there is no extra PCIe slot for a graphics card, such as the many power-efficient Intel N100 systems that do a good job running Frigate. The reason why you'd put Docker into LXC is that's the best supported way to get docker engine working on Proxmox without a VM. You'd want to do it on Proxmox because it brings other benefits like a familiar interface, clustering, Proxmox Backup Server, and a great community. You'd want to run Frigate NVR within Docker because it is the best supported way to run it. At least, this was the case in Proxmox 8. I haven't checked what advancements in Proxmox 9 may have changed this.

roger_ 11mo ago
I have Frigate and a Coral USB running happily in a VM on an N97. GPU pass through is slightly annoying (need to use a custom ROM from here: https://github.com/LongQT-sea/intel-igpu-passthru https://github.com/LongQT-sea/intel-igpu-passthru). I think SRIOV works but haven’t tried. And Coral only works in USB3 mode if you pass the whole PCIe controller.
- nirav72 11mo ago
  I've been debating if I should move my frigate off an aging Unraid server to spare mini PC with Proxmox. The mini has a N97 with 16gb ram. How cameras do you have in your frigate instance on that N97? Just wondering if a N97 is capable of handling 4+ cameras. I do have a Coral TPU for inference & detection.
  - roger_ 11mo ago

itopaloglu83 11mo ago

Proxmox FAQ calls running Docker on LXC a tech preview and “kind of” recommends VMs. At the very bottom of the page. https://pve.proxmox.com/wiki/FAQ https://pve.proxmox.com/wiki/FAQ > While it can be convenient to run “Application Containers” directly as Proxmox Containers, doing so is currently a tech preview. For use cases requiring container orchestration or live migration, it is still recommended to run them inside a Proxmox QEMU virtual machine.

szszrk 11mo ago

It's not always better. Docker on lxc has a lot of advantages. I would rather use plain lxc on production systems, but I've been homelabbing on lxc+docker for years. It's blazing fast and I cut down around 60% of my RAM consumption. It's easy to manage, boots instantly, allows for more elastic separation while still using docker and/or k8s. I love that it allows me to keep using Proxmox Backup Server. I'm postponing homelab upgrade for a few years thanks to that.