13 ms·

LXC vs. Docker

217 points by lycopodiopsida 5y ago

sickygnar 5y ago
I never hear systemd-nspawn mentioned in these discussions. It ships and integrates with systemd and has a decent interface with machinectl. Does anyone use it?
- numlock86 5y ago
  > I never hear systemd-nspawn mentioned in these discussions. It ships and integrates with systemd and has a decent interface with machinectl. I couldn't have said it better. And yes, I use it. Also in production systems.
- goombacloud 5y ago
  The big missing feature that's lacking is to pull Docker images and run them without resorting to hacks.
  - goombacloud 5y ago
    searched and found this: https://raw.githubusercontent.com/moby/moby/master/contrib/download-frozen-image-v2.sh https://raw.githubusercontent.com/moby/moby/master/contrib/d...
- leephillips 5y ago
  That’s what I use whenever I need a container. So simple and flexible.
p0d 5y ago
I've been running my saas on lxc for years. I love that the container is a folder to be copied. Combined with git to push changes to my app all is golden. I tried docker but stuck with lxc.
adamgordonbell 5y ago
I like the docker way of one thing, one process, per container. LXC seems a bit different. However, an exciting thing to me is the Cambrian explosion of alternatives to docker: podman, nerdctl, even lima for creating a linux vm and using containerd on macos looks interesting.
- merlinscholz 5y ago
  I recently started using containerd inside Nomad, a breath of fresh and simple air after failed k8s setups!
  - adamgordonbell 5y ago
    Oh, Nomad looks interesting. Why should someone reach for it vs K8S?
    - dijit 5y ago
      Nomad can run a lot more things but it’s not so batteries included. Nomad is trying to be an orchestrator. Kubernetes is trying to be an operating system for cloud environments. since they aim for being different things they make different trade offs.
    - orthecreedence 5y ago
      I have used nomad at my work for a few years now. I'd say where it shines is running stateless containers simply and easily. If you're trying to run redis, postgres, etc...do it somewhere else. If you're trying to spin up and down massive amounts of queue workers hour by hour, use it as a distributed cron, or hell just run containers for your public api/frontend and keep them running, nomad is great. That said, you're going to be doing some plumbing for things like wiring your services together (Fabio/Consul Connect are good choices), detecting when to add more host machines, etc. As far as how it compares to k8s, I don't know, I haven't used it materially yet.
    - merlinscholz 5y ago
      Cloudflare recently posted a great blog article on how/why they use nomad: https://blog.cloudflare.com/how-we-use-hashicorp-nomad/ https://blog.cloudflare.com/how-we-use-hashicorp-nomad/
dottedmag 5y ago
Apples to oranges. LXC can be directly compared with a small, and quite insignificant, part of Docker: container runtime. Docker became popular not because it can run containers, many tools before Docker could do that (LXC included). Docker became popular because it allows one to build, publish and then consume containers.
- leecarraher 5y ago
  i agree, docker added version control, online repository, with management, and search they also fixed some of the networking headaches in lxc. Prior to developing containerd docker was based on lxc containers.
- drewcoo 5y ago
  Isn't that where LXD is supposed to fit in? https://linuxcontainers.org/lxd/ https://linuxcontainers.org/lxd/
- sarusso 5y ago
  I agree on the apples to oranges, but LXC does not directly compare to a container runtime IMHO.. It is a proper engine to be fair, even if it provides much less functionalities as the Docker engine.
- chriswarbo 5y ago
  > Docker became popular because it allows one to build, publish and then consume containers. True, but Docker is an awful choice for those things (builds are performed "inside out" and aren't reproducible, publishing produces unauditable binary-blobs, consumption bypasses cryptographic security by fetching "latest" tags, etc.)
  - yjftsjthsd-h 5y ago
    It's still superior to everything that came before it (at least, that I'm aware of), and cleared the "good enough" bar. Actually, I'm still not aware of anything that solves those issues without making it way harder to use - ex. nix addresses your points but has an awful learning curve.
bamboozled 5y ago
One major limitation of LXC is that there is no way to easily self host images. Often the the official images for many distributions are buggy. For example, the official Ubuntu images seem to come with a raft of known issues. Based on my limited interactions with it, I'd recommend staying away from LXC unless absolutely neccesary.
- pentium166 5y ago
  I've only played around with LXC/LXD a little bit, what are some of the Ubuntu image issues? I did a quick google, but the first results seemed to be questions about hosting on Ubuntu rather than with the images themselves.
  - silverwind 5y ago
    In my experience, most issues are related to kernel interfaces which LXC disables inside unprivileged containers, paired with software that does not check if those interfaces are there/work before attempting to use them. These issues can be observed in the official Ubuntu image and seem to get worse over time. I would recommend to just use VMs instead.
- unixhero 5y ago
  In Proxmox "self hosting" meaning having a folder with LXC images is part of the distribution. You can download templates from online sources and use as images or create your own templates from already running LXCs. Or maybe you mean self hosting in another way?
- fuzzy2 5y ago
  If you feel that the existing images (of lxc-download?) have too many bugs for your liking, you could also try the classic templates, which use debootstrap and the like to create the rootfs.
- Lifelarper 5y ago
  > there is no way to easily self host images When you run lxd init there's an option to make the server available over the network (default: No), if enabled you can host images from there. lxc remote add myimageserver images.bamboozled.com lxc publish myimage lxc image copy myimage myimageserver lxc launch myimageserver:myimage
unixhero 5y ago
LXC has been so stable and great to work with for many years. I have had services in production on LXC containers and it has been a joy. I can not say the same about things I have tried to maintain in production with Docker, in which I had similar experiences to [0], albeit around that time and therefore arguably not recently. For a fantastic way to work with LXC containers I recommend the free and open Debian based hypervisor distribution Proxmox [1]. [0], https://thehftguy.com/2016/11/01/docker-in-production-an-history-of-failure/ https://thehftguy.com/2016/11/01/docker-in-production-an-his... [1], https://www.proxmox.com/en/proxmox-ve https://www.proxmox.com/en/proxmox-ve
melenaboija 5y ago
I use LXC containers as my development environments. When I changed my setup from expensive Mac Books to an expensive work station with a cheap laptop as front end to work remotely this was the best configuration I found. It took me few hours to have everything running but I love it now. New project is creating a new container add a rule to iptables and I have it ready in few seconds.
- dijit 5y ago
  FWIW I do the same thing but with docker. Exposing the docker daemon on the network and setting DOCKER_HOST I’m able to use the remote machine as if it was local. It’s hugely beneficial, I’ve considered making mini buildfarms that load balance this connection in a deterministic way.
  - fideloper 5y ago
    How do you share code files (set volume mounts) with remote docker via setup?
  - botdan 5y ago
    Do you have any more information about how you're doing this? Whenever I've tried to use Docker as a remote development environment the process felt very foreign and convoluted.
    - dijit 5y ago
      I know what you mean, It depends a little bit on your topology. If you have a secure network then it’s perfectly fine to expose the docker port on the network in plaintext without authentication. Otherwise you can use Port forwarding over SSH. To set up networked docker you can follow this: https://docs.docker.com/engine/security/protect-access/ https://docs.docker.com/engine/security/protect-access/ I’m on the phone so can’t give a detailed guide.
      - markkrj 5y ago
        Or you can use a docker as a ssh host using context, or just $DOCKER_HOST=ssh://myremote... See https://stackoverflow.com/questions/44056501/how-can-i-remotely-connect-to-docker-swarm/68303222#68303222 https://stackoverflow.com/questions/44056501/how-can-i-remot...
synergy20 5y ago
I think docker grew out of lxc initially(to make lxc easier to use), for now, lxc is light weight but it is not portable, docker can run on all OSes, I think that's the key difference: cross-platform apps. LXC remains to be a linux-only thing.
- deadbunny 5y ago
  Just as long as you ignore the linux VM all those docker containers are running in.
buybackoff 5y ago
LXC via Proxmox is great for stateful deployments on baremetal servers. It's very easy to backup entire containers with the state (SQLite, Postgres dir) to e.g. NAS (and with TrueNAS then to S3/B2). Best used with ZFS raid, with quotas and lazy space allocation backups are small or capped. Nothing stops one from running Docker inside LXC. For development I usually just make a dedicated priviledged LXC container with nesting enabled to avoid some known issues and painful config. LXC containers could be on a private network and a reverse proxy on the host could map to the only required ports, without thinking what ports Docker or oneself could have accidentally made public.
- lostlogin 5y ago
  Good comment. It was a revelation to me when I used Proxmox and played with LXCs. Getting an IP per container is really nice.
- ignoramous 5y ago
  > LXC via Proxmox is great for stateful deployments on baremetal Reminds me of (now defunct?) flockport.com They had some interesting demos up on YouTube, showcasing what looked like a sandstorm.io esque setup.
- petre 5y ago
  It's an annoying that you can only make snapshots on a stopped container. With VMs it works in a running VM.
  - nullwarp 5y ago
    Also can't do live migrations or backups and moving storage around is a headache. We've pretty much stopped using LXC containers in Proxmox because of all the little issues.
    - code_biologist 5y ago
      Dumb question, but what have you replaced it with? I’ve been working on setting up Proxmox/LXC on a box at home as we speak — but if it’s not worth the headache I’ll stick to VMs.
password4321 5y ago
Is it accurate to say LXC is to Docker as git is to GitHub, or vim/emacs vs. Visual Studio Code? I haven't seen many examples demonstrating the tooling used to manage LXC containers, but I haven't looked for it either. Docker is everywhere.
- haolez 5y ago
  In the first months of Docker, yes. Nowadays, they are different beasts.
- throwawayboise 5y ago
  lxc launch, lxc list, lxc start, lxc stop, etc.... That's all I've ever needed. Docker is overkill if you just need to run a few containers. There is a point where it makes sense but running a few containers for a small/personal project is not it.
  - mrweasel 5y ago
    Funnily enough I view it the other way around. LXC is a bit overkill, I just wanted to run a container, not an entire VM. In my mind you need to treat LXC containers as VMs, in terms of managements. They need to be patched, monitored and maintained the same ways as a VM. Docker containers still need to patched of cause, many seems to forget that bit, but generally they seem easier to deal with. Of cause that depends on what has been stuffed into the container image. LXC is underrated though, for small projects and business it can be a great alternative to VM platforms.
- sarusso 5y ago
  I recently wrote something to clarify my mind around all this [1]. If we assume that by Docker we mean the Docker engine, then I think you might compare it as you said (maybe more in terms as vim/emacs vs. Visual Studio Code as Git is a technology while GitHub is a platform). But Docker is many things: a company, a command line tool, a container runtime, a container engine, an image format, a registry... [1] https://sarusso.github.io/blog_container_engines_runtimes_orchestrators.html https://sarusso.github.io/blog_container_engines_runtimes_or...
junon 5y ago
LXC/LXD being the clear winner.
fuzzy2 5y ago
I’ve been using LXC as a lightweight “virtualization” platform for over 5 years now, with great success. It allows me to take existing installations of entire operating systems and put them in containers. Awesome stuff. On my home server, I have a VNC terminal server LXC container that is separate from the host system. Combined with ipvlan I can flexibly assign my dedicated server’s IP addresses to containers as required (MAC addresses were locked for a long time). Like, the real IP addresses. No 1:1 NAT. Super useful also for deploying Jitsi and the like. I still use Docker for things that come packaged as Docker images.
- heresie-dabord 5y ago
  > . It allows me to take existing installations of entire operating systems and put them in containers Friend, do you have documentation for this process? Please share your knowledge. ^_^
  - fuzzy2 5y ago
    Nothing too spectacular, I’m afraid. I had to consolidate some physical machines, all running Gentoo Linux. For each, I simply created a Gentoo LXC container and then replaced the rootfs (in /var/lib/lxc/NAME/rootfs) with the one from the physical server. The significant changes from the physical systems were: * rc_provide="net" in rc.conf because base networking is controlled externally * rc_sys="lxc" may or may not be necessary * Disable various net setup services On the host OS (Debian) I have interfaces like this: auto ipvl-main iface ipvl-main inet manual pre-up ip link add link eth0 name ipvl-main type ipvlan mode l2 post-down ip link delete ipvl-main In the container config, they are referenced this way: lxc.net.2.type = phys lxc.net.2.link = ipvl-main lxc.net.2.ipv4.address = 1.2.3.4/29 lxc.net.2.ipv4.gateway = 1.2.3.1 lxc.net.2.ipv6.address = abcd::2/128 lxc.net.2.ipv6.gateway = fe80::1 lxc.net.2.flags = up Later on, I removed the dedicated IP address and set up a reverse proxy instead. Oh yeah, all containers are of course privileged containers. With unprivileged containers, various things may not work as expected.
sarusso 5y ago
Interesting read, not sure why you compared only these two though. There are a plenty of other solutions and Docker is actually many things.. You can use Docker to run containers using Kata for example, which is a runtime providing full HW virtualisation. I wrote something similar, yet much less in detail on Docker and LXC and more as a bird-eye overview to clarify terminology, here: https://sarusso.github.io/blog_container_engines_runtimes_orchestrators.html https://sarusso.github.io/blog_container_engines_runtimes_or...
n3storm 5y ago
Love to hear I am not the only one enjoying LXC rather than Docker
istoica 5y ago
The perfect pair Containerfile vs Dockerfile - Infra as code podman vs docker - https://podman.io https://podman.io podman desktop companion (author here) vs docker desktop ui - https://iongion.github.io/podman-desktop-companion https://iongion.github.io/podman-desktop-companion podman-compose vs docker-compose = there should be no vs here, docker-compose itself can use podman socket for connection OOB as APIs are compatible, but an alternative worth exploring nevertheless. Things are improving at a very fast pace, the aim is to go way beyond parity, give it a chance, you might enjoy it. There is continuous active work that is enabling real choice and choice is always good, pushing everyone up.
- 2OEH8eoCRo0 5y ago
  I enjoy podman. It supported cgroups v2 before Docker and is daemonless.
- nijave 5y ago
  Is networking any better with Podman on Docker Compose files? Last time I tried, most docker-compose files didn't actually work because they created networks that Podman doesn't have privileges to setup unless run as root Afaik, the kernel network APIs are pretty complicated so it's fairly difficult to expose to unprivileged users safely
yokem55 5y ago
LXD (Canonical's daemon/API front end to lxc containers) is great -- as long as you aren't using the god awful snap package they insist on. The snap is probably fine for single dev machines, but it has zero place in anything production. This is because canonical insists on auto-updating and refreshing the snap at random intervals, even when you pin to a specific version channel. Three times I had to manually recover a cluster of lxd systems that broke during a snap refresh because the cluster couldn't cope with the snaps all refreshing at once. Going forward we built and installed lxd from source.
- rambojazz 5y ago
  I was bitten by LXD auto-updating as well. Server was down and I couldn't understand why since I hadn't changed anything.
- alyandon 5y ago
  I got so annoyed with snapd that I finally patched the auto-update functionality to provide control via environment variable. It's ridiculous that this is what I have to personally go through in order to maintain control of when updates are applied on my own systems. If enough people were to ever decide to get together and properly fork snapd and maintain the patched version I'd totally dedicate time to helping out. https://gist.github.com/alyandon/97813f577fe906497495439c37dacf37 https://gist.github.com/alyandon/97813f577fe906497495439c37d...
  - bamboozled 5y ago
    Agreed, it's the stupidest thing around, especially because Linux is about having control of your own systems, then they force updates down your throat, it goes against a lot of the things I use Linux for. I just avoid snap and Ubuntu wherever possible now.
  - agartner 5y ago
    We blocked the annoying snapd autoupdate behavior by setting a http proxy to a nonexistent server. Whenever we had a maintenance window we would unset the proxy, allow the update, then set the set the nonexistent proxy server again. Very annoying.
theteapot 5y ago
> Saying that LXC shares the kernel of its host does not convey the whole picture. In fact, LXC containers are using Linux kernel features to create isolated processes and file systems. So what is Docker doing then??
ruhrharry 5y ago
LXC is quite different from Docker. Docker is used most of the time as an containerized package format for servers and as such is comparable to snap or flatpak on the desktop. You don't have to know Linux administration to use Docker, that is why it is so successfull. LXC on the other hand is lightweight virtualization and one would have a hard time to use it without basic knowledge of administering Linux.
micw 5y ago
A while ago, I spent some time to make LXC run in a docker container. The idea is to have a statefull system managed by LXC run in a docker environment so that management (e.g. Volumes, Ingress and Load Balancer) from K8S can be used for the LXC containers. I still run a few desktops which are accessible by x2go with it on my kubernetes instances. https://github.com/micw/docker-lxc https://github.com/micw/docker-lxc
gerhardhaering 5y ago
This would have been an ok article in 2013-2015. Nothing really has changed wrt. these two technologies since.
lasftew 5y ago
My home server runs Nixos, which is an amazing server operating system: every service is configured in code and fully versioned. I also use this server for development (via SSH), but while Nixos can be used for development, it's relationship with VS Code, its plugins, and many native build tools (Golang, Rust) is very complicated, and I prefer not to do everything the Nix way, which is usually convoluted and poorly documented. LXD is my perfect fit in this scenario: trivial to install on top of Nixos, and once running, allows for launching some minimal development instances of whatever distro flavor of the day in a few seconds. Persistent like a small VM, but booting up within seconds, much more efficient on resources (memory in particular), and - unlike docker - with the full power of systemd and all. Add tailscale and sshd to the mix, for easy, secure and direct remote access to the virtualized system.
malkia 5y ago
I know very little about both, but I'm at the mercy everyday with lxc on my chromebook when running crostini (it's like a VM in a VM in a VM in a...) :) - works great though, at some perf cost, and less GPU support. And still having troubles running most of the docker images out there (either this, or that won't be supported). I guess it makes sense, after all there is always the choice of going with full real linux reinstall, or some other hacky ways. But one thing I was not aware was this: "Docker containers are made to run a single process per container."
kristianpaul 5y ago
At the end the two are different.. why comparing the in the first place? “ LXC, is a serious contender to virtual machines. So, if you are developing a Linux application or working with servers, and need a real Linux environment, LXC should be your go-to. Docker is a complete solution to distribute applications and is particularly loved by developers. Docker solved the local developer configuration tantrum and became a key component in the CI/CD pipeline because it provides isolation between the workload and reproducible environment.”
- anamexis 5y ago
  What are the fundamental differences?
  - kristianpaul 5y ago
    Linux control groups vs a runtime
Nasreddin_Hodja 5y ago
I had to switch to docker after LXC have been snaped.
ricmm 5y ago
LXC and Docker comparisons vastly differ depending on the use case and problem segment. I use LXC as a tiny, C-only library to abstract namespaces and cgroups for embedded usage [1] LXC is a fantastic userland library to easily consume kernel features for containerization without all the noise around it… but the push for the LXD scaffolding around it missed the mark. It should’ve just been a great library and that’s how we use it when running containers on embedded Linux equipment [1] https://pantacor.com/blog/lxc-vs-docker-what-do-you-need-for-iot/ https://pantacor.com/blog/lxc-vs-docker-what-do-you-need-for...

I never hear systemd-nspawn mentioned in these discussions. It ships and integrates with systemd and has a decent interface with machinectl. Does anyone use it?

numlock86 5y ago
> I never hear systemd-nspawn mentioned in these discussions. It ships and integrates with systemd and has a decent interface with machinectl. I couldn't have said it better. And yes, I use it. Also in production systems.
goombacloud 5y ago
The big missing feature that's lacking is to pull Docker images and run them without resorting to hacks.
- goombacloud 5y ago
  searched and found this: https://raw.githubusercontent.com/moby/moby/master/contrib/download-frozen-image-v2.sh https://raw.githubusercontent.com/moby/moby/master/contrib/d...
leephillips 5y ago
That’s what I use whenever I need a container. So simple and flexible.

p0d 5y ago

I've been running my saas on lxc for years. I love that the container is a folder to be copied. Combined with git to push changes to my app all is golden. I tried docker but stuck with lxc.

adamgordonbell 5y ago

I like the docker way of one thing, one process, per container. LXC seems a bit different. However, an exciting thing to me is the Cambrian explosion of alternatives to docker: podman, nerdctl, even lima for creating a linux vm and using containerd on macos looks interesting.

merlinscholz 5y ago
I recently started using containerd inside Nomad, a breath of fresh and simple air after failed k8s setups!
- adamgordonbell 5y ago
  Oh, Nomad looks interesting. Why should someone reach for it vs K8S?
  - dijit 5y ago
    Nomad can run a lot more things but it’s not so batteries included. Nomad is trying to be an orchestrator. Kubernetes is trying to be an operating system for cloud environments. since they aim for being different things they make different trade offs.
  - orthecreedence 5y ago
    I have used nomad at my work for a few years now. I'd say where it shines is running stateless containers simply and easily. If you're trying to run redis, postgres, etc...do it somewhere else. If you're trying to spin up and down massive amounts of queue workers hour by hour, use it as a distributed cron, or hell just run containers for your public api/frontend and keep them running, nomad is great. That said, you're going to be doing some plumbing for things like wiring your services together (Fabio/Consul Connect are good choices), detecting when to add more host machines, etc. As far as how it compares to k8s, I don't know, I haven't used it materially yet.
  - merlinscholz 5y ago
    Cloudflare recently posted a great blog article on how/why they use nomad: https://blog.cloudflare.com/how-we-use-hashicorp-nomad/ https://blog.cloudflare.com/how-we-use-hashicorp-nomad/

dottedmag 5y ago

Apples to oranges. LXC can be directly compared with a small, and quite insignificant, part of Docker: container runtime. Docker became popular not because it can run containers, many tools before Docker could do that (LXC included). Docker became popular because it allows one to build, publish and then consume containers.

leecarraher 5y ago
i agree, docker added version control, online repository, with management, and search they also fixed some of the networking headaches in lxc. Prior to developing containerd docker was based on lxc containers.
drewcoo 5y ago
Isn't that where LXD is supposed to fit in? https://linuxcontainers.org/lxd/ https://linuxcontainers.org/lxd/
sarusso 5y ago
I agree on the apples to oranges, but LXC does not directly compare to a container runtime IMHO.. It is a proper engine to be fair, even if it provides much less functionalities as the Docker engine.
chriswarbo 5y ago
> Docker became popular because it allows one to build, publish and then consume containers. True, but Docker is an awful choice for those things (builds are performed "inside out" and aren't reproducible, publishing produces unauditable binary-blobs, consumption bypasses cryptographic security by fetching "latest" tags, etc.)
- yjftsjthsd-h 5y ago
  It's still superior to everything that came before it (at least, that I'm aware of), and cleared the "good enough" bar. Actually, I'm still not aware of anything that solves those issues without making it way harder to use - ex. nix addresses your points but has an awful learning curve.

bamboozled 5y ago

One major limitation of LXC is that there is no way to easily self host images. Often the the official images for many distributions are buggy. For example, the official Ubuntu images seem to come with a raft of known issues. Based on my limited interactions with it, I'd recommend staying away from LXC unless absolutely neccesary.

pentium166 5y ago
I've only played around with LXC/LXD a little bit, what are some of the Ubuntu image issues? I did a quick google, but the first results seemed to be questions about hosting on Ubuntu rather than with the images themselves.
- silverwind 5y ago
  In my experience, most issues are related to kernel interfaces which LXC disables inside unprivileged containers, paired with software that does not check if those interfaces are there/work before attempting to use them. These issues can be observed in the official Ubuntu image and seem to get worse over time. I would recommend to just use VMs instead.
unixhero 5y ago
In Proxmox "self hosting" meaning having a folder with LXC images is part of the distribution. You can download templates from online sources and use as images or create your own templates from already running LXCs. Or maybe you mean self hosting in another way?
fuzzy2 5y ago
If you feel that the existing images (of lxc-download?) have too many bugs for your liking, you could also try the classic templates, which use debootstrap and the like to create the rootfs.
Lifelarper 5y ago
> there is no way to easily self host images When you run lxd init there's an option to make the server available over the network (default: No), if enabled you can host images from there. lxc remote add myimageserver images.bamboozled.com lxc publish myimage lxc image copy myimage myimageserver lxc launch myimageserver:myimage

unixhero 5y ago

LXC has been so stable and great to work with for many years. I have had services in production on LXC containers and it has been a joy. I can not say the same about things I have tried to maintain in production with Docker, in which I had similar experiences to [0], albeit around that time and therefore arguably not recently. For a fantastic way to work with LXC containers I recommend the free and open Debian based hypervisor distribution Proxmox [1]. [0], https://thehftguy.com/2016/11/01/docker-in-production-an-history-of-failure/ https://thehftguy.com/2016/11/01/docker-in-production-an-his... [1], https://www.proxmox.com/en/proxmox-ve https://www.proxmox.com/en/proxmox-ve

melenaboija 5y ago

I use LXC containers as my development environments. When I changed my setup from expensive Mac Books to an expensive work station with a cheap laptop as front end to work remotely this was the best configuration I found. It took me few hours to have everything running but I love it now. New project is creating a new container add a rule to iptables and I have it ready in few seconds.

dijit 5y ago
FWIW I do the same thing but with docker. Exposing the docker daemon on the network and setting DOCKER_HOST I’m able to use the remote machine as if it was local. It’s hugely beneficial, I’ve considered making mini buildfarms that load balance this connection in a deterministic way.
- fideloper 5y ago
  How do you share code files (set volume mounts) with remote docker via setup?
- botdan 5y ago
  Do you have any more information about how you're doing this? Whenever I've tried to use Docker as a remote development environment the process felt very foreign and convoluted.
  - dijit 5y ago
    I know what you mean, It depends a little bit on your topology. If you have a secure network then it’s perfectly fine to expose the docker port on the network in plaintext without authentication. Otherwise you can use Port forwarding over SSH. To set up networked docker you can follow this: https://docs.docker.com/engine/security/protect-access/ https://docs.docker.com/engine/security/protect-access/ I’m on the phone so can’t give a detailed guide.
    - markkrj 5y ago
      Or you can use a docker as a ssh host using context, or just $DOCKER_HOST=ssh://myremote... See https://stackoverflow.com/questions/44056501/how-can-i-remotely-connect-to-docker-swarm/68303222#68303222 https://stackoverflow.com/questions/44056501/how-can-i-remot...

synergy20 5y ago

I think docker grew out of lxc initially(to make lxc easier to use), for now, lxc is light weight but it is not portable, docker can run on all OSes, I think that's the key difference: cross-platform apps. LXC remains to be a linux-only thing.

deadbunny 5y ago
Just as long as you ignore the linux VM all those docker containers are running in.

buybackoff 5y ago

LXC via Proxmox is great for stateful deployments on baremetal servers. It's very easy to backup entire containers with the state (SQLite, Postgres dir) to e.g. NAS (and with TrueNAS then to S3/B2). Best used with ZFS raid, with quotas and lazy space allocation backups are small or capped. Nothing stops one from running Docker inside LXC. For development I usually just make a dedicated priviledged LXC container with nesting enabled to avoid some known issues and painful config. LXC containers could be on a private network and a reverse proxy on the host could map to the only required ports, without thinking what ports Docker or oneself could have accidentally made public.

lostlogin 5y ago
Good comment. It was a revelation to me when I used Proxmox and played with LXCs. Getting an IP per container is really nice.
ignoramous 5y ago
> LXC via Proxmox is great for stateful deployments on baremetal Reminds me of (now defunct?) flockport.com They had some interesting demos up on YouTube, showcasing what looked like a sandstorm.io esque setup.
petre 5y ago
It's an annoying that you can only make snapshots on a stopped container. With VMs it works in a running VM.
- nullwarp 5y ago
  Also can't do live migrations or backups and moving storage around is a headache. We've pretty much stopped using LXC containers in Proxmox because of all the little issues.
  - code_biologist 5y ago
    Dumb question, but what have you replaced it with? I’ve been working on setting up Proxmox/LXC on a box at home as we speak — but if it’s not worth the headache I’ll stick to VMs.

password4321 5y ago

Is it accurate to say LXC is to Docker as git is to GitHub, or vim/emacs vs. Visual Studio Code? I haven't seen many examples demonstrating the tooling used to manage LXC containers, but I haven't looked for it either. Docker is everywhere.

haolez 5y ago
In the first months of Docker, yes. Nowadays, they are different beasts.
throwawayboise 5y ago
lxc launch, lxc list, lxc start, lxc stop, etc.... That's all I've ever needed. Docker is overkill if you just need to run a few containers. There is a point where it makes sense but running a few containers for a small/personal project is not it.
- mrweasel 5y ago
  Funnily enough I view it the other way around. LXC is a bit overkill, I just wanted to run a container, not an entire VM. In my mind you need to treat LXC containers as VMs, in terms of managements. They need to be patched, monitored and maintained the same ways as a VM. Docker containers still need to patched of cause, many seems to forget that bit, but generally they seem easier to deal with. Of cause that depends on what has been stuffed into the container image. LXC is underrated though, for small projects and business it can be a great alternative to VM platforms.
sarusso 5y ago
I recently wrote something to clarify my mind around all this [1]. If we assume that by Docker we mean the Docker engine, then I think you might compare it as you said (maybe more in terms as vim/emacs vs. Visual Studio Code as Git is a technology while GitHub is a platform). But Docker is many things: a company, a command line tool, a container runtime, a container engine, an image format, a registry... [1] https://sarusso.github.io/blog_container_engines_runtimes_orchestrators.html https://sarusso.github.io/blog_container_engines_runtimes_or...

junon 5y ago

LXC/LXD being the clear winner.

fuzzy2 5y ago

I’ve been using LXC as a lightweight “virtualization” platform for over 5 years now, with great success. It allows me to take existing installations of entire operating systems and put them in containers. Awesome stuff. On my home server, I have a VNC terminal server LXC container that is separate from the host system. Combined with ipvlan I can flexibly assign my dedicated server’s IP addresses to containers as required (MAC addresses were locked for a long time). Like, the real IP addresses. No 1:1 NAT. Super useful also for deploying Jitsi and the like. I still use Docker for things that come packaged as Docker images.

heresie-dabord 5y ago
> . It allows me to take existing installations of entire operating systems and put them in containers Friend, do you have documentation for this process? Please share your knowledge. ^_^
- fuzzy2 5y ago
  Nothing too spectacular, I’m afraid. I had to consolidate some physical machines, all running Gentoo Linux. For each, I simply created a Gentoo LXC container and then replaced the rootfs (in /var/lib/lxc/NAME/rootfs) with the one from the physical server. The significant changes from the physical systems were: * rc_provide="net" in rc.conf because base networking is controlled externally * rc_sys="lxc" may or may not be necessary * Disable various net setup services On the host OS (Debian) I have interfaces like this: auto ipvl-main iface ipvl-main inet manual pre-up ip link add link eth0 name ipvl-main type ipvlan mode l2 post-down ip link delete ipvl-main In the container config, they are referenced this way: lxc.net.2.type = phys lxc.net.2.link = ipvl-main lxc.net.2.ipv4.address = 1.2.3.4/29 lxc.net.2.ipv4.gateway = 1.2.3.1 lxc.net.2.ipv6.address = abcd::2/128 lxc.net.2.ipv6.gateway = fe80::1 lxc.net.2.flags = up Later on, I removed the dedicated IP address and set up a reverse proxy instead. Oh yeah, all containers are of course privileged containers. With unprivileged containers, various things may not work as expected.

sarusso 5y ago

Interesting read, not sure why you compared only these two though. There are a plenty of other solutions and Docker is actually many things.. You can use Docker to run containers using Kata for example, which is a runtime providing full HW virtualisation. I wrote something similar, yet much less in detail on Docker and LXC and more as a bird-eye overview to clarify terminology, here: https://sarusso.github.io/blog_container_engines_runtimes_orchestrators.html https://sarusso.github.io/blog_container_engines_runtimes_or...

n3storm 5y ago

Love to hear I am not the only one enjoying LXC rather than Docker

istoica 5y ago

The perfect pair Containerfile vs Dockerfile - Infra as code podman vs docker - https://podman.io https://podman.io podman desktop companion (author here) vs docker desktop ui - https://iongion.github.io/podman-desktop-companion https://iongion.github.io/podman-desktop-companion podman-compose vs docker-compose = there should be no vs here, docker-compose itself can use podman socket for connection OOB as APIs are compatible, but an alternative worth exploring nevertheless. Things are improving at a very fast pace, the aim is to go way beyond parity, give it a chance, you might enjoy it. There is continuous active work that is enabling real choice and choice is always good, pushing everyone up.

2OEH8eoCRo0 5y ago
I enjoy podman. It supported cgroups v2 before Docker and is daemonless.
nijave 5y ago
Is networking any better with Podman on Docker Compose files? Last time I tried, most docker-compose files didn't actually work because they created networks that Podman doesn't have privileges to setup unless run as root Afaik, the kernel network APIs are pretty complicated so it's fairly difficult to expose to unprivileged users safely

yokem55 5y ago

LXD (Canonical's daemon/API front end to lxc containers) is great -- as long as you aren't using the god awful snap package they insist on. The snap is probably fine for single dev machines, but it has zero place in anything production. This is because canonical insists on auto-updating and refreshing the snap at random intervals, even when you pin to a specific version channel. Three times I had to manually recover a cluster of lxd systems that broke during a snap refresh because the cluster couldn't cope with the snaps all refreshing at once. Going forward we built and installed lxd from source.

rambojazz 5y ago
I was bitten by LXD auto-updating as well. Server was down and I couldn't understand why since I hadn't changed anything.
alyandon 5y ago
I got so annoyed with snapd that I finally patched the auto-update functionality to provide control via environment variable. It's ridiculous that this is what I have to personally go through in order to maintain control of when updates are applied on my own systems. If enough people were to ever decide to get together and properly fork snapd and maintain the patched version I'd totally dedicate time to helping out. https://gist.github.com/alyandon/97813f577fe906497495439c37dacf37 https://gist.github.com/alyandon/97813f577fe906497495439c37d...
- bamboozled 5y ago
  Agreed, it's the stupidest thing around, especially because Linux is about having control of your own systems, then they force updates down your throat, it goes against a lot of the things I use Linux for. I just avoid snap and Ubuntu wherever possible now.
- agartner 5y ago
  We blocked the annoying snapd autoupdate behavior by setting a http proxy to a nonexistent server. Whenever we had a maintenance window we would unset the proxy, allow the update, then set the set the nonexistent proxy server again. Very annoying.

theteapot 5y ago

> Saying that LXC shares the kernel of its host does not convey the whole picture. In fact, LXC containers are using Linux kernel features to create isolated processes and file systems. So what is Docker doing then??

ruhrharry 5y ago

LXC is quite different from Docker. Docker is used most of the time as an containerized package format for servers and as such is comparable to snap or flatpak on the desktop. You don't have to know Linux administration to use Docker, that is why it is so successfull. LXC on the other hand is lightweight virtualization and one would have a hard time to use it without basic knowledge of administering Linux.

micw 5y ago

A while ago, I spent some time to make LXC run in a docker container. The idea is to have a statefull system managed by LXC run in a docker environment so that management (e.g. Volumes, Ingress and Load Balancer) from K8S can be used for the LXC containers. I still run a few desktops which are accessible by x2go with it on my kubernetes instances. https://github.com/micw/docker-lxc https://github.com/micw/docker-lxc

gerhardhaering 5y ago

This would have been an ok article in 2013-2015. Nothing really has changed wrt. these two technologies since.

lasftew 5y ago

My home server runs Nixos, which is an amazing server operating system: every service is configured in code and fully versioned. I also use this server for development (via SSH), but while Nixos can be used for development, it's relationship with VS Code, its plugins, and many native build tools (Golang, Rust) is very complicated, and I prefer not to do everything the Nix way, which is usually convoluted and poorly documented. LXD is my perfect fit in this scenario: trivial to install on top of Nixos, and once running, allows for launching some minimal development instances of whatever distro flavor of the day in a few seconds. Persistent like a small VM, but booting up within seconds, much more efficient on resources (memory in particular), and - unlike docker - with the full power of systemd and all. Add tailscale and sshd to the mix, for easy, secure and direct remote access to the virtualized system.

malkia 5y ago

I know very little about both, but I'm at the mercy everyday with lxc on my chromebook when running crostini (it's like a VM in a VM in a VM in a...) :) - works great though, at some perf cost, and less GPU support. And still having troubles running most of the docker images out there (either this, or that won't be supported). I guess it makes sense, after all there is always the choice of going with full real linux reinstall, or some other hacky ways. But one thing I was not aware was this: "Docker containers are made to run a single process per container."

kristianpaul 5y ago

At the end the two are different.. why comparing the in the first place? “ LXC, is a serious contender to virtual machines. So, if you are developing a Linux application or working with servers, and need a real Linux environment, LXC should be your go-to. Docker is a complete solution to distribute applications and is particularly loved by developers. Docker solved the local developer configuration tantrum and became a key component in the CI/CD pipeline because it provides isolation between the workload and reproducible environment.”

anamexis 5y ago
What are the fundamental differences?
- kristianpaul 5y ago
  Linux control groups vs a runtime

Nasreddin_Hodja 5y ago

I had to switch to docker after LXC have been snaped.

ricmm 5y ago

LXC and Docker comparisons vastly differ depending on the use case and problem segment. I use LXC as a tiny, C-only library to abstract namespaces and cgroups for embedded usage [1] LXC is a fantastic userland library to easily consume kernel features for containerization without all the noise around it… but the push for the LXD scaffolding around it missed the mark. It should’ve just been a great library and that’s how we use it when running containers on embedded Linux equipment [1] https://pantacor.com/blog/lxc-vs-docker-what-do-you-need-for-iot/ https://pantacor.com/blog/lxc-vs-docker-what-do-you-need-for...