4 ms·

> That is, a LXD container runs a full Linux system, exactly as it would be when run on metal or in a VM. This is what makes LXD so nice. It's the power and fl

> That is, a LXD container runs a full Linux system, exactly as it would be when run on metal or in a VM. This is what makes LXD so nice. It's the power and flexibility of a VM with the speed, api, and runability of containers. I always hated that docker forced you to treat a container as a single process and spoke about it as if it's a virtue. Let me decide how many or little processes I want running in a container. Even google runs ssh server alongside their applications. With LXD you can easily port entire services over to containers without completely rewriting the way they behave.

mayank 11y ago
> I always hated that docker forced you to treat a container as a single process and spoke about it as if it's a virtue. Let me decide how many or little processes I want running in a container. You can have as many processes as you like running inside a Docker container, you just a need a single process to "boot" them, even if it's a simple shell script.
- tyingq 11y ago
  The big difference is that LXD can run, in an unprivileged fashion, a systemd based distro without issue, and with a sensible view of /proc. Docker can't currently do that, largely because running a distro in a container isn't an important goal for them.
  - lisivka 11y ago
    Docker can run systemd in unprivileged container: https://github.com/vlisivka/docker-centos7-systemd-unpriv https://github.com/vlisivka/docker-centos7-systemd-unpriv PS. Try it: docker run -dt --stop-signal=$(kill -l RTMIN+3) -v /sys/fs/cgroup:/sys/fs/cgroup:ro --name centos7_systemd vlisivka/docker-centos7-systemd-unpriv
    - tyingq 11y ago
      My impression is that someone created a specific dockerfile to get around the issues. I'm noting that LXD can run, for example, stock debian.
      - JdeBP 11y ago
        This appears to be a fairly complex subject. See https://news.ycombinator.com/item?id=11216950 https://news.ycombinator.com/item?id=11216950 and the original LWN article that it depends from.
  - cyphar 11y ago
    There are quite a few issues with running a systemd-based distro in a container. The first being that /sys and /proc are basically not namespaced. So if you're not careful, systemd might start modifying your host. There's some work to improve this (the cgroup namespace -- yes, that's a thing now) but the point remains that allowing a root process to start touching /proc and /sys inside a container won't end well. I believe that LXD tries to get around this problem by mounting a FUSE filesystem on /proc and /sys that does a bunch of fakery. I don't like that, to be honest. It should be the kernel's job to namespace things properly.
- xorcist 11y ago
  That's not a good idea, for the same reason you don't want a shell script for /sbin/init.
atomic77 11y ago
Indeed, LXD would seem to be a faster, more efficient way of doing what you would do with traditional VMs. I don't see a lot of overlap with docker, despite the superficial similarity.
- willsher 11y ago
  There are still a few gotchas on the low level stuff, especially networking, but also other areas that involve kernel drivers more directly. The security around the containers still isn't as good as a hardware VM, but I think hardware segregation is on the road map. It's ideal for running container inside VMs, though, and offers nice management segregation on those larger VMs.
  - voidz 11y ago
    Can you name some of these networking gotchas you mentioned? And elaborate on the security remark?
    - deleted 11y ago
      [deleted]
    - egil 11y ago
      The major security issue is that containers share the same kernel as the host, with the whole kernel syscall interface as attack vector. VMs run separate kernels that only interface with the host via the hypervisor.
      - markgavalda 11y ago
        And we haven't seen many hypervisor security holes in the past... :) Especially with Xen.
      - cyphar 11y ago
        Well, there's also the issue that "Linux containers" are not a kernel primitive. So there's plenty of syscalls that aren't even namespaced (the keyring ones come to mind). Docker handles this by having a default seccomp profile that disables a bunch of syscalls that aren't namespaced. However, the development of Zones and Jails (on SunOS/Solaris/illumos and FreeBSD) was much more security focused, with the default being "that's not allowed in a (Jail|Zone) until we can make sure it's safe". I really wish Linux had just ported Jails or inspired their security model on them. I do a lot of work with the internals of Docker and runC at SUSE. Trust me, it's not pretty how you have to set up "Linux containers" and there's 1001 gotchas.

mayank 11y ago

> I always hated that docker forced you to treat a container as a single process and spoke about it as if it's a virtue. Let me decide how many or little processes I want running in a container. You can have as many processes as you like running inside a Docker container, you just a need a single process to "boot" them, even if it's a simple shell script.

tyingq 11y ago
The big difference is that LXD can run, in an unprivileged fashion, a systemd based distro without issue, and with a sensible view of /proc. Docker can't currently do that, largely because running a distro in a container isn't an important goal for them.
- lisivka 11y ago
  Docker can run systemd in unprivileged container: https://github.com/vlisivka/docker-centos7-systemd-unpriv https://github.com/vlisivka/docker-centos7-systemd-unpriv PS. Try it: docker run -dt --stop-signal=$(kill -l RTMIN+3) -v /sys/fs/cgroup:/sys/fs/cgroup:ro --name centos7_systemd vlisivka/docker-centos7-systemd-unpriv
  - tyingq 11y ago
    My impression is that someone created a specific dockerfile to get around the issues. I'm noting that LXD can run, for example, stock debian.
    - JdeBP 11y ago
      This appears to be a fairly complex subject. See https://news.ycombinator.com/item?id=11216950 https://news.ycombinator.com/item?id=11216950 and the original LWN article that it depends from.
- cyphar 11y ago
  There are quite a few issues with running a systemd-based distro in a container. The first being that /sys and /proc are basically not namespaced. So if you're not careful, systemd might start modifying your host. There's some work to improve this (the cgroup namespace -- yes, that's a thing now) but the point remains that allowing a root process to start touching /proc and /sys inside a container won't end well. I believe that LXD tries to get around this problem by mounting a FUSE filesystem on /proc and /sys that does a bunch of fakery. I don't like that, to be honest. It should be the kernel's job to namespace things properly.
xorcist 11y ago
That's not a good idea, for the same reason you don't want a shell script for /sbin/init.

atomic77 11y ago

Indeed, LXD would seem to be a faster, more efficient way of doing what you would do with traditional VMs. I don't see a lot of overlap with docker, despite the superficial similarity.

willsher 11y ago
There are still a few gotchas on the low level stuff, especially networking, but also other areas that involve kernel drivers more directly. The security around the containers still isn't as good as a hardware VM, but I think hardware segregation is on the road map. It's ideal for running container inside VMs, though, and offers nice management segregation on those larger VMs.
- voidz 11y ago
  Can you name some of these networking gotchas you mentioned? And elaborate on the security remark?
  - deleted 11y ago
    [deleted]
  - egil 11y ago
    The major security issue is that containers share the same kernel as the host, with the whole kernel syscall interface as attack vector. VMs run separate kernels that only interface with the host via the hypervisor.
    - markgavalda 11y ago
      And we haven't seen many hypervisor security holes in the past... :) Especially with Xen.
    - cyphar 11y ago
      Well, there's also the issue that "Linux containers" are not a kernel primitive. So there's plenty of syscalls that aren't even namespaced (the keyring ones come to mind). Docker handles this by having a default seccomp profile that disables a bunch of syscalls that aren't namespaced. However, the development of Zones and Jails (on SunOS/Solaris/illumos and FreeBSD) was much more security focused, with the default being "that's not allowed in a (Jail|Zone) until we can make sure it's safe". I really wish Linux had just ported Jails or inspired their security model on them. I do a lot of work with the internals of Docker and runC at SUSE. Trust me, it's not pretty how you have to set up "Linux containers" and there's 1001 gotchas.