9 ms·

LXD 2.0: Introduction to LXD

226 points by bpierre 11y ago

nevi-me 11y ago
Congratulations on 2.0! I tried out LXD a few months ago after hitting a few walls with Vagrant, and I've been happy since. Mark Shuttleworth's normally responsive on Google+, and he helped point me in the right direction with initial issues that I encountered. Stéphane and the team have also been helpful on Github with issues. The first thing I struggled with in this containers business was "what's the difference between LXC/LXD and Docker et. al.?", and I am glad that he addressed it first in the post. I've got north of 20 containers on our machines, and LXD allows me to create a container, play around with software config on it, and when I get it right; I just leave that container and update it regularly like the host it's on.
- fulafel 11y ago
  You can also use LXC as Vagrant backend to get away from the crashy and buggy VirtualBox kernel driver. Though ~all the Vagrant images you find online are tied to VirtualBox.
  - mugsie 11y ago
    Yeah, it can be a pain to get images. That said, my workflow for testing anything these days is inside a vagrant lxc box. it is well worth the lack of images for the improved workflow.
IgorPartola 11y ago
Does LXD provide any type of cluster management behavior? As in if I have two host machines and a dozen containers and host A goes down, is it possible to configure a live migration of the latest known state from host A to host B using LXD alone? Or do I need something built on top of it to do that?
- nnutter 11y ago
  I think that's Juju's role. They have a beta LXD driver for Juju. https://jujucharms.com/docs/master/config-LXD https://jujucharms.com/docs/master/config-LXD
- wmf 11y ago
  There's an OpenStack driver for LXD.
- tyingq 11y ago
  LXD and any sort of cluster seem incompatible at the moment. LXD, for example, stores all the LXC config info in a local sqlite database. When you launch a container, it turns the sqlite stored info into an LXC config file, and spawns LXC. You can control remote instances of the LXD daemon, but there's no syncing back and forth from Host1-LXD1 to Host1-LXD2...they operate separately. You can migrate a container from one host to the other, but it's pushing everything (config/image/etc) when you ask it to migrate. So, it's not clustered itself, and it keeps the important info somewhere that's hard to cluster. It would be interesting if they would replace the local sqlite store with something similar to etcd. That still leaves the issue of the disk images though. Proxmox does cluster LXC, so there's that option.
bronson 11y ago
Is anybody making an effort to put the LXD environment on Centos 7? I'd really like to try it but all I have provisioned right now is Centos 7 (and below).
matt_wulfeck 11y ago
> That is, a LXD container runs a full Linux system, exactly as it would be when run on metal or in a VM. This is what makes LXD so nice. It's the power and flexibility of a VM with the speed, api, and runability of containers. I always hated that docker forced you to treat a container as a single process and spoke about it as if it's a virtue. Let me decide how many or little processes I want running in a container. Even google runs ssh server alongside their applications. With LXD you can easily port entire services over to containers without completely rewriting the way they behave.
- mayank 11y ago
  > I always hated that docker forced you to treat a container as a single process and spoke about it as if it's a virtue. Let me decide how many or little processes I want running in a container. You can have as many processes as you like running inside a Docker container, you just a need a single process to "boot" them, even if it's a simple shell script.
  - tyingq 11y ago
    The big difference is that LXD can run, in an unprivileged fashion, a systemd based distro without issue, and with a sensible view of /proc. Docker can't currently do that, largely because running a distro in a container isn't an important goal for them.
    - lisivka 11y ago
      Docker can run systemd in unprivileged container: https://github.com/vlisivka/docker-centos7-systemd-unpriv https://github.com/vlisivka/docker-centos7-systemd-unpriv PS. Try it: docker run -dt --stop-signal=$(kill -l RTMIN+3) -v /sys/fs/cgroup:/sys/fs/cgroup:ro --name centos7_systemd vlisivka/docker-centos7-systemd-unpriv
      - tyingq 11y ago
        My impression is that someone created a specific dockerfile to get around the issues. I'm noting that LXD can run, for example, stock debian.
nzoschke 11y ago
This looks really nice. I build out major productions systems on top of LXC at Heroku, and to this day it is still working incredibly well. So I am really looking forward to further iterations of the tools by Stéphane Graber and co at Canonical. The main thing that jumps out at me is a REST API for this layer. A REST API with Golang bindings to run containers feels so much better than old-school systems programming with exec. I'm currently all in on Docker in production for this very reason. Coordinating stuff with a golang client over HTTP is really really productive.
jaytaylor 11y ago
With so many contenders stirring up the pot, this is an exciting time in the world of Linux Containers! I would love to see an LXD container driver for Mesos. Do any of you know if LXD is APPC-spec compliant?
- stgraber 11y ago
  It's not as the appc spec stands for "app container" and LXD doesn't run application containers but instead full system containers.
alinspired 11y ago
LXD looks similar to OpenVZ or rather Proxmox(with LXC or OpenVZ)
- yo-code-sucks 11y ago
  Because LXD is jusr LXC with an API.
foreigner 11y ago
What’s LXD? At its simplest, LXD is a daemon which provides a REST API to drive LXC >containers. Thanks, that clears it right up.
- daurnimator 11y ago
  Not sure if you're being sarcastic or not?
  - nzoschke 11y ago
    No sarcasm. This is a simple explanation of where LXD fits into the stack. If you don't know what LXC is maybe this isn't clear. But LXC stands for LinuX Containers, and is extremely fundamental technology in the containerization space.
    - daurnimator 11y ago
      (You should possibly have replied to the post above me) > LXC stands for LinuX Containers, and is extremely fundamental technology in the containerization space. Not exactly. LXC was amoung the first pieces of software to take advantage of namespaces when they were added to the kernel; but LXC is not often used. Most of the various containerisation software now available does not use LXC, and it not based on the same codebase.
voltagex_ 11y ago
I'm moving my home NAS back to Debian after a couple of painful months [1] [2] with FreeNAS 9. Would this give me jail-like functionality under Linux? The one thing I enjoyed about FreeNAS/FreeBSD was how easy it was to create and tear down jails for all kinds of uses. [1]: https://blog.voltagex.org/2016/01/08/freenas-9-3-on-the-supermicro-5028a-tn4-a1sri-2758f/ https://blog.voltagex.org/2016/01/08/freenas-9-3-on-the-supe... [2]: https://lists.freebsd.org/pipermail/freebsd-net/2016-February/044619.html https://lists.freebsd.org/pipermail/freebsd-net/2016-Februar...
- nine_k 11y ago
  Plain LXC, rut, or docker are probably closer.
  - voltagex_ 11y ago
    I like docker, but I prefer something closer to a VM. Why would you say go for LXC over LXD? Also, Googling "linux rut" doesn't bring up anything useful, I'm assuming you mean rkt
    - josteink 11y ago
      LXC is very nice. It's very VM-like and why I prefer it for almost everything I deploy instead of using Docker. I know how to manage machines. I don't have the same confidence for Docker-images. LXD seems to be a construction on top of LXC to make managing it easier "at scale", so if all you want is to manually be able to construct "container VMs" on demand, I think LXC is closer to what you actually want: It let's you manage the containers directly, near the iron, without anything getting in your way. Don't get me wrong: LXD seems to have a genuine value-proposal, but for this use-case it seems to be a slight step of indirection, and my guess is that this will cause you more troubles than it's worth.
mrbill 11y ago
Wonder if I'll be able to use LXC/LXD on "plain" Debian, and be able to avoid Ubuntu.
- voidz 11y ago
  Ubuntu, yes, I'd think so. Canonical, however, not. What makes you want to avoid Ubuntu?
  - mrbill 11y ago
    I use "plain" Debian on my servers when possible. Personal preference, mostly. Not really a fan of Ubuntu Server.
vasi 11y ago
I like almost everything about LXD, except the attitude towards networking. The project has rejected simple solutions such as port forwarding, saying that managing the network shouldn't be LXD's job. Instead, they'd like the user to manually configure their own bridges or routes or iptables chains. I can kinda understand their point of view, there's no simple solution that will please everyone. But most developers or IT folks aren't networking experts, and LXD won't be an intuitive tool for them without a simpler mode of operation.
- ansible 11y ago
  I have just finished a small LXC deployment, and the networking did take a little bit of time to figure out what direction to go. I ended up choosing to use bridging to connect all the physical network adapters to the ones in the containers. This is nice because I set up each container with its own IP address, which travels with it when the container is moved to a new host.
- dozzie 11y ago
  > [...] most developers or IT folks aren't networking experts, and LXD won't be an intuitive tool for them without a simpler mode of operation. Really, no expertise is required. Just basic understanding how the heck the network works. If somebody can't grasp what bridge interface is or how NAT operates, he apparently doesn't have the qualifications for writing software.
  - lotyrin 11y ago
    Then I guess most of the developers I've worked with should brush up their sandwich-making skills and quit their jobs.
    - dozzie 11y ago
      You do realize that this is not to their credit, right?
jaequery 11y ago
if someone can create a boot2docker clone for this, i'm in.
- lolidaisuki 11y ago
  Why not just use regular TinyCore? Or any other distribution? LXC containers aren't app containers. It is intended to be used with regular installs, not with single applications.
sandra_saltlake 11y ago
This looks really nice. A REST API with Golang bindings to run containers feels so much better than old-school systems programming with exec.
jccc 11y ago
It simultaneously irks me and pleases me that the first result on DuckDuckGo for "lxd" is the Legion of Extraordinary Dancers. (Google gets it right, presumably because of all my code searches.) "[...] a 2010–2011 web series about two groups of rival dancers: The Alliance of the Dark who are the villains and The Legion of Extraordinary Dancers, the heroes, who discover they have superpowers referred to as 'the Ra' through their dance abilities."
srcmap 11y ago
Very Nice. I want to understand how Linux Container works, started to study the cgroup code in Linux kernel and LXD. Here are couple of notes on Tracing the code of CGroup in Linux kernel and LXD I took. http://www.srcmap.org/sd_share/7/0cc46512/LXD_Code_Trace.html http://www.srcmap.org/sd_share/7/0cc46512/LXD_Code_Trace.htm... http://www.srcmap.org/sd_share/7/060b8d61/Linux_Kernel_CGroup_Control_Group_Code_Trace.html http://www.srcmap.org/sd_share/7/060b8d61/Linux_Kernel_CGrou...
Annatar 11y ago
If I need to run Linux software, I can use an lx-branded zone inside of SmartOS, and run at the speed of bare metal while having ZFS, mdb, kdb, dbx and DTrace, CPU-bursting, and change kernel schedulers on the fly. But then, as I am about to run Linux software inside of an lx-branded zone on SmartOS, I realize that SmartOS has 15,000 packages, with most the of the software I would have run on Linux, available natively on SmartOS: http://www.perkin.org.uk/posts/building-packages-at-scale.html http://www.perkin.org.uk/posts/building-packages-at-scale.ht... ...so then, which operational advantages would LX containers on Linux offer me, that I already do not have on SmartOS with zones?
- lolidaisuki 11y ago
  You can think of it as competition. Other people can think of it as something that they can make people who are already familiar with systems that use the Linux kernel.
  - Annatar 11y ago
    SmartOS is paranoid about correctness of operation and data integrity; Linux and LXC containers are a poor competition in that area, when the OS comes with memory overcommit turned on by default, and default filesystems do not include data write guarantees nor block checksums...
    - lolidaisuki 11y ago
      Are you trying to convince me of something? E: Not all competition is done with features. It also encompasses portabilitiy, ease of use, work forces familiarity and above all marketing.

226 points by bpierre 11y ago

Congratulations on 2.0! I tried out LXD a few months ago after hitting a few walls with Vagrant, and I've been happy since. Mark Shuttleworth's normally responsive on Google+, and he helped point me in the right direction with initial issues that I encountered. Stéphane and the team have also been helpful on Github with issues. The first thing I struggled with in this containers business was "what's the difference between LXC/LXD and Docker et. al.?", and I am glad that he addressed it first in the post. I've got north of 20 containers on our machines, and LXD allows me to create a container, play around with software config on it, and when I get it right; I just leave that container and update it regularly like the host it's on.

fulafel 11y ago
You can also use LXC as Vagrant backend to get away from the crashy and buggy VirtualBox kernel driver. Though ~all the Vagrant images you find online are tied to VirtualBox.
- mugsie 11y ago
  Yeah, it can be a pain to get images. That said, my workflow for testing anything these days is inside a vagrant lxc box. it is well worth the lack of images for the improved workflow.

IgorPartola 11y ago

Does LXD provide any type of cluster management behavior? As in if I have two host machines and a dozen containers and host A goes down, is it possible to configure a live migration of the latest known state from host A to host B using LXD alone? Or do I need something built on top of it to do that?

nnutter 11y ago
I think that's Juju's role. They have a beta LXD driver for Juju. https://jujucharms.com/docs/master/config-LXD https://jujucharms.com/docs/master/config-LXD
wmf 11y ago
There's an OpenStack driver for LXD.
tyingq 11y ago
LXD and any sort of cluster seem incompatible at the moment. LXD, for example, stores all the LXC config info in a local sqlite database. When you launch a container, it turns the sqlite stored info into an LXC config file, and spawns LXC. You can control remote instances of the LXD daemon, but there's no syncing back and forth from Host1-LXD1 to Host1-LXD2...they operate separately. You can migrate a container from one host to the other, but it's pushing everything (config/image/etc) when you ask it to migrate. So, it's not clustered itself, and it keeps the important info somewhere that's hard to cluster. It would be interesting if they would replace the local sqlite store with something similar to etcd. That still leaves the issue of the disk images though. Proxmox does cluster LXC, so there's that option.

bronson 11y ago

Is anybody making an effort to put the LXD environment on Centos 7? I'd really like to try it but all I have provisioned right now is Centos 7 (and below).

matt_wulfeck 11y ago

> That is, a LXD container runs a full Linux system, exactly as it would be when run on metal or in a VM. This is what makes LXD so nice. It's the power and flexibility of a VM with the speed, api, and runability of containers. I always hated that docker forced you to treat a container as a single process and spoke about it as if it's a virtue. Let me decide how many or little processes I want running in a container. Even google runs ssh server alongside their applications. With LXD you can easily port entire services over to containers without completely rewriting the way they behave.

mayank 11y ago
> I always hated that docker forced you to treat a container as a single process and spoke about it as if it's a virtue. Let me decide how many or little processes I want running in a container. You can have as many processes as you like running inside a Docker container, you just a need a single process to "boot" them, even if it's a simple shell script.
- tyingq 11y ago
  The big difference is that LXD can run, in an unprivileged fashion, a systemd based distro without issue, and with a sensible view of /proc. Docker can't currently do that, largely because running a distro in a container isn't an important goal for them.
  - lisivka 11y ago
    Docker can run systemd in unprivileged container: https://github.com/vlisivka/docker-centos7-systemd-unpriv https://github.com/vlisivka/docker-centos7-systemd-unpriv PS. Try it: docker run -dt --stop-signal=$(kill -l RTMIN+3) -v /sys/fs/cgroup:/sys/fs/cgroup:ro --name centos7_systemd vlisivka/docker-centos7-systemd-unpriv
    - tyingq 11y ago
      My impression is that someone created a specific dockerfile to get around the issues. I'm noting that LXD can run, for example, stock debian.

nzoschke 11y ago

This looks really nice. I build out major productions systems on top of LXC at Heroku, and to this day it is still working incredibly well. So I am really looking forward to further iterations of the tools by Stéphane Graber and co at Canonical. The main thing that jumps out at me is a REST API for this layer. A REST API with Golang bindings to run containers feels so much better than old-school systems programming with exec. I'm currently all in on Docker in production for this very reason. Coordinating stuff with a golang client over HTTP is really really productive.

jaytaylor 11y ago

With so many contenders stirring up the pot, this is an exciting time in the world of Linux Containers! I would love to see an LXD container driver for Mesos. Do any of you know if LXD is APPC-spec compliant?

stgraber 11y ago
It's not as the appc spec stands for "app container" and LXD doesn't run application containers but instead full system containers.

alinspired 11y ago

LXD looks similar to OpenVZ or rather Proxmox(with LXC or OpenVZ)

yo-code-sucks 11y ago
Because LXD is jusr LXC with an API.

foreigner 11y ago

What’s LXD? At its simplest, LXD is a daemon which provides a REST API to drive LXC >containers. Thanks, that clears it right up.

daurnimator 11y ago
Not sure if you're being sarcastic or not?
- nzoschke 11y ago
  No sarcasm. This is a simple explanation of where LXD fits into the stack. If you don't know what LXC is maybe this isn't clear. But LXC stands for LinuX Containers, and is extremely fundamental technology in the containerization space.
  - daurnimator 11y ago
    (You should possibly have replied to the post above me) > LXC stands for LinuX Containers, and is extremely fundamental technology in the containerization space. Not exactly. LXC was amoung the first pieces of software to take advantage of namespaces when they were added to the kernel; but LXC is not often used. Most of the various containerisation software now available does not use LXC, and it not based on the same codebase.

voltagex_ 11y ago

I'm moving my home NAS back to Debian after a couple of painful months [1] [2] with FreeNAS 9. Would this give me jail-like functionality under Linux? The one thing I enjoyed about FreeNAS/FreeBSD was how easy it was to create and tear down jails for all kinds of uses. [1]: https://blog.voltagex.org/2016/01/08/freenas-9-3-on-the-supermicro-5028a-tn4-a1sri-2758f/ https://blog.voltagex.org/2016/01/08/freenas-9-3-on-the-supe... [2]: https://lists.freebsd.org/pipermail/freebsd-net/2016-February/044619.html https://lists.freebsd.org/pipermail/freebsd-net/2016-Februar...

nine_k 11y ago
Plain LXC, rut, or docker are probably closer.
- voltagex_ 11y ago
  I like docker, but I prefer something closer to a VM. Why would you say go for LXC over LXD? Also, Googling "linux rut" doesn't bring up anything useful, I'm assuming you mean rkt
  - josteink 11y ago
    LXC is very nice. It's very VM-like and why I prefer it for almost everything I deploy instead of using Docker. I know how to manage machines. I don't have the same confidence for Docker-images. LXD seems to be a construction on top of LXC to make managing it easier "at scale", so if all you want is to manually be able to construct "container VMs" on demand, I think LXC is closer to what you actually want: It let's you manage the containers directly, near the iron, without anything getting in your way. Don't get me wrong: LXD seems to have a genuine value-proposal, but for this use-case it seems to be a slight step of indirection, and my guess is that this will cause you more troubles than it's worth.

mrbill 11y ago

Wonder if I'll be able to use LXC/LXD on "plain" Debian, and be able to avoid Ubuntu.

voidz 11y ago
Ubuntu, yes, I'd think so. Canonical, however, not. What makes you want to avoid Ubuntu?
- mrbill 11y ago
  I use "plain" Debian on my servers when possible. Personal preference, mostly. Not really a fan of Ubuntu Server.

vasi 11y ago

I like almost everything about LXD, except the attitude towards networking. The project has rejected simple solutions such as port forwarding, saying that managing the network shouldn't be LXD's job. Instead, they'd like the user to manually configure their own bridges or routes or iptables chains. I can kinda understand their point of view, there's no simple solution that will please everyone. But most developers or IT folks aren't networking experts, and LXD won't be an intuitive tool for them without a simpler mode of operation.

ansible 11y ago
I have just finished a small LXC deployment, and the networking did take a little bit of time to figure out what direction to go. I ended up choosing to use bridging to connect all the physical network adapters to the ones in the containers. This is nice because I set up each container with its own IP address, which travels with it when the container is moved to a new host.
dozzie 11y ago
> [...] most developers or IT folks aren't networking experts, and LXD won't be an intuitive tool for them without a simpler mode of operation. Really, no expertise is required. Just basic understanding how the heck the network works. If somebody can't grasp what bridge interface is or how NAT operates, he apparently doesn't have the qualifications for writing software.
- lotyrin 11y ago
  Then I guess most of the developers I've worked with should brush up their sandwich-making skills and quit their jobs.
  - dozzie 11y ago
    You do realize that this is not to their credit, right?

jaequery 11y ago

if someone can create a boot2docker clone for this, i'm in.

lolidaisuki 11y ago
Why not just use regular TinyCore? Or any other distribution? LXC containers aren't app containers. It is intended to be used with regular installs, not with single applications.

sandra_saltlake 11y ago

This looks really nice. A REST API with Golang bindings to run containers feels so much better than old-school systems programming with exec.

jccc 11y ago

It simultaneously irks me and pleases me that the first result on DuckDuckGo for "lxd" is the Legion of Extraordinary Dancers. (Google gets it right, presumably because of all my code searches.) "[...] a 2010–2011 web series about two groups of rival dancers: The Alliance of the Dark who are the villains and The Legion of Extraordinary Dancers, the heroes, who discover they have superpowers referred to as 'the Ra' through their dance abilities."

srcmap 11y ago

Very Nice. I want to understand how Linux Container works, started to study the cgroup code in Linux kernel and LXD. Here are couple of notes on Tracing the code of CGroup in Linux kernel and LXD I took. http://www.srcmap.org/sd_share/7/0cc46512/LXD_Code_Trace.html http://www.srcmap.org/sd_share/7/0cc46512/LXD_Code_Trace.htm... http://www.srcmap.org/sd_share/7/060b8d61/Linux_Kernel_CGroup_Control_Group_Code_Trace.html http://www.srcmap.org/sd_share/7/060b8d61/Linux_Kernel_CGrou...

Annatar 11y ago

If I need to run Linux software, I can use an lx-branded zone inside of SmartOS, and run at the speed of bare metal while having ZFS, mdb, kdb, dbx and DTrace, CPU-bursting, and change kernel schedulers on the fly. But then, as I am about to run Linux software inside of an lx-branded zone on SmartOS, I realize that SmartOS has 15,000 packages, with most the of the software I would have run on Linux, available natively on SmartOS: http://www.perkin.org.uk/posts/building-packages-at-scale.html http://www.perkin.org.uk/posts/building-packages-at-scale.ht... ...so then, which operational advantages would LX containers on Linux offer me, that I already do not have on SmartOS with zones?

lolidaisuki 11y ago
You can think of it as competition. Other people can think of it as something that they can make people who are already familiar with systems that use the Linux kernel.
- Annatar 11y ago
  SmartOS is paranoid about correctness of operation and data integrity; Linux and LXC containers are a poor competition in that area, when the OS comes with memory overcommit turned on by default, and default filesystems do not include data write guarantees nor block checksums...
  - lolidaisuki 11y ago
    Are you trying to convince me of something? E: Not all competition is done with features. It also encompasses portabilitiy, ease of use, work forces familiarity and above all marketing.