9 ms·

LXD 3.8 has been released

141 points by mariuz 8y ago

chrisper 8y ago
LXD is amazing. It replaced most VMs in my homeserver setup.
vikrantrathore 8y ago
Good work @stgraber and team and congratulations on continuously improving and constantly releasing new version and updates. Indeed being able to do this when majority of the container funding is going to Kubernetes or Docker is a great achievement. Has been using LXD for last 6 years in production. Its a little gem hiding with all the spotlight on Kubernetes and Docker. Indeed the way LXD works, it naturally provides a path from physical servers and VM instances to containers and back. The important part is it runs containers in userspace providing a much better security then docker, but seems developers don't care about it. Also you can use traditional ansible, puppet or chef to create and manage images and containers directly instead of learning shell scripts and Dockerfile way with additional cognitive load. The only issue I am facing is mounting a shared NFS or CIFS in userspace container using FUSE, since the client drivers for this fs needs to be run through a root process. Hopefully it will be resolved in future. Tried Lizardfs instead of NFS and the system failed in the middle and my issues on the github is there for weeks without any feedback. I will try glusterfs and also looking and beegfs lets see if it can work.
- insertnickname 8y ago
  Are you talking about LXC or LXD? I don't think LXD existed six years ago.
  - simosx 8y ago
    The first commit for LXD on github is from November 2014: commit 3153b57369b7cf3e96979897c335a5c1d69e7f07 Author: Stéphane Graber Date: Wed Nov 5 10:09:28 2014 -0500 Add licensing and contributing guidelines Signed-off-by: Stéphane Graber <stgraber@ubuntu.com> Source: https://github.com/lxc/lxd https://github.com/lxc/lxd There is a migration path from LXC to LXD and probably the OP migrated at some point in time.
- StreamBright 8y ago
  Couldn’t agree more. I think the biggest problem for more adoption is the lack of blog posts and documentation. I wish somebody would write articles about how to use LXD/LXC in production catered towards developers. Maybe even having a side by side comparison with Docker.
  - chrisper 8y ago
    I think the issue is also that for a long time (or still is?) it was Ubuntu only.
    - StreamBright 8y ago
      I don’t think so https://discuss.linuxcontainers.org/t/lxd-on-centos-7/1250 https://discuss.linuxcontainers.org/t/lxd-on-centos-7/1250
    - simosx 8y ago
      There were some issues with specific kernel features not being upstreamed timely. Having said that, here is a distribution usage for the snap package of LXD, https://snapcraft.io/lxd https://snapcraft.io/lxd (see that the end of the page).
      - stgraber 8y ago
        At this point, the only feature that I'm aware of which we have in the Ubuntu kernel and hasn't been merged upstream yet is support for overlayfs inside user namespaces. The other big one for a long time was fuse inside user namespaces, but that has been merged upstream in 4.18. Some of the AppArmor features were also a sticky point for a while as upstream was lagging behind quite a bit, but the AppArmor maintainer has since fixed that, so recent Linux kernels have everything that we use.
    - cyphar 8y ago
      LXC has worked on many distributions for a very long time, as has LXD. Most of the time, all of the main developers have been Canonical employees (with some exceptions), so this might've impacted people's view on what it supported. Also LXD's recommended distribution is through snaps (which gives it an Ubuntu-only flavour) -- but it's available in many different repositories in many different distributions.
- cyphar 8y ago
  Don't get me wrong, I'm a huge fan of LXC and I've collaborated with the LXC folks with upstream kernel work (they're really clever guys). > The important part is it runs containers in userspace providing a much better security then docker, but seems developers don't care about it. I'm really not sure what you mean here by "runs containers in userspace". LXC and runc (the runtime underneath Docker) both use the same kernel primitives and generally work in fairly similar ways. LXC has more specific work in order to be able to have more VM-like containers (such as having "console" support and the "fun" that is booting systemd inside containers), but it's fundamentally not different to runc containers. In many cases, kernel work by one of us will benefit the other (right now we're working on several kernel patches in parallel, and each of us is considering how we can integrate the others' work into our projects). I do think LXC has better protections against certain attacks (and I worked with them on some of those protections) and has a much nicer design in many aspects. So you could argue it has some security upsides over Docker or runc, but that's a different topic to what primitives they use -- they're effectively indistinguishable at that level. (I'm one of the maintainers of runc.)
  - hardwaresofton 8y ago
    I've also struggled with the distinction, but I believe the commenter above was referring to LXC (and LXD's) "system containers". Full user namespacing has been in from very early on (day 1?) which is the "increased security" -- this is where I believed the difference between "system containers" for LXC and regular "application containers" was. runc was not always the runtime underneath Docker, and IIRC it has not always had support for proper user namespacing for as long as LXC has. Also, LXC has so many more features packed in I think it's a disservice to compare it to runc in the first place. It should go without saying but I would absolutely love for you to sprinkle some knowledge on me wherever I'm wrong about either of these. [EDIT] - Getting downvotes so figured maybe I said something egregious so I did some digging: - LXC had user namespaces @ v1.0 in 2013[0] - Docker got user namespaces @ 1.10 in 2016[1] This is the distinction I meant. [0]: https://stgraber.org/2014/01/01/lxc-1-0-security-features/ https://stgraber.org/2014/01/01/lxc-1-0-security-features/ [1]: https://blog.docker.com/2016/02/docker-1-10/ https://blog.docker.com/2016/02/docker-1-10/
- viraptor 8y ago
  > Also you can use traditional ansible, puppet or chef to create and manage images and containers directly instead of learning shell scripts and Dockerfile way with additional cognitive load. You can certainly provision docker containers using ansible / chef / puppet. It's not popular, because they do way more than is required in those situations normally, but it doesn't stop you. A lot of larger applications on dockerhub are done that way - splunk's dockerfile is just an ansible starter for example. Puppet even has a page about it: https://puppet.com/blog/running-puppet-software-docker-containers https://puppet.com/blog/running-puppet-software-docker-conta...
  - u801e 8y ago
    > You can certainly provision docker containers using ansible / chef / puppet. It's not popular, because they do way more than is required in those situations normally Could you elaborate on what you're referring to what they're doing beyond what's required? > A lot of larger applications on dockerhub are done that way From what I understand, you have to write the configuration management in a different way such that it doesn't require restarting services after configuration files have been updated. Converting an existing playbook or cookbook to work that way isn't trivial depending on how many things it's doing to provision the server.
    - viraptor 8y ago
      > what they're doing beyond what's required? Chef, puppet, ansible and other config management tools are designed to run nicely in an existing system. That means if you want to do action X, they'll usually check if it's possible, try to resolve internal dependencies, handle potential errors, register the state change for report and other things. Most of that is unnecessary when you're building a docker image from scratch. You control all the dependencies - they're the lines before this one. You don't care about nice reports of partial failures - they should cause immediate aborts. You don't care about managing state - each step is it's own fully defined state. For example, if I remember correctly, chef will query available package list when you request something installed. It will also continue running independent recipes if the installation fails. When building images I want literally "apt-get install foo || crash_and_burn". > you have to write the configuration management in a different way such that it doesn't require restarting services after configuration files have been updated In my experience with cfmgmt - this needs to be implemented whether you build images or update running systems. There always comes a time when you want to change something but not trigger service updates. When building images that time is always.
- vikrantrathore 8y ago
  Also I would like to point out that the lxd community is helpful, engaging and absolutely wonderful. I had an issue with client ip address in haproxy container on Google compute instance you can see on https://discuss.linuxcontainers.org/t/how-to-get-real-client-ip-when-using-lxd-to-forward-port-80/2079 https://discuss.linuxcontainers.org/t/how-to-get-real-client... An issue was created and to my surprise a fix landed in next release. It helped me to launch haproxy containers binding port 80/443 on compute instance and have correct client ip in haproxy logs. This helped me to have analytics from haproxy logs directly.
  - simosx 8y ago
    Here is that feature request for the PROXY protocol, https://github.com/lxc/lxd/issues/4786 https://github.com/lxc/lxd/issues/4786 Was reported on July 14th and committed on July 19th.
simosx 8y ago
LXD provides "system containers" in contrast to the "application containers" from Docker. You create a "system container" and it keeps running as a stock Linux distribution (many flavors are supported). It takes about a couple of seconds to create a new container, and a bit less to remove it.
- hardwaresofton 8y ago
  LXC is capable of both[0], but Docker wasn't for a long time -- but it got the marketshare IMO because of how approachable it was and the community effort.. [0]: https://linuxcontainers.org/lxc/introduction/ https://linuxcontainers.org/lxc/introduction/
- CameronNemo 8y ago
  Note that LXC handles application containers perfectly fine now, and it does so in a pretty simple way. Read up on the lxc-oci template and the lxc-execute command if you would like to play around with it!
artellectual 8y ago
I've replaced kubernetes / docker with LXD to host https://www.codemy.net/posts https://www.codemy.net/posts and various other services. All I have to say is overall much less moving parts, much lower management cost (uses less time) and much more efficient workflow. For me using LXD has been a dream. Congrats to the team. I plan on releasing some of the work I've been doing on LXD soon, hopefully it will push lxd into the spotlight over kubernetes and docker.
- gtirloni 8y ago
  How many servers are you using to host that?
  - artellectual 8y ago
    TL;DR 5 but I could be using 1 I have 5 machine in a cluster right now running various services, it is definitely over provisioned, I'm barely using the machine's resources. I could technically host the whole thing on one server, but given what I'm building, I wanted to test out how my solution scales in a clustered setup. It's got an elixir backend, so the elixir apps are also clustered sharing data using LXDs fan networking. All the services get deployed / updated into the cluster via an automated CI / CD
- Uberphallus 8y ago
  How is orchestration done with LXD instead of Kubernetes?
  - cyphar 8y ago
    LXD supports orchestration since 3.0. It's actually quite neat -- you can do live-migration between different hosts. The design is very much "fediverse" where you start containers on different hosts (so it doesn't quite fit the Kubernetes "purge and rebuild" orchestration model).
  - artellectual 8y ago
    I'm using terraform (it has an LXD plugin) to handle the orchestration in the CI / CD server. So basically. I push code terraform detects if container running service exists, if not it creates the containers, if it does it will upgrade the service in the existing containers without destroying, so my deploys are super fast. I will have a more detailed video series coming out soon on the site I mentioned above.
hardwaresofton 8y ago
Possibly an unpopular opinion, but I believe one could easily build a better platform than Kubernetes as it sits now on top of LXD. The amount of features baked into LXC and exposed by LXD is bonkers. LXC doesn't have the flexibility (for example you can't just add another runtime underneath it), but the amount of features it has packed in it could have given earlier k8s versions a run for their money. Also, a lot of the implementation flexibility that Kubernetes is seeing now (ex. runtimeClass) is the result of years of inflexibility and people finding ways to work around it (frakti, etc) and build at a slower pace (which is also a good thing, slow measured progress and all that) -- my point is that LXC might have have been the better base, all things equal. Moving VM workloads right into LXD without needing to port them to be "cloud native" might have saved people a bunch of time, never having to deal with complexities of overlay networks (as long as you could set up LXD properly) would also likely have saved people some time.
- gtirloni 8y ago
  I think it's debatable if adopting LXC/LXD early in the project would have helped. In any case, all you have to do these days is to satisfy the Container Runtime Interface (CRI). https://github.com/ktsakalozos/lxdlet https://github.com/ktsakalozos/lxdlet https://github.com/enzian/kube-lxd https://github.com/enzian/kube-lxd
  - hardwaresofton 8y ago
    I did not know these existed, thank you so much. I was going to try and write an runtimeClass-compliant operator or a containerd v2 runtime for LXD system containers one of these weekends. kube-lxd and lxdlet seem pretty old though containerd has moved at breakneck speed in the last year... Looking through the code it might need some TLC
simosx 8y ago
LXD is used in Crostini (https://www.reddit.com/r/Crostini/ https://www.reddit.com/r/Crostini/) as a way to be able to run Linux applications in chromebooks. In addition, Samsung is somehow using LXD in their new phones, with "Linux on DeX". In that way, you can get a Linux interface with Ubuntu when you plug your phone to a monitor/TV. If you use a Linux desktop, you can get to run GUI applications in a LXD "system container", therefore isolating the files from your host's filesystem. Here is how to do this by sharing the X server of the host (filesystem separation but sharing the X for convenience), https://blog.simos.info/how-to-easily-run-graphics-accelerated-gui-apps-in-lxd-containers-on-your-ubuntu-desktop/ https://blog.simos.info/how-to-easily-run-graphics-accelerat... You can run things like CS GO DeathZone in such a LXD container with full GPU acceleration, and even closed-source NVidia driver. Some people also got an LXD container to run in a separate X server, thus having even better isolation.
- zertrin 8y ago
  I use LXD containers to develop an app with a lot of big external dependencies that are currently not yet managed in a "nice" way (i.e. for now it's extract source tarball / make / make install). I installed CLion inside the container and use the host X server to display the IDE and the app being developed. The containers are so useful to isolate the main system from these make install and to be able to install the dependencies or rollback to a previous version in case of mistake without fearing side effects on the host system.
- shittyadmin 8y ago
  Thanks for that link on an easier way to run GUI apps through LXD. I was using LXD containers to experiment with some cryptocurrencies and wound up using SSH and X forwarding to run the GUIs, that looks like a much better option.
OnAndOnAndOn 8y ago
What has constantly bothered me about LXD is how heavily tied it is to Ubuntu. With Docker, you do not need to worry about moving to another major distro like Fedora, CentOS, Debian, etc. - it's officially supported and readily available. The LXD answer to this is "snaps", which are a whole extra packaging system and yet another Ubuntu technology you have to invest in - making you start to wonder if you're swallowing a spider to catch a fly. What then increases my concern is that the two other major technology pushes Ubuntu has made into the wider ecosystem - Upstart and Mir - have both failed to take off. Adopting a niche technology - especially for something as fundamental as a hypervisor - is already a hard choice. The LXD team is making adoption even harder by not offering official repos for the major distros.
- simosx 8y ago
  As a user you should be unaffected by the packaging of software as long as it works. A Linux distribution can repackage LXD in their native packaging format and some distributions are doing it already (Debian, Alpine, Fedora, etc). If you are familiar with packaging, you can help promote those packages to the official repositories. Debian: https://wiki.debian.org/LXD https://wiki.debian.org/LXD Fedora: https://copr.fedorainfracloud.org/coprs/ganto/lxd/ https://copr.fedorainfracloud.org/coprs/ganto/lxd/ Alpine: https://git.alpinelinux.org/cgit/aports/tree/testing/lxd?h=master https://git.alpinelinux.org/cgit/aports/tree/testing/lxd?h=m...
  - OnAndOnAndOn 8y ago
    As a user, you should be unaffected, in an ideal world. However that is not the case. You should always be concerned about the security of your supply chain, and should always be pulling packages from official sources only (official by the standards of the distro and/or developers you trust).
- cyphar 8y ago
  LXD is available as a package on many distributions. While they do recommend snaps (because they can push you updates faster than the distribution), that's not the only way of getting it. Also, while the main development team are Canonical employees, LXC existed long before the lead developer was hired by Canonical -- and predates many projects. It's not going to be going away soon -- I'd bet that the development team would just move to a different employer if Canonical wanted to stop supporting its development.
curt15 8y ago
I've dabbled occasionally with Fedora's new "Silverblue" edition which provides atomic system updates and expects users to do much of their work in "pet" docker containers. However, I understand that docker subscribes to the single-application-per-container philosophy which makes it a little awkward to adapt to the pet container workflow (for instance, systemd doesn't work inside). LXD, being closer to a full-blooded VM, would seem better suited to such an application.
- Spivak 8y ago
  You can absolutely run systemd in a container. Red Hat has put a lot of work into this arrangement. https://developers.redhat.com/blog/2016/09/13/running-systemd-in-a-non-privileged-container/ https://developers.redhat.com/blog/2016/09/13/running-system...
  - CameronNemo 8y ago
    You can, but why would you want to?
    - mixmastamyk 8y ago
      To restart a process if it crashes?
- moosingin3space 8y ago
  For these "pet containers", Fedora has been building this: https://github.com/debarshiray/fedora-toolbox https://github.com/debarshiray/fedora-toolbox. The Toolbox is built on top of runc, but uses it to run a mutable container. I'm excited about Silverblue, just need to get a few more apps into sandbox-ready Flatpaks...(Firefox, I'm looking at you)
Jnr 8y ago
I am unsure how you are supposed to use LXD since from the readme is seems like it is only a REST interface? Why not use lxc directly? But I am a big fan of lxc. I have most of my stuff in containers now-days and it has been a very pleasant experience using lxc.
- simosx 8y ago
  If you use the command "lxc" to manage your container, then you are using LXD. The command "lxc" communicates with the LXD supervisor over REST for you. If instead you are using commands like "lxc-start", then you are using "LXC". There are two possible aspects of confusion: 1. LXD uses the "lxc" command line utility for all the management of LXD containers. 2. LXC is both the name for the Linux kernel "Linux Containers" functionality and the first/early implementation of tools (those "lxc-????") for Linux Containers.
nickjj 8y ago
LXCs are still useful to use in a world with Docker. For example, if you're testing your infrastructure configuration code, you need some type of server to run it against. Using a base Ubuntu 18.04 Docker image might work, but there's a lot of things that Docker is incapable of allowing you to configure, where as with an LXC, you can. In other words, using Docker as an infrastructure target gives me about 80% confidence that things will work on a real host, but with LXCs that confidence level goes up to 99%. So it's common to have a snapshotted LXC image that you can use as a target system, so you can spin up a new fresh LXC based Linux distro in a few seconds. It's a lot faster than using something like Vagrant and fills a much needed gap for testing how you provision servers.
jlokier 8y ago
LXD is nice. I use it for most of my services where a VM isn't required (e.g. because of running Windows or a particular old kernel). I particularly like that it replaces what I used to use debootstrap for, and quicker too. Unfortunately, I've got some older LXC (v1.0) containers that fail to boot under LXD, which means I need to have both container systems installed on my hosts. It's unfortunate that this means I have to have both the "dash" commands like `lxc-ls` and the "space" commands `lxc ls` which return separate lists. That's because one of them is the LXC project, and one of them is the LXD project, confusingly using very similar names for commands. The one feature of LXC that I really miss in LXD is the `lxc-console` command. Among other things, without console it's harder to debug those LXC containers to figure out why they aren't booting under LXD.
- sciurus 8y ago
  Why can't you replace the older LXC containers with new ones managed via LXD?
  - jlokier 8y ago
    Because they don't boot successfully in LXD - their init doesn't like the LXD environment. Obviously this can be fixed but it's non-trivial to figure out why. For the time being it's easier to keep using LXC for those.
    - jlokier 8y ago
      Oh, perhaps I misunderstood your question. If you mean why can't I completely replace them with newly built containers, it's because they are complete but rather old OS images with years of configuration history in them, running specific versions of mission critical applications. They are older than LXC, and were originally run on real servers, then under chroot and UML, then under KVM, before LXC or Docker were a thing. To be tinkered with most carefully. As much as I would like to restart from scratch, with new versions of everything, experience has taught me it would be unwise to do it in a hurry, or without extensive testing. So yes they can be replaced and will be eventually, but it's not a small job and will need time scheduled to it. For the time being it's the Right Thing To Do(tm) to keep the existing images still running as they are.
LinuxBender 8y ago
I've heard a lot about LXD from a coworker and would love to try it. From what I have found, I have to install and run snapd on CentOS. [1]. Are there plans to remove this requirement in the future? It looks to be in edge/testing on Alpine so that is promising. [2] [1] - https://discuss.linuxcontainers.org/t/lxd-on-centos-7/1250 https://discuss.linuxcontainers.org/t/lxd-on-centos-7/1250 [2] - https://pkgs.alpinelinux.org/packages?name=lxd&branch=edge https://pkgs.alpinelinux.org/packages?name=lxd&branch=edge
- simosx 8y ago
  Currently I think the official LXD packages will be snap packages for all distros, with the exception of the Ubuntu LTS releases. That is, in Ubuntu 16.04 LTS and Ubuntu 18.04 LTS you get LXD as an official deb package. Obviously, this does not preclude the distributions from packaging LXD in their own packaging format. Apart from the AlpineLinux effort to package LXD, there is also a Debian effort, https://wiki.debian.org/LXD https://wiki.debian.org/LXD
  - stgraber 8y ago
    Note that Ubuntu 20.04 will be using the snap package, so we only really need to focus on one thing to validate for all distros. Users upgrading from 18.04 to 18.10 are already being automatically migrated over to the snap. We have 3 tracks for the snap right now: - 2.0 LTS - 3.0 LTS - latest So users that want to stay on an upstream LTS release can still do even after switching to the snap.
- u801e 8y ago
  They have packages in the copr repository for Fedora[1], but I haven't found anything similar for CentOS. [1] https://copr.fedorainfracloud.org/coprs/ganto/lxd/ https://copr.fedorainfracloud.org/coprs/ganto/lxd/
- CameronNemo 8y ago
  That Alpine Linux package is woefully out of date (3.2), and will not compile with more recent versions of Go, AFAIK. Void Linux has LXD LTS (3.0.3) and latest (3.8) available.
fasteo 8y ago
I am quite happy using LXD. Is there any built-in way to expose a tcp port ? I am using iptables in the host to do this. thanks
- artellectual 8y ago
  LXD has proxy devices, which allows you to map your host's port to your containers ports. https://github.com/lxc/lxd/blob/master/doc/containers.md#type-proxy https://github.com/lxc/lxd/blob/master/doc/containers.md#typ... Makes life very easy, I used to have to mess with iptables too but with proxy device things became much simpler.
  - fasteo 8y ago
    ahh, much better. Thanks for the pointer!
  - indigodaddy 8y ago
    This would force you to https on the lxc/LXD endpoint that you proxied to correct? I prefer to just iptables 80/443 (guess could proxy that part too instead of iptables) to an LXD HAProxy container, and then have HAProxy do the proxying and domain-based load-balancing/acl's, coupled with the advantage of terminated SSL in one place (the HAproxy container) for all (sub)domains/services.
jpeeler 8y ago
Does anybody know of a recent (given how quickly this space moves) comparison of LXD vs systemd-nspawn? I assume that the latter has less features given the primary reason for its existence is to test systemd, but I'd love to know more.
- CameronNemo 8y ago
  systemd-nspawn is a very simple container runtime. LXD is a hypervisor. They aren't great comparison points.
  - jpeeler 8y ago
    That's fair. What I really want to know is how systemd-nspawn compares to LXC. Reading further, I see that LXD uses liblxc for container management. This comment about LXD using lxc command naming also was helpful: https://news.ycombinator.com/item?id=18672510 https://news.ycombinator.com/item?id=18672510
sigjuice 8y ago
I have tried LXD casually over the years and it might seem frivolous, but I can't get past the lxc commands making rectangles in my terminals. Is there a way to disable that and output plain lines like most Unix commands?
- mixmastamyk 8y ago
  There's no quiet option?

141 points by mariuz 8y ago

LXD is amazing. It replaced most VMs in my homeserver setup.

Good work @stgraber and team and congratulations on continuously improving and constantly releasing new version and updates. Indeed being able to do this when majority of the container funding is going to Kubernetes or Docker is a great achievement. Has been using LXD for last 6 years in production. Its a little gem hiding with all the spotlight on Kubernetes and Docker. Indeed the way LXD works, it naturally provides a path from physical servers and VM instances to containers and back. The important part is it runs containers in userspace providing a much better security then docker, but seems developers don't care about it. Also you can use traditional ansible, puppet or chef to create and manage images and containers directly instead of learning shell scripts and Dockerfile way with additional cognitive load. The only issue I am facing is mounting a shared NFS or CIFS in userspace container using FUSE, since the client drivers for this fs needs to be run through a root process. Hopefully it will be resolved in future. Tried Lizardfs instead of NFS and the system failed in the middle and my issues on the github is there for weeks without any feedback. I will try glusterfs and also looking and beegfs lets see if it can work.

insertnickname 8y ago
Are you talking about LXC or LXD? I don't think LXD existed six years ago.
- simosx 8y ago
  The first commit for LXD on github is from November 2014: commit 3153b57369b7cf3e96979897c335a5c1d69e7f07 Author: Stéphane Graber Date: Wed Nov 5 10:09:28 2014 -0500 Add licensing and contributing guidelines Signed-off-by: Stéphane Graber <stgraber@ubuntu.com> Source: https://github.com/lxc/lxd https://github.com/lxc/lxd There is a migration path from LXC to LXD and probably the OP migrated at some point in time.
StreamBright 8y ago
Couldn’t agree more. I think the biggest problem for more adoption is the lack of blog posts and documentation. I wish somebody would write articles about how to use LXD/LXC in production catered towards developers. Maybe even having a side by side comparison with Docker.
- chrisper 8y ago
  I think the issue is also that for a long time (or still is?) it was Ubuntu only.
  - StreamBright 8y ago
    I don’t think so https://discuss.linuxcontainers.org/t/lxd-on-centos-7/1250 https://discuss.linuxcontainers.org/t/lxd-on-centos-7/1250
  - simosx 8y ago
    There were some issues with specific kernel features not being upstreamed timely. Having said that, here is a distribution usage for the snap package of LXD, https://snapcraft.io/lxd https://snapcraft.io/lxd (see that the end of the page).
    - stgraber 8y ago
      At this point, the only feature that I'm aware of which we have in the Ubuntu kernel and hasn't been merged upstream yet is support for overlayfs inside user namespaces. The other big one for a long time was fuse inside user namespaces, but that has been merged upstream in 4.18. Some of the AppArmor features were also a sticky point for a while as upstream was lagging behind quite a bit, but the AppArmor maintainer has since fixed that, so recent Linux kernels have everything that we use.
  - cyphar 8y ago
    LXC has worked on many distributions for a very long time, as has LXD. Most of the time, all of the main developers have been Canonical employees (with some exceptions), so this might've impacted people's view on what it supported. Also LXD's recommended distribution is through snaps (which gives it an Ubuntu-only flavour) -- but it's available in many different repositories in many different distributions.
cyphar 8y ago
Don't get me wrong, I'm a huge fan of LXC and I've collaborated with the LXC folks with upstream kernel work (they're really clever guys). > The important part is it runs containers in userspace providing a much better security then docker, but seems developers don't care about it. I'm really not sure what you mean here by "runs containers in userspace". LXC and runc (the runtime underneath Docker) both use the same kernel primitives and generally work in fairly similar ways. LXC has more specific work in order to be able to have more VM-like containers (such as having "console" support and the "fun" that is booting systemd inside containers), but it's fundamentally not different to runc containers. In many cases, kernel work by one of us will benefit the other (right now we're working on several kernel patches in parallel, and each of us is considering how we can integrate the others' work into our projects). I do think LXC has better protections against certain attacks (and I worked with them on some of those protections) and has a much nicer design in many aspects. So you could argue it has some security upsides over Docker or runc, but that's a different topic to what primitives they use -- they're effectively indistinguishable at that level. (I'm one of the maintainers of runc.)
- hardwaresofton 8y ago
  I've also struggled with the distinction, but I believe the commenter above was referring to LXC (and LXD's) "system containers". Full user namespacing has been in from very early on (day 1?) which is the "increased security" -- this is where I believed the difference between "system containers" for LXC and regular "application containers" was. runc was not always the runtime underneath Docker, and IIRC it has not always had support for proper user namespacing for as long as LXC has. Also, LXC has so many more features packed in I think it's a disservice to compare it to runc in the first place. It should go without saying but I would absolutely love for you to sprinkle some knowledge on me wherever I'm wrong about either of these. [EDIT] - Getting downvotes so figured maybe I said something egregious so I did some digging: - LXC had user namespaces @ v1.0 in 2013[0] - Docker got user namespaces @ 1.10 in 2016[1] This is the distinction I meant. [0]: https://stgraber.org/2014/01/01/lxc-1-0-security-features/ https://stgraber.org/2014/01/01/lxc-1-0-security-features/ [1]: https://blog.docker.com/2016/02/docker-1-10/ https://blog.docker.com/2016/02/docker-1-10/
viraptor 8y ago
> Also you can use traditional ansible, puppet or chef to create and manage images and containers directly instead of learning shell scripts and Dockerfile way with additional cognitive load. You can certainly provision docker containers using ansible / chef / puppet. It's not popular, because they do way more than is required in those situations normally, but it doesn't stop you. A lot of larger applications on dockerhub are done that way - splunk's dockerfile is just an ansible starter for example. Puppet even has a page about it: https://puppet.com/blog/running-puppet-software-docker-containers https://puppet.com/blog/running-puppet-software-docker-conta...
- u801e 8y ago
  > You can certainly provision docker containers using ansible / chef / puppet. It's not popular, because they do way more than is required in those situations normally Could you elaborate on what you're referring to what they're doing beyond what's required? > A lot of larger applications on dockerhub are done that way From what I understand, you have to write the configuration management in a different way such that it doesn't require restarting services after configuration files have been updated. Converting an existing playbook or cookbook to work that way isn't trivial depending on how many things it's doing to provision the server.
  - viraptor 8y ago
    > what they're doing beyond what's required? Chef, puppet, ansible and other config management tools are designed to run nicely in an existing system. That means if you want to do action X, they'll usually check if it's possible, try to resolve internal dependencies, handle potential errors, register the state change for report and other things. Most of that is unnecessary when you're building a docker image from scratch. You control all the dependencies - they're the lines before this one. You don't care about nice reports of partial failures - they should cause immediate aborts. You don't care about managing state - each step is it's own fully defined state. For example, if I remember correctly, chef will query available package list when you request something installed. It will also continue running independent recipes if the installation fails. When building images I want literally "apt-get install foo || crash_and_burn". > you have to write the configuration management in a different way such that it doesn't require restarting services after configuration files have been updated In my experience with cfmgmt - this needs to be implemented whether you build images or update running systems. There always comes a time when you want to change something but not trigger service updates. When building images that time is always.
vikrantrathore 8y ago
Also I would like to point out that the lxd community is helpful, engaging and absolutely wonderful. I had an issue with client ip address in haproxy container on Google compute instance you can see on https://discuss.linuxcontainers.org/t/how-to-get-real-client-ip-when-using-lxd-to-forward-port-80/2079 https://discuss.linuxcontainers.org/t/how-to-get-real-client... An issue was created and to my surprise a fix landed in next release. It helped me to launch haproxy containers binding port 80/443 on compute instance and have correct client ip in haproxy logs. This helped me to have analytics from haproxy logs directly.
- simosx 8y ago
  Here is that feature request for the PROXY protocol, https://github.com/lxc/lxd/issues/4786 https://github.com/lxc/lxd/issues/4786 Was reported on July 14th and committed on July 19th.

simosx 8y ago

LXD provides "system containers" in contrast to the "application containers" from Docker. You create a "system container" and it keeps running as a stock Linux distribution (many flavors are supported). It takes about a couple of seconds to create a new container, and a bit less to remove it.

hardwaresofton 8y ago
LXC is capable of both[0], but Docker wasn't for a long time -- but it got the marketshare IMO because of how approachable it was and the community effort.. [0]: https://linuxcontainers.org/lxc/introduction/ https://linuxcontainers.org/lxc/introduction/
CameronNemo 8y ago
Note that LXC handles application containers perfectly fine now, and it does so in a pretty simple way. Read up on the lxc-oci template and the lxc-execute command if you would like to play around with it!

artellectual 8y ago

I've replaced kubernetes / docker with LXD to host https://www.codemy.net/posts https://www.codemy.net/posts and various other services. All I have to say is overall much less moving parts, much lower management cost (uses less time) and much more efficient workflow. For me using LXD has been a dream. Congrats to the team. I plan on releasing some of the work I've been doing on LXD soon, hopefully it will push lxd into the spotlight over kubernetes and docker.

gtirloni 8y ago
How many servers are you using to host that?
- artellectual 8y ago
  TL;DR 5 but I could be using 1 I have 5 machine in a cluster right now running various services, it is definitely over provisioned, I'm barely using the machine's resources. I could technically host the whole thing on one server, but given what I'm building, I wanted to test out how my solution scales in a clustered setup. It's got an elixir backend, so the elixir apps are also clustered sharing data using LXDs fan networking. All the services get deployed / updated into the cluster via an automated CI / CD
Uberphallus 8y ago
How is orchestration done with LXD instead of Kubernetes?
- cyphar 8y ago
  LXD supports orchestration since 3.0. It's actually quite neat -- you can do live-migration between different hosts. The design is very much "fediverse" where you start containers on different hosts (so it doesn't quite fit the Kubernetes "purge and rebuild" orchestration model).
- artellectual 8y ago
  I'm using terraform (it has an LXD plugin) to handle the orchestration in the CI / CD server. So basically. I push code terraform detects if container running service exists, if not it creates the containers, if it does it will upgrade the service in the existing containers without destroying, so my deploys are super fast. I will have a more detailed video series coming out soon on the site I mentioned above.

hardwaresofton 8y ago

Possibly an unpopular opinion, but I believe one could easily build a better platform than Kubernetes as it sits now on top of LXD. The amount of features baked into LXC and exposed by LXD is bonkers. LXC doesn't have the flexibility (for example you can't just add another runtime underneath it), but the amount of features it has packed in it could have given earlier k8s versions a run for their money. Also, a lot of the implementation flexibility that Kubernetes is seeing now (ex. runtimeClass) is the result of years of inflexibility and people finding ways to work around it (frakti, etc) and build at a slower pace (which is also a good thing, slow measured progress and all that) -- my point is that LXC might have have been the better base, all things equal. Moving VM workloads right into LXD without needing to port them to be "cloud native" might have saved people a bunch of time, never having to deal with complexities of overlay networks (as long as you could set up LXD properly) would also likely have saved people some time.

gtirloni 8y ago
I think it's debatable if adopting LXC/LXD early in the project would have helped. In any case, all you have to do these days is to satisfy the Container Runtime Interface (CRI). https://github.com/ktsakalozos/lxdlet https://github.com/ktsakalozos/lxdlet https://github.com/enzian/kube-lxd https://github.com/enzian/kube-lxd
- hardwaresofton 8y ago
  I did not know these existed, thank you so much. I was going to try and write an runtimeClass-compliant operator or a containerd v2 runtime for LXD system containers one of these weekends. kube-lxd and lxdlet seem pretty old though containerd has moved at breakneck speed in the last year... Looking through the code it might need some TLC

simosx 8y ago

LXD is used in Crostini (https://www.reddit.com/r/Crostini/ https://www.reddit.com/r/Crostini/) as a way to be able to run Linux applications in chromebooks. In addition, Samsung is somehow using LXD in their new phones, with "Linux on DeX". In that way, you can get a Linux interface with Ubuntu when you plug your phone to a monitor/TV. If you use a Linux desktop, you can get to run GUI applications in a LXD "system container", therefore isolating the files from your host's filesystem. Here is how to do this by sharing the X server of the host (filesystem separation but sharing the X for convenience), https://blog.simos.info/how-to-easily-run-graphics-accelerated-gui-apps-in-lxd-containers-on-your-ubuntu-desktop/ https://blog.simos.info/how-to-easily-run-graphics-accelerat... You can run things like CS GO DeathZone in such a LXD container with full GPU acceleration, and even closed-source NVidia driver. Some people also got an LXD container to run in a separate X server, thus having even better isolation.

zertrin 8y ago
I use LXD containers to develop an app with a lot of big external dependencies that are currently not yet managed in a "nice" way (i.e. for now it's extract source tarball / make / make install). I installed CLion inside the container and use the host X server to display the IDE and the app being developed. The containers are so useful to isolate the main system from these make install and to be able to install the dependencies or rollback to a previous version in case of mistake without fearing side effects on the host system.
shittyadmin 8y ago
Thanks for that link on an easier way to run GUI apps through LXD. I was using LXD containers to experiment with some cryptocurrencies and wound up using SSH and X forwarding to run the GUIs, that looks like a much better option.

OnAndOnAndOn 8y ago

What has constantly bothered me about LXD is how heavily tied it is to Ubuntu. With Docker, you do not need to worry about moving to another major distro like Fedora, CentOS, Debian, etc. - it's officially supported and readily available. The LXD answer to this is "snaps", which are a whole extra packaging system and yet another Ubuntu technology you have to invest in - making you start to wonder if you're swallowing a spider to catch a fly. What then increases my concern is that the two other major technology pushes Ubuntu has made into the wider ecosystem - Upstart and Mir - have both failed to take off. Adopting a niche technology - especially for something as fundamental as a hypervisor - is already a hard choice. The LXD team is making adoption even harder by not offering official repos for the major distros.

simosx 8y ago
As a user you should be unaffected by the packaging of software as long as it works. A Linux distribution can repackage LXD in their native packaging format and some distributions are doing it already (Debian, Alpine, Fedora, etc). If you are familiar with packaging, you can help promote those packages to the official repositories. Debian: https://wiki.debian.org/LXD https://wiki.debian.org/LXD Fedora: https://copr.fedorainfracloud.org/coprs/ganto/lxd/ https://copr.fedorainfracloud.org/coprs/ganto/lxd/ Alpine: https://git.alpinelinux.org/cgit/aports/tree/testing/lxd?h=master https://git.alpinelinux.org/cgit/aports/tree/testing/lxd?h=m...
- OnAndOnAndOn 8y ago
  As a user, you should be unaffected, in an ideal world. However that is not the case. You should always be concerned about the security of your supply chain, and should always be pulling packages from official sources only (official by the standards of the distro and/or developers you trust).
cyphar 8y ago
LXD is available as a package on many distributions. While they do recommend snaps (because they can push you updates faster than the distribution), that's not the only way of getting it. Also, while the main development team are Canonical employees, LXC existed long before the lead developer was hired by Canonical -- and predates many projects. It's not going to be going away soon -- I'd bet that the development team would just move to a different employer if Canonical wanted to stop supporting its development.

curt15 8y ago

I've dabbled occasionally with Fedora's new "Silverblue" edition which provides atomic system updates and expects users to do much of their work in "pet" docker containers. However, I understand that docker subscribes to the single-application-per-container philosophy which makes it a little awkward to adapt to the pet container workflow (for instance, systemd doesn't work inside). LXD, being closer to a full-blooded VM, would seem better suited to such an application.

Spivak 8y ago
You can absolutely run systemd in a container. Red Hat has put a lot of work into this arrangement. https://developers.redhat.com/blog/2016/09/13/running-systemd-in-a-non-privileged-container/ https://developers.redhat.com/blog/2016/09/13/running-system...
- CameronNemo 8y ago
  You can, but why would you want to?
  - mixmastamyk 8y ago
    To restart a process if it crashes?
moosingin3space 8y ago
For these "pet containers", Fedora has been building this: https://github.com/debarshiray/fedora-toolbox https://github.com/debarshiray/fedora-toolbox. The Toolbox is built on top of runc, but uses it to run a mutable container. I'm excited about Silverblue, just need to get a few more apps into sandbox-ready Flatpaks...(Firefox, I'm looking at you)

Jnr 8y ago

I am unsure how you are supposed to use LXD since from the readme is seems like it is only a REST interface? Why not use lxc directly? But I am a big fan of lxc. I have most of my stuff in containers now-days and it has been a very pleasant experience using lxc.

simosx 8y ago
If you use the command "lxc" to manage your container, then you are using LXD. The command "lxc" communicates with the LXD supervisor over REST for you. If instead you are using commands like "lxc-start", then you are using "LXC". There are two possible aspects of confusion: 1. LXD uses the "lxc" command line utility for all the management of LXD containers. 2. LXC is both the name for the Linux kernel "Linux Containers" functionality and the first/early implementation of tools (those "lxc-????") for Linux Containers.

nickjj 8y ago

LXCs are still useful to use in a world with Docker. For example, if you're testing your infrastructure configuration code, you need some type of server to run it against. Using a base Ubuntu 18.04 Docker image might work, but there's a lot of things that Docker is incapable of allowing you to configure, where as with an LXC, you can. In other words, using Docker as an infrastructure target gives me about 80% confidence that things will work on a real host, but with LXCs that confidence level goes up to 99%. So it's common to have a snapshotted LXC image that you can use as a target system, so you can spin up a new fresh LXC based Linux distro in a few seconds. It's a lot faster than using something like Vagrant and fills a much needed gap for testing how you provision servers.

jlokier 8y ago

LXD is nice. I use it for most of my services where a VM isn't required (e.g. because of running Windows or a particular old kernel). I particularly like that it replaces what I used to use debootstrap for, and quicker too. Unfortunately, I've got some older LXC (v1.0) containers that fail to boot under LXD, which means I need to have both container systems installed on my hosts. It's unfortunate that this means I have to have both the "dash" commands like `lxc-ls` and the "space" commands `lxc ls` which return separate lists. That's because one of them is the LXC project, and one of them is the LXD project, confusingly using very similar names for commands. The one feature of LXC that I really miss in LXD is the `lxc-console` command. Among other things, without console it's harder to debug those LXC containers to figure out why they aren't booting under LXD.

sciurus 8y ago
Why can't you replace the older LXC containers with new ones managed via LXD?
- jlokier 8y ago
  Because they don't boot successfully in LXD - their init doesn't like the LXD environment. Obviously this can be fixed but it's non-trivial to figure out why. For the time being it's easier to keep using LXC for those.
  - jlokier 8y ago
    Oh, perhaps I misunderstood your question. If you mean why can't I completely replace them with newly built containers, it's because they are complete but rather old OS images with years of configuration history in them, running specific versions of mission critical applications. They are older than LXC, and were originally run on real servers, then under chroot and UML, then under KVM, before LXC or Docker were a thing. To be tinkered with most carefully. As much as I would like to restart from scratch, with new versions of everything, experience has taught me it would be unwise to do it in a hurry, or without extensive testing. So yes they can be replaced and will be eventually, but it's not a small job and will need time scheduled to it. For the time being it's the Right Thing To Do(tm) to keep the existing images still running as they are.

LinuxBender 8y ago

I've heard a lot about LXD from a coworker and would love to try it. From what I have found, I have to install and run snapd on CentOS. [1]. Are there plans to remove this requirement in the future? It looks to be in edge/testing on Alpine so that is promising. [2] [1] - https://discuss.linuxcontainers.org/t/lxd-on-centos-7/1250 https://discuss.linuxcontainers.org/t/lxd-on-centos-7/1250 [2] - https://pkgs.alpinelinux.org/packages?name=lxd&branch=edge https://pkgs.alpinelinux.org/packages?name=lxd&branch=edge

simosx 8y ago
Currently I think the official LXD packages will be snap packages for all distros, with the exception of the Ubuntu LTS releases. That is, in Ubuntu 16.04 LTS and Ubuntu 18.04 LTS you get LXD as an official deb package. Obviously, this does not preclude the distributions from packaging LXD in their own packaging format. Apart from the AlpineLinux effort to package LXD, there is also a Debian effort, https://wiki.debian.org/LXD https://wiki.debian.org/LXD
- stgraber 8y ago
  Note that Ubuntu 20.04 will be using the snap package, so we only really need to focus on one thing to validate for all distros. Users upgrading from 18.04 to 18.10 are already being automatically migrated over to the snap. We have 3 tracks for the snap right now: - 2.0 LTS - 3.0 LTS - latest So users that want to stay on an upstream LTS release can still do even after switching to the snap.
u801e 8y ago
They have packages in the copr repository for Fedora[1], but I haven't found anything similar for CentOS. [1] https://copr.fedorainfracloud.org/coprs/ganto/lxd/ https://copr.fedorainfracloud.org/coprs/ganto/lxd/
CameronNemo 8y ago
That Alpine Linux package is woefully out of date (3.2), and will not compile with more recent versions of Go, AFAIK. Void Linux has LXD LTS (3.0.3) and latest (3.8) available.

fasteo 8y ago

I am quite happy using LXD. Is there any built-in way to expose a tcp port ? I am using iptables in the host to do this. thanks

artellectual 8y ago
LXD has proxy devices, which allows you to map your host's port to your containers ports. https://github.com/lxc/lxd/blob/master/doc/containers.md#type-proxy https://github.com/lxc/lxd/blob/master/doc/containers.md#typ... Makes life very easy, I used to have to mess with iptables too but with proxy device things became much simpler.
- fasteo 8y ago
  ahh, much better. Thanks for the pointer!
- indigodaddy 8y ago
  This would force you to https on the lxc/LXD endpoint that you proxied to correct? I prefer to just iptables 80/443 (guess could proxy that part too instead of iptables) to an LXD HAProxy container, and then have HAProxy do the proxying and domain-based load-balancing/acl's, coupled with the advantage of terminated SSL in one place (the HAproxy container) for all (sub)domains/services.

jpeeler 8y ago

Does anybody know of a recent (given how quickly this space moves) comparison of LXD vs systemd-nspawn? I assume that the latter has less features given the primary reason for its existence is to test systemd, but I'd love to know more.

CameronNemo 8y ago
systemd-nspawn is a very simple container runtime. LXD is a hypervisor. They aren't great comparison points.
- jpeeler 8y ago
  That's fair. What I really want to know is how systemd-nspawn compares to LXC. Reading further, I see that LXD uses liblxc for container management. This comment about LXD using lxc command naming also was helpful: https://news.ycombinator.com/item?id=18672510 https://news.ycombinator.com/item?id=18672510

sigjuice 8y ago

I have tried LXD casually over the years and it might seem frivolous, but I can't get past the lxc commands making rectangles in my terminals. Is there a way to disable that and output plain lines like most Unix commands?

mixmastamyk 8y ago
There's no quiet option?