5 ms·

Good work @stgraber and team and congratulations on continuously improving and constantly releasing new version and updates. Indeed being able to do this when m

Good work @stgraber and team and congratulations on continuously improving and constantly releasing new version and updates. Indeed being able to do this when majority of the container funding is going to Kubernetes or Docker is a great achievement. Has been using LXD for last 6 years in production. Its a little gem hiding with all the spotlight on Kubernetes and Docker. Indeed the way LXD works, it naturally provides a path from physical servers and VM instances to containers and back. The important part is it runs containers in userspace providing a much better security then docker, but seems developers don't care about it. Also you can use traditional ansible, puppet or chef to create and manage images and containers directly instead of learning shell scripts and Dockerfile way with additional cognitive load. The only issue I am facing is mounting a shared NFS or CIFS in userspace container using FUSE, since the client drivers for this fs needs to be run through a root process. Hopefully it will be resolved in future. Tried Lizardfs instead of NFS and the system failed in the middle and my issues on the github is there for weeks without any feedback. I will try glusterfs and also looking and beegfs lets see if it can work.

insertnickname 8y ago
Are you talking about LXC or LXD? I don't think LXD existed six years ago.
- simosx 8y ago
  The first commit for LXD on github is from November 2014: commit 3153b57369b7cf3e96979897c335a5c1d69e7f07 Author: Stéphane Graber Date: Wed Nov 5 10:09:28 2014 -0500 Add licensing and contributing guidelines Signed-off-by: Stéphane Graber <stgraber@ubuntu.com> Source: https://github.com/lxc/lxd https://github.com/lxc/lxd There is a migration path from LXC to LXD and probably the OP migrated at some point in time.
StreamBright 8y ago
Couldn’t agree more. I think the biggest problem for more adoption is the lack of blog posts and documentation. I wish somebody would write articles about how to use LXD/LXC in production catered towards developers. Maybe even having a side by side comparison with Docker.
- chrisper 8y ago
  I think the issue is also that for a long time (or still is?) it was Ubuntu only.
  - StreamBright 8y ago
    I don’t think so https://discuss.linuxcontainers.org/t/lxd-on-centos-7/1250 https://discuss.linuxcontainers.org/t/lxd-on-centos-7/1250
  - simosx 8y ago
    There were some issues with specific kernel features not being upstreamed timely. Having said that, here is a distribution usage for the snap package of LXD, https://snapcraft.io/lxd https://snapcraft.io/lxd (see that the end of the page).
    - stgraber 8y ago
      At this point, the only feature that I'm aware of which we have in the Ubuntu kernel and hasn't been merged upstream yet is support for overlayfs inside user namespaces. The other big one for a long time was fuse inside user namespaces, but that has been merged upstream in 4.18. Some of the AppArmor features were also a sticky point for a while as upstream was lagging behind quite a bit, but the AppArmor maintainer has since fixed that, so recent Linux kernels have everything that we use.
  - cyphar 8y ago
    LXC has worked on many distributions for a very long time, as has LXD. Most of the time, all of the main developers have been Canonical employees (with some exceptions), so this might've impacted people's view on what it supported. Also LXD's recommended distribution is through snaps (which gives it an Ubuntu-only flavour) -- but it's available in many different repositories in many different distributions.
- simosx 8y ago
  I have written some entry-level tutorials shown at https://discuss.linuxcontainers.org/t/the-lxd-tutorials-of-simos/1228 https://discuss.linuxcontainers.org/t/the-lxd-tutorials-of-s... I would like as well to see usage examples in the official documentation.
cyphar 8y ago
Don't get me wrong, I'm a huge fan of LXC and I've collaborated with the LXC folks with upstream kernel work (they're really clever guys). > The important part is it runs containers in userspace providing a much better security then docker, but seems developers don't care about it. I'm really not sure what you mean here by "runs containers in userspace". LXC and runc (the runtime underneath Docker) both use the same kernel primitives and generally work in fairly similar ways. LXC has more specific work in order to be able to have more VM-like containers (such as having "console" support and the "fun" that is booting systemd inside containers), but it's fundamentally not different to runc containers. In many cases, kernel work by one of us will benefit the other (right now we're working on several kernel patches in parallel, and each of us is considering how we can integrate the others' work into our projects). I do think LXC has better protections against certain attacks (and I worked with them on some of those protections) and has a much nicer design in many aspects. So you could argue it has some security upsides over Docker or runc, but that's a different topic to what primitives they use -- they're effectively indistinguishable at that level. (I'm one of the maintainers of runc.)
- hardwaresofton 8y ago
  I've also struggled with the distinction, but I believe the commenter above was referring to LXC (and LXD's) "system containers". Full user namespacing has been in from very early on (day 1?) which is the "increased security" -- this is where I believed the difference between "system containers" for LXC and regular "application containers" was. runc was not always the runtime underneath Docker, and IIRC it has not always had support for proper user namespacing for as long as LXC has. Also, LXC has so many more features packed in I think it's a disservice to compare it to runc in the first place. It should go without saying but I would absolutely love for you to sprinkle some knowledge on me wherever I'm wrong about either of these. [EDIT] - Getting downvotes so figured maybe I said something egregious so I did some digging: - LXC had user namespaces @ v1.0 in 2013[0] - Docker got user namespaces @ 1.10 in 2016[1] This is the distinction I meant. [0]: https://stgraber.org/2014/01/01/lxc-1-0-security-features/ https://stgraber.org/2014/01/01/lxc-1-0-security-features/ [1]: https://blog.docker.com/2016/02/docker-1-10/ https://blog.docker.com/2016/02/docker-1-10/
viraptor 8y ago
> Also you can use traditional ansible, puppet or chef to create and manage images and containers directly instead of learning shell scripts and Dockerfile way with additional cognitive load. You can certainly provision docker containers using ansible / chef / puppet. It's not popular, because they do way more than is required in those situations normally, but it doesn't stop you. A lot of larger applications on dockerhub are done that way - splunk's dockerfile is just an ansible starter for example. Puppet even has a page about it: https://puppet.com/blog/running-puppet-software-docker-containers https://puppet.com/blog/running-puppet-software-docker-conta...
- u801e 8y ago
  > You can certainly provision docker containers using ansible / chef / puppet. It's not popular, because they do way more than is required in those situations normally Could you elaborate on what you're referring to what they're doing beyond what's required? > A lot of larger applications on dockerhub are done that way From what I understand, you have to write the configuration management in a different way such that it doesn't require restarting services after configuration files have been updated. Converting an existing playbook or cookbook to work that way isn't trivial depending on how many things it's doing to provision the server.
  - viraptor 8y ago
    > what they're doing beyond what's required? Chef, puppet, ansible and other config management tools are designed to run nicely in an existing system. That means if you want to do action X, they'll usually check if it's possible, try to resolve internal dependencies, handle potential errors, register the state change for report and other things. Most of that is unnecessary when you're building a docker image from scratch. You control all the dependencies - they're the lines before this one. You don't care about nice reports of partial failures - they should cause immediate aborts. You don't care about managing state - each step is it's own fully defined state. For example, if I remember correctly, chef will query available package list when you request something installed. It will also continue running independent recipes if the installation fails. When building images I want literally "apt-get install foo || crash_and_burn". > you have to write the configuration management in a different way such that it doesn't require restarting services after configuration files have been updated In my experience with cfmgmt - this needs to be implemented whether you build images or update running systems. There always comes a time when you want to change something but not trigger service updates. When building images that time is always.
vikrantrathore 8y ago
Also I would like to point out that the lxd community is helpful, engaging and absolutely wonderful. I had an issue with client ip address in haproxy container on Google compute instance you can see on https://discuss.linuxcontainers.org/t/how-to-get-real-client-ip-when-using-lxd-to-forward-port-80/2079 https://discuss.linuxcontainers.org/t/how-to-get-real-client... An issue was created and to my surprise a fix landed in next release. It helped me to launch haproxy containers binding port 80/443 on compute instance and have correct client ip in haproxy logs. This helped me to have analytics from haproxy logs directly.
- simosx 8y ago
  Here is that feature request for the PROXY protocol, https://github.com/lxc/lxd/issues/4786 https://github.com/lxc/lxd/issues/4786 Was reported on July 14th and committed on July 19th.

insertnickname 8y ago

Are you talking about LXC or LXD? I don't think LXD existed six years ago.

simosx 8y ago
The first commit for LXD on github is from November 2014: commit 3153b57369b7cf3e96979897c335a5c1d69e7f07 Author: Stéphane Graber Date: Wed Nov 5 10:09:28 2014 -0500 Add licensing and contributing guidelines Signed-off-by: Stéphane Graber <stgraber@ubuntu.com> Source: https://github.com/lxc/lxd https://github.com/lxc/lxd There is a migration path from LXC to LXD and probably the OP migrated at some point in time.

StreamBright 8y ago

Couldn’t agree more. I think the biggest problem for more adoption is the lack of blog posts and documentation. I wish somebody would write articles about how to use LXD/LXC in production catered towards developers. Maybe even having a side by side comparison with Docker.

chrisper 8y ago
I think the issue is also that for a long time (or still is?) it was Ubuntu only.
- StreamBright 8y ago
  I don’t think so https://discuss.linuxcontainers.org/t/lxd-on-centos-7/1250 https://discuss.linuxcontainers.org/t/lxd-on-centos-7/1250
- simosx 8y ago
  There were some issues with specific kernel features not being upstreamed timely. Having said that, here is a distribution usage for the snap package of LXD, https://snapcraft.io/lxd https://snapcraft.io/lxd (see that the end of the page).
  - stgraber 8y ago
    At this point, the only feature that I'm aware of which we have in the Ubuntu kernel and hasn't been merged upstream yet is support for overlayfs inside user namespaces. The other big one for a long time was fuse inside user namespaces, but that has been merged upstream in 4.18. Some of the AppArmor features were also a sticky point for a while as upstream was lagging behind quite a bit, but the AppArmor maintainer has since fixed that, so recent Linux kernels have everything that we use.
- cyphar 8y ago
  LXC has worked on many distributions for a very long time, as has LXD. Most of the time, all of the main developers have been Canonical employees (with some exceptions), so this might've impacted people's view on what it supported. Also LXD's recommended distribution is through snaps (which gives it an Ubuntu-only flavour) -- but it's available in many different repositories in many different distributions.
simosx 8y ago
I have written some entry-level tutorials shown at https://discuss.linuxcontainers.org/t/the-lxd-tutorials-of-simos/1228 https://discuss.linuxcontainers.org/t/the-lxd-tutorials-of-s... I would like as well to see usage examples in the official documentation.

cyphar 8y ago

Don't get me wrong, I'm a huge fan of LXC and I've collaborated with the LXC folks with upstream kernel work (they're really clever guys). > The important part is it runs containers in userspace providing a much better security then docker, but seems developers don't care about it. I'm really not sure what you mean here by "runs containers in userspace". LXC and runc (the runtime underneath Docker) both use the same kernel primitives and generally work in fairly similar ways. LXC has more specific work in order to be able to have more VM-like containers (such as having "console" support and the "fun" that is booting systemd inside containers), but it's fundamentally not different to runc containers. In many cases, kernel work by one of us will benefit the other (right now we're working on several kernel patches in parallel, and each of us is considering how we can integrate the others' work into our projects). I do think LXC has better protections against certain attacks (and I worked with them on some of those protections) and has a much nicer design in many aspects. So you could argue it has some security upsides over Docker or runc, but that's a different topic to what primitives they use -- they're effectively indistinguishable at that level. (I'm one of the maintainers of runc.)

hardwaresofton 8y ago
I've also struggled with the distinction, but I believe the commenter above was referring to LXC (and LXD's) "system containers". Full user namespacing has been in from very early on (day 1?) which is the "increased security" -- this is where I believed the difference between "system containers" for LXC and regular "application containers" was. runc was not always the runtime underneath Docker, and IIRC it has not always had support for proper user namespacing for as long as LXC has. Also, LXC has so many more features packed in I think it's a disservice to compare it to runc in the first place. It should go without saying but I would absolutely love for you to sprinkle some knowledge on me wherever I'm wrong about either of these. [EDIT] - Getting downvotes so figured maybe I said something egregious so I did some digging: - LXC had user namespaces @ v1.0 in 2013[0] - Docker got user namespaces @ 1.10 in 2016[1] This is the distinction I meant. [0]: https://stgraber.org/2014/01/01/lxc-1-0-security-features/ https://stgraber.org/2014/01/01/lxc-1-0-security-features/ [1]: https://blog.docker.com/2016/02/docker-1-10/ https://blog.docker.com/2016/02/docker-1-10/

viraptor 8y ago

> Also you can use traditional ansible, puppet or chef to create and manage images and containers directly instead of learning shell scripts and Dockerfile way with additional cognitive load. You can certainly provision docker containers using ansible / chef / puppet. It's not popular, because they do way more than is required in those situations normally, but it doesn't stop you. A lot of larger applications on dockerhub are done that way - splunk's dockerfile is just an ansible starter for example. Puppet even has a page about it: https://puppet.com/blog/running-puppet-software-docker-containers https://puppet.com/blog/running-puppet-software-docker-conta...

u801e 8y ago
> You can certainly provision docker containers using ansible / chef / puppet. It's not popular, because they do way more than is required in those situations normally Could you elaborate on what you're referring to what they're doing beyond what's required? > A lot of larger applications on dockerhub are done that way From what I understand, you have to write the configuration management in a different way such that it doesn't require restarting services after configuration files have been updated. Converting an existing playbook or cookbook to work that way isn't trivial depending on how many things it's doing to provision the server.
- viraptor 8y ago
  > what they're doing beyond what's required? Chef, puppet, ansible and other config management tools are designed to run nicely in an existing system. That means if you want to do action X, they'll usually check if it's possible, try to resolve internal dependencies, handle potential errors, register the state change for report and other things. Most of that is unnecessary when you're building a docker image from scratch. You control all the dependencies - they're the lines before this one. You don't care about nice reports of partial failures - they should cause immediate aborts. You don't care about managing state - each step is it's own fully defined state. For example, if I remember correctly, chef will query available package list when you request something installed. It will also continue running independent recipes if the installation fails. When building images I want literally "apt-get install foo || crash_and_burn". > you have to write the configuration management in a different way such that it doesn't require restarting services after configuration files have been updated In my experience with cfmgmt - this needs to be implemented whether you build images or update running systems. There always comes a time when you want to change something but not trigger service updates. When building images that time is always.

vikrantrathore 8y ago

Also I would like to point out that the lxd community is helpful, engaging and absolutely wonderful. I had an issue with client ip address in haproxy container on Google compute instance you can see on https://discuss.linuxcontainers.org/t/how-to-get-real-client-ip-when-using-lxd-to-forward-port-80/2079 https://discuss.linuxcontainers.org/t/how-to-get-real-client... An issue was created and to my surprise a fix landed in next release. It helped me to launch haproxy containers binding port 80/443 on compute instance and have correct client ip in haproxy logs. This helped me to have analytics from haproxy logs directly.

simosx 8y ago
Here is that feature request for the PROXY protocol, https://github.com/lxc/lxd/issues/4786 https://github.com/lxc/lxd/issues/4786 Was reported on July 14th and committed on July 19th.