7 ms·

LXC Networking introduction

136 points by gyre007 13y ago

ambiate 13y ago
We are currently virtualizing our infrastructure's desktops with QVD and LXC. There have been a few challenges, but overall, it has been painless and exciting.
hardwaresofton 13y ago
Fantastic article, super in-depth, and well written. One of the most in-depth guides to anything lxc related I've ever seen. Thanks for demystifying a little bit of lxc
droopybuns 13y ago
Thanks for sharing this link. LXC has gotten me very excited about testing new linux services on a home box again. I always worry about exposing a server to the internet with any new services. The idea that a compromise on the box could leak everything that's on the box usually leads me to avoid exploring new services. LXCs seem to give me hope that I could experience a compromise, but not lose everything.
- nisa 13y ago
  LXC is not secure at the moment. root in a LXC container can lead to root on the host. There is unfortunately no good summary of the problems - here is my list (take it with a grain of salt - a lot of these problems are mitigated in docker.io and with AppArmor in Ubuntu): - Without CONFIG_USER_NS and a newer kernel a lot of problematic things can happen. If /proc or /sysfs is mounted on the container DoS or escalation to root is possible: http://blog.bofh.it/debian/id_413 http://blog.bofh.it/debian/id_413 - At the moment no stock distro kernel has CONFIG_USER_NS enabled. - There are some issues related to remounting filesystems rw and altering files - Mounting cgroups in the container can also lead to problems - DoS and aquiring more ressources - Capabilities. You stock Linux distribution won't boot without CAP_SYS_ADMIN (see man 7 capabilities) - there are a lot of other capabilities that could be troublesome. - Not sure about this one: http://seclists.org/oss-sec/2011/q3/385 http://seclists.org/oss-sec/2011/q3/385 So for running services without CAP_SYS_ADMIN and with dropping a lot of other capabilities it can be considered somewhat safe. For everything else it's probably dangerous. Not sure if all these issues are still a problem today but if you are running lxc on e.g. a current Debian Wheezy you have to know about all of them.
  - evol262 13y ago
    Exactly what security features do you think Docker is providing that LXC isn't?
    - nisa 13y ago
      LXC provides all the features. Except some protection that can at the moment only be provided by AppArmor or another MAC system. I'm not sure what are the LXC defaults are - But I assume that docker uses more restrictive defaults: https://github.com/dotcloud/docker/blob/master/lxc_template.go https://github.com/dotcloud/docker/blob/master/lxc_template.... I'm still learning about LXC so my post regarding security may be inaccurate. I'd just thought I'd share it because a lot of people think it's as secure as a virtual machine. I hope soon it is.
      - evol262 13y ago
        I find the number of people schilling for Docker mindblowing. "I don't know much about LXC, and I don't know what the defaults are, but I assume that this trivial wrapper around LXC provides more security".
        nisa 13y ago
        Sorry this was not my intention. I actually prefer LXC to docker myself and did not wanted to shill for anything. I just wanted to point out that of the (possible) security problems that can happen when using LXC docker mitigates most by just not using them: They don't allow mounts, they drop CAP_SYS_ADMIN. I just posted the config file. It was just a well intended warning - similiar to the warnings in the Ubuntu docs: https://help.ubuntu.com/12.04/serverguide/lxc.html#lxc-security https://help.ubuntu.com/12.04/serverguide/lxc.html#lxc-secur... and Gentoo: https://wiki.gentoo.org/wiki/LXC#MAJOR_Temporary_Problems_with_LXC_-_READ_THIS https://wiki.gentoo.org/wiki/LXC#MAJOR_Temporary_Problems_wi... Here is the default configuration for Ubuntu 13.10 in comparision: https://gist.github.com/anonymous/7550932 https://gist.github.com/anonymous/7550932
  - justincormack 13y ago
- j_s 13y ago
  A look at docker/LXC security back at the end of August 2013: http://blog.docker.io/2013/08/containers-docker-how-secure-are-they/ http://blog.docker.io/2013/08/containers-docker-how-secure-a...
rdtsc 13y ago
LXC is interesting. It has been around probably for 5 or 7 years or so. At least I remember looking at it then. Then it just kind of made slow progress. It has been in the kernel since probably since 2.6.30, for example. But aside from kernel and virtualization forum discussions, it just didn't get much attention. That was kind of odd because it is a very cool piece of technology. Then I guess something happened in the last year or so and all of the sudden it became very popular. dotCloud certainly has a lot to do with it. Was it better libvirt integration, too, perhaps: http://libvirt.org/drvlxc.html http://libvirt.org/drvlxc.html Now of course it cannot completely replace KVM because it is more of a container than a full virtualization. So running Windows VMs on will not work. For Linux one could probably have a farm of hosts based on various distros (with LXC enabled) and that would provide the ability to run various Linux OS guests, by picking the hosts that matches it.
- thelambentonion 13y ago
  Maybe I have an incomplete understanding of Docker and LXC, but I feel like this is retreading a lot of ground that Joyent has covered with SmartOS. Virtualization with KVM, networking with Crossbow, and VM exporting with ZFS datasets have all been around for the past few years without getting all of this attention.
  - maccam94 13y ago
    KVM has more performance overhead than LXC. Sure, SmartOS has "zones" which are a similar concept, but then you're stuck with the SmartOS (Solaris) userspace. Plus, with LXC the physical host is running Linux, so hardware compatibility is less of a concern.
    - thelambentonion 13y ago
      > KVM has more performance overhead than LXC When I was using it, KVM had some small memory and CPU overhead, but it's a fair point. > ...so hardware compatibility is less of a concern. That's definitely a problem. I'd very much like for SmartOS to branch out from just Intel, but they seem to be keeping development centered around the hardware they use for Joyent Cloud.
zobzu 13y ago
10 pages to make one part of LXC understandable, is one of LXC's issues ;-)
- gesman 13y ago
  ...plus now the enterprise needs to hire more expensive sysadmins to babysit all of it. That's exactly what enterprises are wanting to avoid. Our sysadmin left after he deployed LXC "goodness" to make things "better" and we are still in a recovery mode from this.
  - zobzu 13y ago
    thats an interesting point, but yes, LXC isn't all that simple/easy, it's the main issue. I don't know why they made it so complicated. The namespacing technology underneath is pretty straight forward. I'm sure they'd argue for days how that's not true (plus some of the LXC folks actually implemented the namespacing) - but at the end of the day I make my "jail" with the "unshare" command and mount, much simpler..
willvarfar 13y ago
Are there nice GUI/web tools for configuring docker and LXC firewalling etc?
binocarlos 13y ago
Great article! For me Docker has opened up what is great about the underlying LXC technology - a proper logical wall between apps on the same hardware. The networking was the hardest thing (for me) to grasp in Docker - this guide opens up a whole new level. It's cool however that if Docker won't do something networky I can drill down a layer to LXC instead.
deleted 13y ago
[deleted]
ezequiel-garzon 13y ago
Somewhat related, are there any cookbooks on how to "transform a heavy-duty server into several virtual machines"? By this I mean something that includes, among other things, mapping N available external IP addresses to N virtual machines. I realize a good prior understanding of networking, iptables, etc. would make sense. But... still... any well-detailed recipes out there? (I have tried to look them up, but everything seems to assume more knowledge than what I have in mind!)
seldo 13y ago
A minor correction -- dotCloud the PaaS does not actually use Docker in production, as the article claims. It uses something pretty similar, which was the inspiration for the creation of Docker, but not Docker itself. [Source: @solomonstre, in person at a Docker meetup a few weeks ago]
- shykes 13y ago
  That's correct (I'm the @solomonstre in question :). We do our best to not imply the contrary by accident. Docker is a clean slate which incorporates all our operational learnings from dotcloud - but it is a full rewrite and this not yet production ready.
emmelaich 13y ago
A bit of a tangent, but I wonder what happened to lguest .. and how does (did) it compare to lxc? Looked promising, but I never had success getting it going despite the promises on the webpage. And it looks like little activity and still quite a low profile after all these years. http://lguest.ozlabs.org/ http://en.wikipedia.org/wiki/Lguest
patrickg_zill 13y ago
I like LXC but for now all my virtualized stuff runs on KVM (if needed, like an old CentOS install with Oracle), or OpenVZ. I haven't sat down yet and compared all the features of LXC + Docker with OpenVZ yet, though.

136 points by gyre007 13y ago

We are currently virtualizing our infrastructure's desktops with QVD and LXC. There have been a few challenges, but overall, it has been painless and exciting.

hardwaresofton 13y ago

Fantastic article, super in-depth, and well written. One of the most in-depth guides to anything lxc related I've ever seen. Thanks for demystifying a little bit of lxc

droopybuns 13y ago

Thanks for sharing this link. LXC has gotten me very excited about testing new linux services on a home box again. I always worry about exposing a server to the internet with any new services. The idea that a compromise on the box could leak everything that's on the box usually leads me to avoid exploring new services. LXCs seem to give me hope that I could experience a compromise, but not lose everything.

nisa 13y ago
LXC is not secure at the moment. root in a LXC container can lead to root on the host. There is unfortunately no good summary of the problems - here is my list (take it with a grain of salt - a lot of these problems are mitigated in docker.io and with AppArmor in Ubuntu): - Without CONFIG_USER_NS and a newer kernel a lot of problematic things can happen. If /proc or /sysfs is mounted on the container DoS or escalation to root is possible: http://blog.bofh.it/debian/id_413 http://blog.bofh.it/debian/id_413 - At the moment no stock distro kernel has CONFIG_USER_NS enabled. - There are some issues related to remounting filesystems rw and altering files - Mounting cgroups in the container can also lead to problems - DoS and aquiring more ressources - Capabilities. You stock Linux distribution won't boot without CAP_SYS_ADMIN (see man 7 capabilities) - there are a lot of other capabilities that could be troublesome. - Not sure about this one: http://seclists.org/oss-sec/2011/q3/385 http://seclists.org/oss-sec/2011/q3/385 So for running services without CAP_SYS_ADMIN and with dropping a lot of other capabilities it can be considered somewhat safe. For everything else it's probably dangerous. Not sure if all these issues are still a problem today but if you are running lxc on e.g. a current Debian Wheezy you have to know about all of them.
- evol262 13y ago
  Exactly what security features do you think Docker is providing that LXC isn't?
  - nisa 13y ago
    LXC provides all the features. Except some protection that can at the moment only be provided by AppArmor or another MAC system. I'm not sure what are the LXC defaults are - But I assume that docker uses more restrictive defaults: https://github.com/dotcloud/docker/blob/master/lxc_template.go https://github.com/dotcloud/docker/blob/master/lxc_template.... I'm still learning about LXC so my post regarding security may be inaccurate. I'd just thought I'd share it because a lot of people think it's as secure as a virtual machine. I hope soon it is.
    - evol262 13y ago
      I find the number of people schilling for Docker mindblowing. "I don't know much about LXC, and I don't know what the defaults are, but I assume that this trivial wrapper around LXC provides more security".
      - nisa 13y ago
        Sorry this was not my intention. I actually prefer LXC to docker myself and did not wanted to shill for anything. I just wanted to point out that of the (possible) security problems that can happen when using LXC docker mitigates most by just not using them: They don't allow mounts, they drop CAP_SYS_ADMIN. I just posted the config file. It was just a well intended warning - similiar to the warnings in the Ubuntu docs: https://help.ubuntu.com/12.04/serverguide/lxc.html#lxc-security https://help.ubuntu.com/12.04/serverguide/lxc.html#lxc-secur... and Gentoo: https://wiki.gentoo.org/wiki/LXC#MAJOR_Temporary_Problems_with_LXC_-_READ_THIS https://wiki.gentoo.org/wiki/LXC#MAJOR_Temporary_Problems_wi... Here is the default configuration for Ubuntu 13.10 in comparision: https://gist.github.com/anonymous/7550932 https://gist.github.com/anonymous/7550932
- justincormack 13y ago
j_s 13y ago
A look at docker/LXC security back at the end of August 2013: http://blog.docker.io/2013/08/containers-docker-how-secure-are-they/ http://blog.docker.io/2013/08/containers-docker-how-secure-a...

rdtsc 13y ago

LXC is interesting. It has been around probably for 5 or 7 years or so. At least I remember looking at it then. Then it just kind of made slow progress. It has been in the kernel since probably since 2.6.30, for example. But aside from kernel and virtualization forum discussions, it just didn't get much attention. That was kind of odd because it is a very cool piece of technology. Then I guess something happened in the last year or so and all of the sudden it became very popular. dotCloud certainly has a lot to do with it. Was it better libvirt integration, too, perhaps: http://libvirt.org/drvlxc.html http://libvirt.org/drvlxc.html Now of course it cannot completely replace KVM because it is more of a container than a full virtualization. So running Windows VMs on will not work. For Linux one could probably have a farm of hosts based on various distros (with LXC enabled) and that would provide the ability to run various Linux OS guests, by picking the hosts that matches it.

thelambentonion 13y ago
Maybe I have an incomplete understanding of Docker and LXC, but I feel like this is retreading a lot of ground that Joyent has covered with SmartOS. Virtualization with KVM, networking with Crossbow, and VM exporting with ZFS datasets have all been around for the past few years without getting all of this attention.
- maccam94 13y ago
  KVM has more performance overhead than LXC. Sure, SmartOS has "zones" which are a similar concept, but then you're stuck with the SmartOS (Solaris) userspace. Plus, with LXC the physical host is running Linux, so hardware compatibility is less of a concern.
  - thelambentonion 13y ago
    > KVM has more performance overhead than LXC When I was using it, KVM had some small memory and CPU overhead, but it's a fair point. > ...so hardware compatibility is less of a concern. That's definitely a problem. I'd very much like for SmartOS to branch out from just Intel, but they seem to be keeping development centered around the hardware they use for Joyent Cloud.

zobzu 13y ago

10 pages to make one part of LXC understandable, is one of LXC's issues ;-)

gesman 13y ago
...plus now the enterprise needs to hire more expensive sysadmins to babysit all of it. That's exactly what enterprises are wanting to avoid. Our sysadmin left after he deployed LXC "goodness" to make things "better" and we are still in a recovery mode from this.
- zobzu 13y ago
  thats an interesting point, but yes, LXC isn't all that simple/easy, it's the main issue. I don't know why they made it so complicated. The namespacing technology underneath is pretty straight forward. I'm sure they'd argue for days how that's not true (plus some of the LXC folks actually implemented the namespacing) - but at the end of the day I make my "jail" with the "unshare" command and mount, much simpler..

willvarfar 13y ago

Are there nice GUI/web tools for configuring docker and LXC firewalling etc?

binocarlos 13y ago

Great article! For me Docker has opened up what is great about the underlying LXC technology - a proper logical wall between apps on the same hardware. The networking was the hardest thing (for me) to grasp in Docker - this guide opens up a whole new level. It's cool however that if Docker won't do something networky I can drill down a layer to LXC instead.

deleted 13y ago

[deleted]

ezequiel-garzon 13y ago

Somewhat related, are there any cookbooks on how to "transform a heavy-duty server into several virtual machines"? By this I mean something that includes, among other things, mapping N available external IP addresses to N virtual machines. I realize a good prior understanding of networking, iptables, etc. would make sense. But... still... any well-detailed recipes out there? (I have tried to look them up, but everything seems to assume more knowledge than what I have in mind!)

seldo 13y ago

A minor correction -- dotCloud the PaaS does not actually use Docker in production, as the article claims. It uses something pretty similar, which was the inspiration for the creation of Docker, but not Docker itself. [Source: @solomonstre, in person at a Docker meetup a few weeks ago]

shykes 13y ago
That's correct (I'm the @solomonstre in question :). We do our best to not imply the contrary by accident. Docker is a clean slate which incorporates all our operational learnings from dotcloud - but it is a full rewrite and this not yet production ready.

emmelaich 13y ago

A bit of a tangent, but I wonder what happened to lguest .. and how does (did) it compare to lxc? Looked promising, but I never had success getting it going despite the promises on the webpage. And it looks like little activity and still quite a low profile after all these years. http://lguest.ozlabs.org/ http://en.wikipedia.org/wiki/Lguest

patrickg_zill 13y ago

I like LXC but for now all my virtualized stuff runs on KVM (if needed, like an old CentOS install with Oracle), or OpenVZ. I haven't sat down yet and compared all the features of LXC + Docker with OpenVZ yet, though.