6 ms·

Announcing Docker 1.7: Multi-host networking, plugins and orchestration updates

79 points by eloycoto 11y ago

hmottestad 11y ago
Awesome awesomeness. I read a blog post here about vxlan adding some 10% overhead in terms of both latency and bandwith, which seems very acceptable to me :) CoreOS has had this for a while in flannel (ref the same article as above), so it's not like it's a completely new thing, but if you prefer going with the crowd, then having overlay networking built right into docker will make for an awesome experience when using docker compose and docker swarm. Edit: Article: http://www.generictestdomain.net/docker/weave/networking/stupidity/2015/04/05/weave-is-kinda-slow/ http://www.generictestdomain.net/docker/weave/networking/stu... And the results: Flannel VXLan 96.55% bandwidth utilisation 140.52% latency. Without overlay latency was 91.8µs, and with vxlan it was 129µs.
- Nrsolis 11y ago
  I can't say I'm particularly happy with the choice of VxLAN as a tunneling technology but I'm not particularly surprised either.
  - Nrsolis 11y ago
    THIS is what I'm talking about: http://cloudarchitectmusings.com/2013/01/03/word-of-caution-about-overextending-the-use-of-vxlan/ http://cloudarchitectmusings.com/2013/01/03/word-of-caution-... VxLAN is a technology with a very limited horizon in terms of functionality. If you're intending to really leverage containers in a super dense use case, then you'd be well advised to adopt a technology that has ready made ASIC support for cross the physical-to-virtual network boundary.
    - wmf 11y ago
      That article says VXLAN isn't good for DCI, but DCI isn't a good idea anyway[1] and most people aren't trying to use Docker that way. And VXLAN is pretty much the only encapsulation format that has good ASIC support in both NICs and switches. [1] http://blog.ipspace.net/2014/10/vxlan-and-otv-saga-continues.html http://blog.ipspace.net/2014/10/vxlan-and-otv-saga-continues... http://blog.ipspace.net/2013/09/sooner-or-later-someone-will-pay-for.html http://blog.ipspace.net/2013/09/sooner-or-later-someone-will... http://blog.ipspace.net/2015/01/latency-killer-of-spread-out.html http://blog.ipspace.net/2015/01/latency-killer-of-spread-out...
      - Nrsolis 11y ago
        Uh, no. MPLS has had SOLID ASIC support for maybe two decades now. Everyone who is putting in VxLAN-capable hardware is doing so because they have oodles of ESXi deployed and VxLAN is the only technology that's supported by VMware. If you're not wedded to the ESXi hypervisor, you can deploy either MPLS or MPLSoverGREoverIP (for your non-MPLS capable endpoints) and capture a whole stack of goodness to include interworking across an entire WAN infrastructure for DC-to-DC VM movement and EVPN support. MPLS has been doing L2VPN support for ages. Man, NSX/VxLAN didn't even have a real control plane until recently. They were going to use MULTICAST for their table updates! MULTICAST!
        jpgvm 11y ago
        So much FUD. First of all VxLAN is already nicely offloaded by the majority of NICs and if not being terminated doesn't required hardware support in switching equipment. VxLAN is almost no lock-in. Currently to do MPLS in any shape or form you are going to be locked into proprietary network gear. MPLS has almost no software implementations, whereas Linux has native VxLAN support in vanilla bridging mode and OVS. Forget MPLS on Windows. MPLS is great tech don't get me wrong, however it's restricted to the carrier space because it doesn't play well with others.
        Nrsolis 11y ago
        You're thinking of a very narrow use-case for VxLAN. What happens when your VM needs to talk to legacy networking gear? What happens when your VM(s) need to talk to a legacy security infrastructure that's mandated because of regulatory concerns? What happens when you need to route between two VxLAN domains? You're going to need to provide the VxLAN VTEP functionality someplace and that requires hardware support in the network someplace. Waving your hands and saying "that's not a concern" won't cut it. I have a customer that's facing this issue RIGHT NOW. And no, MPLS isn't a proprietary standard. It's open and it's available on oodles of networking hardware. It's interoperable and it's a proven technology that scales. Developing a tunneling protocol is the easy part. Developing a scalable control plane to handle RIB and FIB state is another matter entirely. Getting all of that to operate on a chipset that can do 2-4Tbps per slot is another thing altogether. Go check out some of the HUGE cloud infrastructure guys. They aren't using BrandX networking hardware to build their data-centers. They're using the Big-3 players.
andyl 11y ago
I can't keep up with the churn. Will come back to Docker in a year or two to see if things have stabilized.
- shykes 11y ago
  FWIW we add a lot of new features, but we break very few APIs. API stability is extremely important to us, so you can actually build stable things on Docker and trust that we won't break it down the road.
  - jpallen 11y ago
    I've had APIs subtly break on me between at least 2 releases (I think it's actually 3, but one time may have been user error). The main culprits were changing the parameters which are passed on container creation vs on container start, and changing error messages/response codes.
    - cpuguy83 11y ago
      Yes, we have changed response codes when the old ones were just wrong. Like returning a 500 when really it should be a 404, that kind of stuff. As far as start/create params, these should not be changed at all, other than that you can pass the options that you could one only pass on start to create. If something has broken here, it was most likely a mistake and should be reported on GH. Even if we change the underlying configuration, we do version the API and make sure we are sending/receiving the same structs over the wire. TLDR; if something broke in the API, please report.
  - cshimmin 11y ago
    If I understood parent's comment (or maybe I just interpreted to agree with my own point of view), the implication is that some people are punting on even getting started w/ Docker, because things are moving too quickly. You may not break API's, but if the best-practices I learn today are completely obsolete by this time next year, that's still a bummer.
- toomuchtodo 11y ago
general_failure 11y ago
Still no user namespace support? I don't get how one can use for production websites without this. Especially if your run arbitrary containers from the docker registry. Or this not the suggested model anymore?
- lk4d4 11y ago
  There is support in runc https://github.com/opencontainers/runc.git https://github.com/opencontainers/runc.git It is still unsupported in docker though
- deleted 11y ago
  [deleted]
  - general_failure 11y ago
    user namespace is not the same as naming your containers. http://man7.org/linux/man-pages/man7/user_namespaces.7.html http://man7.org/linux/man-pages/man7/user_namespaces.7.html
    - jgrowl 11y ago
      Ahh, gotcha. I misunderstood.
  - deleted 11y ago
    [deleted]
- deleted 11y ago
  [deleted]
- davexunit 11y ago
  Do any other container projects support user namespaces? I'm curious about what the implementation would look like. Several things change when a user namespace is added to the mix, such as the container not being able to create new device nodes. Would it be possible for unprivileged users to create containers that had network access or is a daemon running as root still necessary?
jpallen 11y ago
The volume plugins look very interesting for our use case, but I can't find any documentation about how to actually write a plugin? Do I have to reverse engineer it from the Flocker example provided? Edit: Nevermind, I found it (https://github.com/docker/docker/blob/master/experimental/plugin_api.md https://github.com/docker/docker/blob/master/experimental/pl...)! Three clicks in, through various blog articles with lots of other links. Maybe consider making this more obvious?
bkeroack 11y ago
Rather than run all network traffic through their single daemon, my feeling is that it's better to abstract your containers into pods and discoverable services like Kubernetes. Increasingly my feeling regarding Docker, Inc is "too little, too late". They seem to be chasing every market (enterprise, startup, developer) and therefore are mediocre at a lot of things while expert at none.
- shykes 11y ago
  Docker's new networking model is exactly what you describe. In fact it was developed with a lot of feedback from the Google team, so if you like Kubernete's networking model you'll be in known territory. It's true that the Docker daemon by default bundles these functionalities by default. But under the hood, the networking system is actually a separate binary called "dnet", so it's going to be very easy to rip it out and make the daemon less monolithic.
falcolas 11y ago
I'm sorry, I have to be a bit negative on this one. - No release notes! (as of 1 hour after story posted) - Network stack re-written. They kinda flubbed network the first time, creating assymetric network and double nat conditions. And within one release, there's a complete re-write with a new networking model? This is going to take a lot of proving to ensure that it's actually going to work for production systems. In the meantime, we at least are going to continue using --net=host for our containers. - Another storage format. Can we simply get one that's stable, please? - Volume re-write. Ditto networking. - [EDIT2] Disregard previous edit entirely - now you're simply disallowed from using devicemapper entirely, if you're using the officially compiled Docker binary for Ubuntu 14.04 and 15.04? [4] What the actual fuck is going on here Docker? AUFS has significant performance issues (not even mentioning the deprecated part), OverlayFS requires a release candidate Linux kernel, ZFS as a storage backend is brand spanking new, and btrfs is as stable as a three legged chair. By the way, this affects CentOS and RHEL. [EDIT]: 1.7 appears to have fixed the superficial problem - using the wrong devicemapper drivers. Would still prefer to have a proper package. [ORIGINAL] The whole "devicemapper on Ubuntu 14.4" [1] snafu appears to still in full force [2]. Why can't they offer properly compiled OS packages? They're distributing them as OS/Architecture specific packages... Note - AUFS is not a real choice for (at least) node applications, we ran into an issue back on 1.6 where there was a low level mutex limiting concurrency on AUFS which did not appear on devicemapper. [3] - [1] https://github.com/docker/docker/issues/4036 https://github.com/docker/docker/issues/4036 [2] https://github.com/Capgemini/Apollo/issues/315 https://github.com/Capgemini/Apollo/issues/315 [3] https://github.com/docker/docker/issues/13268 https://github.com/docker/docker/issues/13268 [4] https://github.com/docker/docker/issues/14035 https://github.com/docker/docker/issues/14035 (look down for "vbatts" comment with the bolded summary header)
- zupancik 11y ago
  Bit new to Docker - so does this mean Docker 1.7 won't work with RHEL? Specifically I'm working on RHEL 6.6 machines, and I'm already stuck with Docker 1.5 so curious to know whether this will affect me
  - falcolas 11y ago
shred45 11y ago
They've also RUINED their docs pages. The typography hurts my eyes and the link layout makes it a pain to find anything specific. Takes me 10 times as long to find anything. :(
- falcolas 11y ago
  Yeah. I'm actually missing their previous documentation, which was much more readable and had some links on the left hand side down to anchors within a page. Now it's a bit of an inconsistent mess to find something.
- geku 11y ago
  Same here, I also find the new navigation a bit unconventional for such a large page structure.

79 points by eloycoto 11y ago

Awesome awesomeness. I read a blog post here about vxlan adding some 10% overhead in terms of both latency and bandwith, which seems very acceptable to me :) CoreOS has had this for a while in flannel (ref the same article as above), so it's not like it's a completely new thing, but if you prefer going with the crowd, then having overlay networking built right into docker will make for an awesome experience when using docker compose and docker swarm. Edit: Article: http://www.generictestdomain.net/docker/weave/networking/stupidity/2015/04/05/weave-is-kinda-slow/ http://www.generictestdomain.net/docker/weave/networking/stu... And the results: Flannel VXLan 96.55% bandwidth utilisation 140.52% latency. Without overlay latency was 91.8µs, and with vxlan it was 129µs.

Nrsolis 11y ago
I can't say I'm particularly happy with the choice of VxLAN as a tunneling technology but I'm not particularly surprised either.
- Nrsolis 11y ago
  THIS is what I'm talking about: http://cloudarchitectmusings.com/2013/01/03/word-of-caution-about-overextending-the-use-of-vxlan/ http://cloudarchitectmusings.com/2013/01/03/word-of-caution-... VxLAN is a technology with a very limited horizon in terms of functionality. If you're intending to really leverage containers in a super dense use case, then you'd be well advised to adopt a technology that has ready made ASIC support for cross the physical-to-virtual network boundary.
  - wmf 11y ago
    That article says VXLAN isn't good for DCI, but DCI isn't a good idea anyway[1] and most people aren't trying to use Docker that way. And VXLAN is pretty much the only encapsulation format that has good ASIC support in both NICs and switches. [1] http://blog.ipspace.net/2014/10/vxlan-and-otv-saga-continues.html http://blog.ipspace.net/2014/10/vxlan-and-otv-saga-continues... http://blog.ipspace.net/2013/09/sooner-or-later-someone-will-pay-for.html http://blog.ipspace.net/2013/09/sooner-or-later-someone-will... http://blog.ipspace.net/2015/01/latency-killer-of-spread-out.html http://blog.ipspace.net/2015/01/latency-killer-of-spread-out...
    - Nrsolis 11y ago
      Uh, no. MPLS has had SOLID ASIC support for maybe two decades now. Everyone who is putting in VxLAN-capable hardware is doing so because they have oodles of ESXi deployed and VxLAN is the only technology that's supported by VMware. If you're not wedded to the ESXi hypervisor, you can deploy either MPLS or MPLSoverGREoverIP (for your non-MPLS capable endpoints) and capture a whole stack of goodness to include interworking across an entire WAN infrastructure for DC-to-DC VM movement and EVPN support. MPLS has been doing L2VPN support for ages. Man, NSX/VxLAN didn't even have a real control plane until recently. They were going to use MULTICAST for their table updates! MULTICAST!
      - jpgvm 11y ago
        So much FUD. First of all VxLAN is already nicely offloaded by the majority of NICs and if not being terminated doesn't required hardware support in switching equipment. VxLAN is almost no lock-in. Currently to do MPLS in any shape or form you are going to be locked into proprietary network gear. MPLS has almost no software implementations, whereas Linux has native VxLAN support in vanilla bridging mode and OVS. Forget MPLS on Windows. MPLS is great tech don't get me wrong, however it's restricted to the carrier space because it doesn't play well with others.
        Nrsolis 11y ago
        You're thinking of a very narrow use-case for VxLAN. What happens when your VM needs to talk to legacy networking gear? What happens when your VM(s) need to talk to a legacy security infrastructure that's mandated because of regulatory concerns? What happens when you need to route between two VxLAN domains? You're going to need to provide the VxLAN VTEP functionality someplace and that requires hardware support in the network someplace. Waving your hands and saying "that's not a concern" won't cut it. I have a customer that's facing this issue RIGHT NOW. And no, MPLS isn't a proprietary standard. It's open and it's available on oodles of networking hardware. It's interoperable and it's a proven technology that scales. Developing a tunneling protocol is the easy part. Developing a scalable control plane to handle RIB and FIB state is another matter entirely. Getting all of that to operate on a chipset that can do 2-4Tbps per slot is another thing altogether. Go check out some of the HUGE cloud infrastructure guys. They aren't using BrandX networking hardware to build their data-centers. They're using the Big-3 players.

andyl 11y ago

I can't keep up with the churn. Will come back to Docker in a year or two to see if things have stabilized.

shykes 11y ago
FWIW we add a lot of new features, but we break very few APIs. API stability is extremely important to us, so you can actually build stable things on Docker and trust that we won't break it down the road.
- jpallen 11y ago
  I've had APIs subtly break on me between at least 2 releases (I think it's actually 3, but one time may have been user error). The main culprits were changing the parameters which are passed on container creation vs on container start, and changing error messages/response codes.
  - cpuguy83 11y ago
    Yes, we have changed response codes when the old ones were just wrong. Like returning a 500 when really it should be a 404, that kind of stuff. As far as start/create params, these should not be changed at all, other than that you can pass the options that you could one only pass on start to create. If something has broken here, it was most likely a mistake and should be reported on GH. Even if we change the underlying configuration, we do version the API and make sure we are sending/receiving the same structs over the wire. TLDR; if something broke in the API, please report.
- cshimmin 11y ago
  If I understood parent's comment (or maybe I just interpreted to agree with my own point of view), the implication is that some people are punting on even getting started w/ Docker, because things are moving too quickly. You may not break API's, but if the best-practices I learn today are completely obsolete by this time next year, that's still a bummer.
toomuchtodo 11y ago

general_failure 11y ago

Still no user namespace support? I don't get how one can use for production websites without this. Especially if your run arbitrary containers from the docker registry. Or this not the suggested model anymore?

lk4d4 11y ago
There is support in runc https://github.com/opencontainers/runc.git https://github.com/opencontainers/runc.git It is still unsupported in docker though
deleted 11y ago
[deleted]
- general_failure 11y ago
  user namespace is not the same as naming your containers. http://man7.org/linux/man-pages/man7/user_namespaces.7.html http://man7.org/linux/man-pages/man7/user_namespaces.7.html
  - jgrowl 11y ago
    Ahh, gotcha. I misunderstood.
- deleted 11y ago
  [deleted]
deleted 11y ago
[deleted]
davexunit 11y ago
Do any other container projects support user namespaces? I'm curious about what the implementation would look like. Several things change when a user namespace is added to the mix, such as the container not being able to create new device nodes. Would it be possible for unprivileged users to create containers that had network access or is a daemon running as root still necessary?

jpallen 11y ago

The volume plugins look very interesting for our use case, but I can't find any documentation about how to actually write a plugin? Do I have to reverse engineer it from the Flocker example provided? Edit: Nevermind, I found it (https://github.com/docker/docker/blob/master/experimental/plugin_api.md https://github.com/docker/docker/blob/master/experimental/pl...)! Three clicks in, through various blog articles with lots of other links. Maybe consider making this more obvious?

bkeroack 11y ago

Rather than run all network traffic through their single daemon, my feeling is that it's better to abstract your containers into pods and discoverable services like Kubernetes. Increasingly my feeling regarding Docker, Inc is "too little, too late". They seem to be chasing every market (enterprise, startup, developer) and therefore are mediocre at a lot of things while expert at none.

shykes 11y ago
Docker's new networking model is exactly what you describe. In fact it was developed with a lot of feedback from the Google team, so if you like Kubernete's networking model you'll be in known territory. It's true that the Docker daemon by default bundles these functionalities by default. But under the hood, the networking system is actually a separate binary called "dnet", so it's going to be very easy to rip it out and make the daemon less monolithic.

falcolas 11y ago

I'm sorry, I have to be a bit negative on this one. - No release notes! (as of 1 hour after story posted) - Network stack re-written. They kinda flubbed network the first time, creating assymetric network and double nat conditions. And within one release, there's a complete re-write with a new networking model? This is going to take a lot of proving to ensure that it's actually going to work for production systems. In the meantime, we at least are going to continue using --net=host for our containers. - Another storage format. Can we simply get one that's stable, please? - Volume re-write. Ditto networking. - [EDIT2] Disregard previous edit entirely - now you're simply disallowed from using devicemapper entirely, if you're using the officially compiled Docker binary for Ubuntu 14.04 and 15.04? [4] What the actual fuck is going on here Docker? AUFS has significant performance issues (not even mentioning the deprecated part), OverlayFS requires a release candidate Linux kernel, ZFS as a storage backend is brand spanking new, and btrfs is as stable as a three legged chair. By the way, this affects CentOS and RHEL. [EDIT]: 1.7 appears to have fixed the superficial problem - using the wrong devicemapper drivers. Would still prefer to have a proper package. [ORIGINAL] The whole "devicemapper on Ubuntu 14.4" [1] snafu appears to still in full force [2]. Why can't they offer properly compiled OS packages? They're distributing them as OS/Architecture specific packages... Note - AUFS is not a real choice for (at least) node applications, we ran into an issue back on 1.6 where there was a low level mutex limiting concurrency on AUFS which did not appear on devicemapper. [3] - [1] https://github.com/docker/docker/issues/4036 https://github.com/docker/docker/issues/4036 [2] https://github.com/Capgemini/Apollo/issues/315 https://github.com/Capgemini/Apollo/issues/315 [3] https://github.com/docker/docker/issues/13268 https://github.com/docker/docker/issues/13268 [4] https://github.com/docker/docker/issues/14035 https://github.com/docker/docker/issues/14035 (look down for "vbatts" comment with the bolded summary header)

zupancik 11y ago
Bit new to Docker - so does this mean Docker 1.7 won't work with RHEL? Specifically I'm working on RHEL 6.6 machines, and I'm already stuck with Docker 1.5 so curious to know whether this will affect me
- falcolas 11y ago

shred45 11y ago

They've also RUINED their docs pages. The typography hurts my eyes and the link layout makes it a pain to find anything specific. Takes me 10 times as long to find anything. :(

falcolas 11y ago
Yeah. I'm actually missing their previous documentation, which was much more readable and had some links on the left hand side down to anchors within a page. Now it's a bit of an inconsistent mess to find something.
geku 11y ago
Same here, I also find the new navigation a bit unconventional for such a large page structure.