6 ms·

Awesome awesomeness. I read a blog post here about vxlan adding some 10% overhead in terms of both latency and bandwith, which seems very acceptable to me :) C

Awesome awesomeness. I read a blog post here about vxlan adding some 10% overhead in terms of both latency and bandwith, which seems very acceptable to me :) CoreOS has had this for a while in flannel (ref the same article as above), so it's not like it's a completely new thing, but if you prefer going with the crowd, then having overlay networking built right into docker will make for an awesome experience when using docker compose and docker swarm. Edit: Article: http://www.generictestdomain.net/docker/weave/networking/stupidity/2015/04/05/weave-is-kinda-slow/ http://www.generictestdomain.net/docker/weave/networking/stu... And the results: Flannel VXLan 96.55% bandwidth utilisation 140.52% latency. Without overlay latency was 91.8µs, and with vxlan it was 129µs.

Nrsolis 11y ago
I can't say I'm particularly happy with the choice of VxLAN as a tunneling technology but I'm not particularly surprised either.
- Nrsolis 11y ago
  THIS is what I'm talking about: http://cloudarchitectmusings.com/2013/01/03/word-of-caution-about-overextending-the-use-of-vxlan/ http://cloudarchitectmusings.com/2013/01/03/word-of-caution-... VxLAN is a technology with a very limited horizon in terms of functionality. If you're intending to really leverage containers in a super dense use case, then you'd be well advised to adopt a technology that has ready made ASIC support for cross the physical-to-virtual network boundary.
  - wmf 11y ago
    That article says VXLAN isn't good for DCI, but DCI isn't a good idea anyway[1] and most people aren't trying to use Docker that way. And VXLAN is pretty much the only encapsulation format that has good ASIC support in both NICs and switches. [1] http://blog.ipspace.net/2014/10/vxlan-and-otv-saga-continues.html http://blog.ipspace.net/2014/10/vxlan-and-otv-saga-continues... http://blog.ipspace.net/2013/09/sooner-or-later-someone-will-pay-for.html http://blog.ipspace.net/2013/09/sooner-or-later-someone-will... http://blog.ipspace.net/2015/01/latency-killer-of-spread-out.html http://blog.ipspace.net/2015/01/latency-killer-of-spread-out...
    - Nrsolis 11y ago
      Uh, no. MPLS has had SOLID ASIC support for maybe two decades now. Everyone who is putting in VxLAN-capable hardware is doing so because they have oodles of ESXi deployed and VxLAN is the only technology that's supported by VMware. If you're not wedded to the ESXi hypervisor, you can deploy either MPLS or MPLSoverGREoverIP (for your non-MPLS capable endpoints) and capture a whole stack of goodness to include interworking across an entire WAN infrastructure for DC-to-DC VM movement and EVPN support. MPLS has been doing L2VPN support for ages. Man, NSX/VxLAN didn't even have a real control plane until recently. They were going to use MULTICAST for their table updates! MULTICAST!
      - jpgvm 11y ago
        So much FUD. First of all VxLAN is already nicely offloaded by the majority of NICs and if not being terminated doesn't required hardware support in switching equipment. VxLAN is almost no lock-in. Currently to do MPLS in any shape or form you are going to be locked into proprietary network gear. MPLS has almost no software implementations, whereas Linux has native VxLAN support in vanilla bridging mode and OVS. Forget MPLS on Windows. MPLS is great tech don't get me wrong, however it's restricted to the carrier space because it doesn't play well with others.
        Nrsolis 11y ago
        You're thinking of a very narrow use-case for VxLAN. What happens when your VM needs to talk to legacy networking gear? What happens when your VM(s) need to talk to a legacy security infrastructure that's mandated because of regulatory concerns? What happens when you need to route between two VxLAN domains? You're going to need to provide the VxLAN VTEP functionality someplace and that requires hardware support in the network someplace. Waving your hands and saying "that's not a concern" won't cut it. I have a customer that's facing this issue RIGHT NOW. And no, MPLS isn't a proprietary standard. It's open and it's available on oodles of networking hardware. It's interoperable and it's a proven technology that scales. Developing a tunneling protocol is the easy part. Developing a scalable control plane to handle RIB and FIB state is another matter entirely. Getting all of that to operate on a chipset that can do 2-4Tbps per slot is another thing altogether. Go check out some of the HUGE cloud infrastructure guys. They aren't using BrandX networking hardware to build their data-centers. They're using the Big-3 players.

Nrsolis 11y ago

I can't say I'm particularly happy with the choice of VxLAN as a tunneling technology but I'm not particularly surprised either.

Nrsolis 11y ago
THIS is what I'm talking about: http://cloudarchitectmusings.com/2013/01/03/word-of-caution-about-overextending-the-use-of-vxlan/ http://cloudarchitectmusings.com/2013/01/03/word-of-caution-... VxLAN is a technology with a very limited horizon in terms of functionality. If you're intending to really leverage containers in a super dense use case, then you'd be well advised to adopt a technology that has ready made ASIC support for cross the physical-to-virtual network boundary.
- wmf 11y ago
  That article says VXLAN isn't good for DCI, but DCI isn't a good idea anyway[1] and most people aren't trying to use Docker that way. And VXLAN is pretty much the only encapsulation format that has good ASIC support in both NICs and switches. [1] http://blog.ipspace.net/2014/10/vxlan-and-otv-saga-continues.html http://blog.ipspace.net/2014/10/vxlan-and-otv-saga-continues... http://blog.ipspace.net/2013/09/sooner-or-later-someone-will-pay-for.html http://blog.ipspace.net/2013/09/sooner-or-later-someone-will... http://blog.ipspace.net/2015/01/latency-killer-of-spread-out.html http://blog.ipspace.net/2015/01/latency-killer-of-spread-out...
  - Nrsolis 11y ago
    Uh, no. MPLS has had SOLID ASIC support for maybe two decades now. Everyone who is putting in VxLAN-capable hardware is doing so because they have oodles of ESXi deployed and VxLAN is the only technology that's supported by VMware. If you're not wedded to the ESXi hypervisor, you can deploy either MPLS or MPLSoverGREoverIP (for your non-MPLS capable endpoints) and capture a whole stack of goodness to include interworking across an entire WAN infrastructure for DC-to-DC VM movement and EVPN support. MPLS has been doing L2VPN support for ages. Man, NSX/VxLAN didn't even have a real control plane until recently. They were going to use MULTICAST for their table updates! MULTICAST!
    - jpgvm 11y ago
      So much FUD. First of all VxLAN is already nicely offloaded by the majority of NICs and if not being terminated doesn't required hardware support in switching equipment. VxLAN is almost no lock-in. Currently to do MPLS in any shape or form you are going to be locked into proprietary network gear. MPLS has almost no software implementations, whereas Linux has native VxLAN support in vanilla bridging mode and OVS. Forget MPLS on Windows. MPLS is great tech don't get me wrong, however it's restricted to the carrier space because it doesn't play well with others.
      - Nrsolis 11y ago
        You're thinking of a very narrow use-case for VxLAN. What happens when your VM needs to talk to legacy networking gear? What happens when your VM(s) need to talk to a legacy security infrastructure that's mandated because of regulatory concerns? What happens when you need to route between two VxLAN domains? You're going to need to provide the VxLAN VTEP functionality someplace and that requires hardware support in the network someplace. Waving your hands and saying "that's not a concern" won't cut it. I have a customer that's facing this issue RIGHT NOW. And no, MPLS isn't a proprietary standard. It's open and it's available on oodles of networking hardware. It's interoperable and it's a proven technology that scales. Developing a tunneling protocol is the easy part. Developing a scalable control plane to handle RIB and FIB state is another matter entirely. Getting all of that to operate on a chipset that can do 2-4Tbps per slot is another thing altogether. Go check out some of the HUGE cloud infrastructure guys. They aren't using BrandX networking hardware to build their data-centers. They're using the Big-3 players.