5 ms·

THIS is what I'm talking about: http://cloudarchitectmusings.com/2013/01/03/word-of-caution-about-overextending-the-use-of-vxlan/ http://cloudarchitectmusings.

by Nrsolis 11y ago

THIS is what I'm talking about: http://cloudarchitectmusings.com/2013/01/03/word-of-caution-about-overextending-the-use-of-vxlan/ http://cloudarchitectmusings.com/2013/01/03/word-of-caution-... VxLAN is a technology with a very limited horizon in terms of functionality. If you're intending to really leverage containers in a super dense use case, then you'd be well advised to adopt a technology that has ready made ASIC support for cross the physical-to-virtual network boundary.

wmf 11y ago
That article says VXLAN isn't good for DCI, but DCI isn't a good idea anyway[1] and most people aren't trying to use Docker that way. And VXLAN is pretty much the only encapsulation format that has good ASIC support in both NICs and switches. [1] http://blog.ipspace.net/2014/10/vxlan-and-otv-saga-continues.html http://blog.ipspace.net/2014/10/vxlan-and-otv-saga-continues... http://blog.ipspace.net/2013/09/sooner-or-later-someone-will-pay-for.html http://blog.ipspace.net/2013/09/sooner-or-later-someone-will... http://blog.ipspace.net/2015/01/latency-killer-of-spread-out.html http://blog.ipspace.net/2015/01/latency-killer-of-spread-out...
- Nrsolis 11y ago
  Uh, no. MPLS has had SOLID ASIC support for maybe two decades now. Everyone who is putting in VxLAN-capable hardware is doing so because they have oodles of ESXi deployed and VxLAN is the only technology that's supported by VMware. If you're not wedded to the ESXi hypervisor, you can deploy either MPLS or MPLSoverGREoverIP (for your non-MPLS capable endpoints) and capture a whole stack of goodness to include interworking across an entire WAN infrastructure for DC-to-DC VM movement and EVPN support. MPLS has been doing L2VPN support for ages. Man, NSX/VxLAN didn't even have a real control plane until recently. They were going to use MULTICAST for their table updates! MULTICAST!
  - jpgvm 11y ago
    So much FUD. First of all VxLAN is already nicely offloaded by the majority of NICs and if not being terminated doesn't required hardware support in switching equipment. VxLAN is almost no lock-in. Currently to do MPLS in any shape or form you are going to be locked into proprietary network gear. MPLS has almost no software implementations, whereas Linux has native VxLAN support in vanilla bridging mode and OVS. Forget MPLS on Windows. MPLS is great tech don't get me wrong, however it's restricted to the carrier space because it doesn't play well with others.
    - Nrsolis 11y ago
      You're thinking of a very narrow use-case for VxLAN. What happens when your VM needs to talk to legacy networking gear? What happens when your VM(s) need to talk to a legacy security infrastructure that's mandated because of regulatory concerns? What happens when you need to route between two VxLAN domains? You're going to need to provide the VxLAN VTEP functionality someplace and that requires hardware support in the network someplace. Waving your hands and saying "that's not a concern" won't cut it. I have a customer that's facing this issue RIGHT NOW. And no, MPLS isn't a proprietary standard. It's open and it's available on oodles of networking hardware. It's interoperable and it's a proven technology that scales. Developing a tunneling protocol is the easy part. Developing a scalable control plane to handle RIB and FIB state is another matter entirely. Getting all of that to operate on a chipset that can do 2-4Tbps per slot is another thing altogether. Go check out some of the HUGE cloud infrastructure guys. They aren't using BrandX networking hardware to build their data-centers. They're using the Big-3 players.
      - jpgvm 11y ago
        No, they are not using the Big-3 players. ISPs are using Alcatel and Juniper still sure, big enterprise shops are still swearing by Cisco. Big cloud infrastructure has all moved to Linux on merchant silicon, either stuff like Quanta, Pico8 or even more DIY like OCP. As for terminating VxLAN on legacy gear, yeah that is probably a bad idea and someone probably made a bad decision to end up with an architecture that requires that. As for control plane. Most people using VxLAN at scale have their own control planes. They are actually stupidly easy to build because the edge is so easy to work with. I built my own that integrates with the Linux native VxLAN implementation using netlink to program the forwarding table. At the end of the day VxLAN isn't a great replacement for MPLS but it's a great encapsulation system for fully software defined datacenters that have already invested into a full control plane for all compute and networking (think Mesos, Kubernetes). It's also a good choice for smaller scale stuff because it scales down nicely. Multicast forwarding might suck in a real DC situation but for a lot of newbies playing around it's a good way to get into doing L2oL3. I don't think the WAN cases matter that much. At the point where it's a problem you have the people around to make said problem go away. Specifically trying to do cross DC IP address mobility is dumb in the first place. Most models that actually work well cross datacenter simply terminate it in one place and bring it up the workload on an entirely new set of resources in the new DC. This is much easier and is shared nothing usually (except maybe the dataset, but that is usually stored in S3/HDFS/Ceph/other distributed store here). Long story short, it does it's job fine. Use it for something it wasn't built for and yes it will hurt you.

by Nrsolis 11y ago

wmf 11y ago

That article says VXLAN isn't good for DCI, but DCI isn't a good idea anyway[1] and most people aren't trying to use Docker that way. And VXLAN is pretty much the only encapsulation format that has good ASIC support in both NICs and switches. [1] http://blog.ipspace.net/2014/10/vxlan-and-otv-saga-continues.html http://blog.ipspace.net/2014/10/vxlan-and-otv-saga-continues... http://blog.ipspace.net/2013/09/sooner-or-later-someone-will-pay-for.html http://blog.ipspace.net/2013/09/sooner-or-later-someone-will... http://blog.ipspace.net/2015/01/latency-killer-of-spread-out.html http://blog.ipspace.net/2015/01/latency-killer-of-spread-out...

Nrsolis 11y ago
Uh, no. MPLS has had SOLID ASIC support for maybe two decades now. Everyone who is putting in VxLAN-capable hardware is doing so because they have oodles of ESXi deployed and VxLAN is the only technology that's supported by VMware. If you're not wedded to the ESXi hypervisor, you can deploy either MPLS or MPLSoverGREoverIP (for your non-MPLS capable endpoints) and capture a whole stack of goodness to include interworking across an entire WAN infrastructure for DC-to-DC VM movement and EVPN support. MPLS has been doing L2VPN support for ages. Man, NSX/VxLAN didn't even have a real control plane until recently. They were going to use MULTICAST for their table updates! MULTICAST!
- jpgvm 11y ago
  So much FUD. First of all VxLAN is already nicely offloaded by the majority of NICs and if not being terminated doesn't required hardware support in switching equipment. VxLAN is almost no lock-in. Currently to do MPLS in any shape or form you are going to be locked into proprietary network gear. MPLS has almost no software implementations, whereas Linux has native VxLAN support in vanilla bridging mode and OVS. Forget MPLS on Windows. MPLS is great tech don't get me wrong, however it's restricted to the carrier space because it doesn't play well with others.
  - Nrsolis 11y ago
    You're thinking of a very narrow use-case for VxLAN. What happens when your VM needs to talk to legacy networking gear? What happens when your VM(s) need to talk to a legacy security infrastructure that's mandated because of regulatory concerns? What happens when you need to route between two VxLAN domains? You're going to need to provide the VxLAN VTEP functionality someplace and that requires hardware support in the network someplace. Waving your hands and saying "that's not a concern" won't cut it. I have a customer that's facing this issue RIGHT NOW. And no, MPLS isn't a proprietary standard. It's open and it's available on oodles of networking hardware. It's interoperable and it's a proven technology that scales. Developing a tunneling protocol is the easy part. Developing a scalable control plane to handle RIB and FIB state is another matter entirely. Getting all of that to operate on a chipset that can do 2-4Tbps per slot is another thing altogether. Go check out some of the HUGE cloud infrastructure guys. They aren't using BrandX networking hardware to build their data-centers. They're using the Big-3 players.
    - jpgvm 11y ago
      No, they are not using the Big-3 players. ISPs are using Alcatel and Juniper still sure, big enterprise shops are still swearing by Cisco. Big cloud infrastructure has all moved to Linux on merchant silicon, either stuff like Quanta, Pico8 or even more DIY like OCP. As for terminating VxLAN on legacy gear, yeah that is probably a bad idea and someone probably made a bad decision to end up with an architecture that requires that. As for control plane. Most people using VxLAN at scale have their own control planes. They are actually stupidly easy to build because the edge is so easy to work with. I built my own that integrates with the Linux native VxLAN implementation using netlink to program the forwarding table. At the end of the day VxLAN isn't a great replacement for MPLS but it's a great encapsulation system for fully software defined datacenters that have already invested into a full control plane for all compute and networking (think Mesos, Kubernetes). It's also a good choice for smaller scale stuff because it scales down nicely. Multicast forwarding might suck in a real DC situation but for a lot of newbies playing around it's a good way to get into doing L2oL3. I don't think the WAN cases matter that much. At the point where it's a problem you have the people around to make said problem go away. Specifically trying to do cross DC IP address mobility is dumb in the first place. Most models that actually work well cross datacenter simply terminate it in one place and bring it up the workload on an entirely new set of resources in the new DC. This is much easier and is shared nothing usually (except maybe the dataset, but that is usually stored in S3/HDFS/Ceph/other distributed store here). Long story short, it does it's job fine. Use it for something it wasn't built for and yes it will hurt you.