6 ms·

There is literally no equipment available in the market that does what these Google's switches do. Cisco, Juniper, et al, protecting their technology and invest

by blrgeek 15y ago

There is literally no equipment available in the market that does what these Google's switches do. Cisco, Juniper, et al, protecting their technology and investments into switching over the last 30 years, just didn't have the balls to kill their old lines by doing this wholeheartedly. Essentially, they have spent the last 20 yrs building their software which runs on Motorola, MIPS, PowerPC, etc., running arcane switching protocols - not always interoperably even. And these 'software-less' switches can be made by almost anyone, since the software is their secret sauce. Think of it as going from Minicomputers, which were custom boxes which had custom hw/sw from a few vendors, to PCs, which have an 'open' design and are designed for interoperability. That's what OpenFlow does to the switching/networking ecosystem. And since none of the incumbents want to commit hara-kiri, a few startups are trying to do this, Nicira, BigSwitch, etc. Many others have OpenFlow compatible switches, but nowhere near the scale that Google would need in their datacenters. Brilliant stuff. And I'd love to see Cisco die because of this - they've kept the industry back for long enough.

noobface 15y ago
Fuck Spanning Tree edit: down voters, have you configured STP in a data center? have you had a single vmware esx instance shut down the root VLAN on a DC? Spanning tree is being addressed with solutions like this.
- wmf 15y ago
  As much as I agree with you, WAN backbones weren't running spanning tree to begin with.
  - noobface 15y ago
    Nope, they still link the instances. L2 domains traverse WANs in DCs now a days.
    - packetslave 15y ago
      Only in some (read: insane) datacenter designs. You don't need spanned layer 2 domains unless you're doing crazy things like long-distance Vmotion. There's almost always a better way that invites less pain.
      - noobface 15y ago
        Totally agree, but there are protocols in place to tunnel layer 2 domains to encompass multiple data centers. VMotion between DCs is not as crazy as it sounds, it's just prohibitively expensive.
- DanBC 15y ago
  There's a big difference between saying "This is good / bad; I've configured X and you have to put up with Y." and an empty "Fuck spanning tree". You have knowledge. I could learn stuff from you. I learn almost nothing from your comment "fuck spanning tree". This kind of behind the curtain stuff is mysterious to many people. I would welcome something that taught me more about it. I'd especially welcome informed insights from someone who works with the technology.
  - noobface 15y ago
    Sorry, I forget my audience when I'm on HN. I professionally teach classes regarding L2 networking and servers, so I do hope I have some insight to share. I was a Cisco systems engineer at one point in my life. Spanning Tree Protocol: http://en.wikipedia.org/wiki/Spanning_Tree_Protocol http://en.wikipedia.org/wiki/Spanning_Tree_Protocol STP as it's called, builds linear networks. Simple, single paths, through layer 2 (see: ethernet) networks. Think of spanning tree as a large state table tracking all MAC addresses on a network. If the state table realizes there a duplicate entries (ie: duplicate paths) for a single MAC address, it literally brings down the entire network to recreate path without duplicate entries. Most managed (commercial) ethernet switches, speak the spanning tree protocol; this protocol allows synchronization of MAC tables between switches. However, by default, the vmware vswitch does not speak this protocol. This creates problems when you multi-home servers (connect a single server to multiple siwtches). The vswitch does not participate in spanning tree, and the default vswitch "load-balances" by transmitting frames from the various ports it has accessible. This, in the traditional switches' eyes, constitutes a loop in the network and can bring an entire ethernet domain down. This is a horrific scenario during which all participating hosts lose network access for 15 seconds or more, depending on the configuration (STP vs. RVSTP). If the vswitch remains active with it's default settings, the network may be down until a network engineer realizes the problem or the server is taken offline. The reason I say "fuck spanning tree" is as a network engineer, I've taken entire data center's off-line due to a mistaken configuration on a ESX host (which I did not have visibility into at the time). This is obviously not a good way to go about production practices. Network coordination services, like the one developed by Google, stand a good chance of replacing this antiquated protocol. Everyone in the ethernet networking world has been plagued by STP and its related quirks. I'm, for one, very happy to see its demise and hope for a future clear of such, potentially, disruptive technology with data centers.
anusinha 15y ago
It seems as though ASICs are the method of choice for every high performance system. The examples that come to mind are supercomputer interconnects and the completely engineered from scratch Anton[1] machines that DE Shaw Research uses for very specific computational chemistry applications. [1]: http://en.wikipedia.org/wiki/Anton_%28computer%29 http://en.wikipedia.org/wiki/Anton_%28computer%29
- Symmetry 15y ago
  Yes, going from a general purpose CPU to an ASIC will generally improve performance per watt by two or three orders of magnitude.
- spydum 15y ago
  the packet switching still occurs in ASIC, it's the routing forwarding table population which can be done in software
CWuestefeld 15y ago
I'm nothing like a network engineer. But is the centralization described in this article a good thing for the Internet as a whole? I can see that within an organization's internal network, they can assess the importance of different communications, and route accordingly. So on the internal side, it's potentially a big win. But across the globe, who can assign the priority of traffic accurately and impartially? And isn't the decentralized nature of the current architecture an important feature, because of the way it can route around problems (be they technical or regulatory) of its own accord, without requiring a higher authority to tell it how (and thus without being susceptible to the agenda of that authority)?
- rumcajz 15y ago
  The point is that the switch is programmable. You can implement the centralised behaviour. You can implement decentralised one. You can implement anything you want.
  - CWuestefeld 15y ago
    Do there exist heuristics for decentralized traffic handling that this is able to support, but cannot be implemented through traditional approaches?
    - joestringer 15y ago
      The idea is to build a globally consistant view of the network. Currently, each node builds up its own view of the global state and routes based on this. Sure, distributed protocols allow us to share this information, but it's not guaranteed that the state will be accurate a few hops away. OpenFlow allows an entity to keep a global consistant state, and calculate the rules by which each of the nodes should forward. This logically centralised control can then enable higher utilisation of the network. Think about traffic reports on the radio -- If you are driving and know that there is a bottleneck on one highway, then you can take an alternative route. EDIT: I use "Global" in this context for within an AS, not necessarily internet-wide.
justincormack 15y ago
Sorry, "literally no equipment," except the openflow stuff? Which shipped some time back.
KevinEldon 15y ago
On the OpenFlow.org site I see that HP has a firmware upgrade for their switches that supports OpenFlow and IBM has an OpenFlow switch.
6ren 15y ago
If the customers and the way of buying is the same, Cisco will acquire and proceed. Dramatic technology changes don't kill incumbents if they are "sustaining" to their customers and way of doing business.

by blrgeek 15y ago

noobface 15y ago

Fuck Spanning Tree edit: down voters, have you configured STP in a data center? have you had a single vmware esx instance shut down the root VLAN on a DC? Spanning tree is being addressed with solutions like this.

wmf 15y ago
As much as I agree with you, WAN backbones weren't running spanning tree to begin with.
- noobface 15y ago
  Nope, they still link the instances. L2 domains traverse WANs in DCs now a days.
  - packetslave 15y ago
    Only in some (read: insane) datacenter designs. You don't need spanned layer 2 domains unless you're doing crazy things like long-distance Vmotion. There's almost always a better way that invites less pain.
    - noobface 15y ago
      Totally agree, but there are protocols in place to tunnel layer 2 domains to encompass multiple data centers. VMotion between DCs is not as crazy as it sounds, it's just prohibitively expensive.
DanBC 15y ago
There's a big difference between saying "This is good / bad; I've configured X and you have to put up with Y." and an empty "Fuck spanning tree". You have knowledge. I could learn stuff from you. I learn almost nothing from your comment "fuck spanning tree". This kind of behind the curtain stuff is mysterious to many people. I would welcome something that taught me more about it. I'd especially welcome informed insights from someone who works with the technology.
- noobface 15y ago
  Sorry, I forget my audience when I'm on HN. I professionally teach classes regarding L2 networking and servers, so I do hope I have some insight to share. I was a Cisco systems engineer at one point in my life. Spanning Tree Protocol: http://en.wikipedia.org/wiki/Spanning_Tree_Protocol http://en.wikipedia.org/wiki/Spanning_Tree_Protocol STP as it's called, builds linear networks. Simple, single paths, through layer 2 (see: ethernet) networks. Think of spanning tree as a large state table tracking all MAC addresses on a network. If the state table realizes there a duplicate entries (ie: duplicate paths) for a single MAC address, it literally brings down the entire network to recreate path without duplicate entries. Most managed (commercial) ethernet switches, speak the spanning tree protocol; this protocol allows synchronization of MAC tables between switches. However, by default, the vmware vswitch does not speak this protocol. This creates problems when you multi-home servers (connect a single server to multiple siwtches). The vswitch does not participate in spanning tree, and the default vswitch "load-balances" by transmitting frames from the various ports it has accessible. This, in the traditional switches' eyes, constitutes a loop in the network and can bring an entire ethernet domain down. This is a horrific scenario during which all participating hosts lose network access for 15 seconds or more, depending on the configuration (STP vs. RVSTP). If the vswitch remains active with it's default settings, the network may be down until a network engineer realizes the problem or the server is taken offline. The reason I say "fuck spanning tree" is as a network engineer, I've taken entire data center's off-line due to a mistaken configuration on a ESX host (which I did not have visibility into at the time). This is obviously not a good way to go about production practices. Network coordination services, like the one developed by Google, stand a good chance of replacing this antiquated protocol. Everyone in the ethernet networking world has been plagued by STP and its related quirks. I'm, for one, very happy to see its demise and hope for a future clear of such, potentially, disruptive technology with data centers.

anusinha 15y ago

It seems as though ASICs are the method of choice for every high performance system. The examples that come to mind are supercomputer interconnects and the completely engineered from scratch Anton[1] machines that DE Shaw Research uses for very specific computational chemistry applications. [1]: http://en.wikipedia.org/wiki/Anton_%28computer%29 http://en.wikipedia.org/wiki/Anton_%28computer%29

Symmetry 15y ago
Yes, going from a general purpose CPU to an ASIC will generally improve performance per watt by two or three orders of magnitude.
spydum 15y ago
the packet switching still occurs in ASIC, it's the routing forwarding table population which can be done in software

CWuestefeld 15y ago

I'm nothing like a network engineer. But is the centralization described in this article a good thing for the Internet as a whole? I can see that within an organization's internal network, they can assess the importance of different communications, and route accordingly. So on the internal side, it's potentially a big win. But across the globe, who can assign the priority of traffic accurately and impartially? And isn't the decentralized nature of the current architecture an important feature, because of the way it can route around problems (be they technical or regulatory) of its own accord, without requiring a higher authority to tell it how (and thus without being susceptible to the agenda of that authority)?

rumcajz 15y ago
The point is that the switch is programmable. You can implement the centralised behaviour. You can implement decentralised one. You can implement anything you want.
- CWuestefeld 15y ago
  Do there exist heuristics for decentralized traffic handling that this is able to support, but cannot be implemented through traditional approaches?
  - joestringer 15y ago
    The idea is to build a globally consistant view of the network. Currently, each node builds up its own view of the global state and routes based on this. Sure, distributed protocols allow us to share this information, but it's not guaranteed that the state will be accurate a few hops away. OpenFlow allows an entity to keep a global consistant state, and calculate the rules by which each of the nodes should forward. This logically centralised control can then enable higher utilisation of the network. Think about traffic reports on the radio -- If you are driving and know that there is a bottleneck on one highway, then you can take an alternative route. EDIT: I use "Global" in this context for within an AS, not necessarily internet-wide.

justincormack 15y ago

Sorry, "literally no equipment," except the openflow stuff? Which shipped some time back.

KevinEldon 15y ago

On the OpenFlow.org site I see that HP has a firmware upgrade for their switches that supports OpenFlow and IBM has an OpenFlow switch.

6ren 15y ago

If the customers and the way of buying is the same, Cisco will acquire and proceed. Dramatic technology changes don't kill incumbents if they are "sustaining" to their customers and way of doing business.