4 ms·

> And L2 switching is broken by design anyway, but that's a story of hysterical raisins for another day Well I'm curious. I can't think of any significant way

> And L2 switching is broken by design anyway, but that's a story of hysterical raisins for another day Well I'm curious. I can't think of any significant way it is "broken by design", so either this is hyperbole or I'm so used to the brokenness I'm not even thinking about it.

p_l 5y ago
The story as I learnt it goes around this way - hopefully on this forum someone with first-hand knowledge could chime in: 1. Ethernet happens, is designed around bus topology with shared medium and everyone talks by filtering out messages for themselves (with half of the addresses for multicast) 2. Digital works on moving ethernet from bus to star topology, design explicitly disallows connecting stars to each other without L3 router 3. Unfortunately, a non-trivial product range ends up based on LAT - essentially serial port over ethernet - and supposedly because of miscommunication LAT is very... raw-ethernet solution. No way to route it sensibly. 4. Suddenly, there's a need for larger L2 segments, except Ethernet has no way to support them (it finally gained one around starting ~2005 by throwing everything you know about L2 switching out) 5. It's too late to add features to ethernet that would make it work in larger span than single star, and possibility of loops bringing down exists, so do multicast storms (those weren't fixed). 6. The budget doesn't allow to put in a lot of computing power, a z80 gets thrown in. Spanning Tree Protocol gets created in vague hope to mitigate the curse of large L2 ethernet zones. We get stuck with primitive MAC learning 7. Genie is out of the box, and since you can crap out a too-large ethernet network much cheaper than do a proper routed one, the curse continues. Since cheap is the king, you often do not even get STP. Large scale networks fail when interns misconnect cables, multigigabit backbones end up doing 10mbit because STP made an ancient switch in the cleaning closet into root of the tree. Cats and Dogs living together, etc. 8. From around ~2005, proposals to fix it proper show up. Solution? Put routing into ethernet, using IS-IS for routing. On the other side, increasingly crazy centralized "decentralized" SDNs also try to setup L2 forwarding to deal with applications that can't deal with real IP subnetting. Somehow passing ethernet over XMPP over TLS (with BGP involved somewhere) is still better than ethernet's mac-learning.
- kazen44 5y ago
  avoid spanning tree like the plague in large datacenter networks as well. because of the scale and it becomes an impossible black box. there is a reason evpn exists, and is it to solve this exact issue by making gateways handle all logic normally stretched across to the other side of a l2vpn.
  - ArchOversight 5y ago
    I am a firm believer in layer 3 everywhere. No more layer 2 connectivity as much as possible. It's how I deployed VM's at scale using BGP from the VM host to the top of rack switch. VM's could route to each other, but no layer 2 connectivity. It allowed for easy migration of systems between VM hosts too, as the ToR would learn the /128 or /32 and traffic would route to the new VM host.
    - p_l 5y ago
      Exactly - your L2 Ethernet shouldn't go beyond immediate connection between end system and first L3 router, in DC conditions it should be to Tor... Or on-Hypervisor router. Larger L2 spans should be done only when required, and preferably with things like TRILL/SPB.
- wmf 5y ago
  Ethernet switching isn't broken. STP works fine at reasonable scale (as long as you leave it on) and 1980s history isn't relevant now. Obviously routing is better than switching and we can now do routing to the host with affordable "L3 switches", but switching is still usable.
  - p_l 5y ago
    The very reason we're talking about IPv4 vs IPv6 is because of 1980 (and 80s, 1980) history, concerning people getting convinced that the temporary solution that IPv4 was supposed to be will be won't need more than obviously short 32bits and will be replaced with something better before wide adoption.
  - 5y ago

p_l 5y ago

The story as I learnt it goes around this way - hopefully on this forum someone with first-hand knowledge could chime in: 1. Ethernet happens, is designed around bus topology with shared medium and everyone talks by filtering out messages for themselves (with half of the addresses for multicast) 2. Digital works on moving ethernet from bus to star topology, design explicitly disallows connecting stars to each other without L3 router 3. Unfortunately, a non-trivial product range ends up based on LAT - essentially serial port over ethernet - and supposedly because of miscommunication LAT is very... raw-ethernet solution. No way to route it sensibly. 4. Suddenly, there's a need for larger L2 segments, except Ethernet has no way to support them (it finally gained one around starting ~2005 by throwing everything you know about L2 switching out) 5. It's too late to add features to ethernet that would make it work in larger span than single star, and possibility of loops bringing down exists, so do multicast storms (those weren't fixed). 6. The budget doesn't allow to put in a lot of computing power, a z80 gets thrown in. Spanning Tree Protocol gets created in vague hope to mitigate the curse of large L2 ethernet zones. We get stuck with primitive MAC learning 7. Genie is out of the box, and since you can crap out a too-large ethernet network much cheaper than do a proper routed one, the curse continues. Since cheap is the king, you often do not even get STP. Large scale networks fail when interns misconnect cables, multigigabit backbones end up doing 10mbit because STP made an ancient switch in the cleaning closet into root of the tree. Cats and Dogs living together, etc. 8. From around ~2005, proposals to fix it proper show up. Solution? Put routing into ethernet, using IS-IS for routing. On the other side, increasingly crazy centralized "decentralized" SDNs also try to setup L2 forwarding to deal with applications that can't deal with real IP subnetting. Somehow passing ethernet over XMPP over TLS (with BGP involved somewhere) is still better than ethernet's mac-learning.

kazen44 5y ago
avoid spanning tree like the plague in large datacenter networks as well. because of the scale and it becomes an impossible black box. there is a reason evpn exists, and is it to solve this exact issue by making gateways handle all logic normally stretched across to the other side of a l2vpn.
- ArchOversight 5y ago
  I am a firm believer in layer 3 everywhere. No more layer 2 connectivity as much as possible. It's how I deployed VM's at scale using BGP from the VM host to the top of rack switch. VM's could route to each other, but no layer 2 connectivity. It allowed for easy migration of systems between VM hosts too, as the ToR would learn the /128 or /32 and traffic would route to the new VM host.
  - p_l 5y ago
    Exactly - your L2 Ethernet shouldn't go beyond immediate connection between end system and first L3 router, in DC conditions it should be to Tor... Or on-Hypervisor router. Larger L2 spans should be done only when required, and preferably with things like TRILL/SPB.
wmf 5y ago
Ethernet switching isn't broken. STP works fine at reasonable scale (as long as you leave it on) and 1980s history isn't relevant now. Obviously routing is better than switching and we can now do routing to the host with affordable "L3 switches", but switching is still usable.
- p_l 5y ago
  The very reason we're talking about IPv4 vs IPv6 is because of 1980 (and 80s, 1980) history, concerning people getting convinced that the temporary solution that IPv4 was supposed to be will be won't need more than obviously short 32bits and will be replaced with something better before wide adoption.
- 5y ago