3 ms·

More importantly it's probably to 2 redundant routers on SIX side also so that both can handle HW failure. On the SIX side the ports are probably already in a

by jpgvm 5y ago

More importantly it's probably to 2 redundant routers on SIX side also so that both can handle HW failure. On the SIX side the ports are probably already in a redundant chassis but having 2 ports probably still scores you connections to two fully independent chassis in different failure zones.

kloch 5y ago
The last time I got a tour of the SIX, there were switches on two different floors in the Westin building. This is more for capacity than redundancy. The original location is literally a janitor's closet in a hallway and it's so tight that exhaust fans are built into the door. The second location was in proper four-post racks in a small room. It is possible they have ports on both switches for redundancy but more likely both are on the newer location. The SIX is also built into other buildings and they could connect to more than one, but there is typically only 200G backhaul to the other sites. https://www.seattleix.net/topology https://www.seattleix.net/topology 2x100G to an IX are usually connected as separate ports, each with their own assigned IP ont he IX subnet. BGP multipath can enable load balancing across both ports. Otherwise you might have some peers configured on one port and others on the second. These days I like to have all peers on both ports and use BGP multipath. If both ports are connected to a single switch on the IX side and the same router on the customer side then LACP can be used to combine them into a single 200G port with one IP. This is a less common configuration unless you are doing more than two ports (4x100G as 2x 2x100G lags for example). There is another consideration: what happens if there is an icmp or arp flood on the IX? It's not supposed to happen but sometimes it does, even on the most major and well-run exchanges. In that case you might want to have all of your IX ports on one of your routers, keeping the other one "clean", just in case your control plane policing doesn't mitigate it. Different networks place different priority on IX ports. In most cases it's ok if it's down for a short time (traffic will just shift to paid transit) so redundancy is not essential. In other cases it may be too much traffic to reroute so you care more about IX reliability and redundancy.
- jpgvm 5y ago
  Just want to say really great summary. I know these things but I bet many folks reading this glean an insight or two. Been a lot time since I have done all this stuff though. I sort of miss it sometimes (though not the pager that comes with the territory).
throw0101a 5y ago
> More importantly it's probably to 2 redundant routers on SIX side also so that both can handle HW failure. Strictly speaking, IXPs do not work at OSI Layer 3, but usually at Layer 2. All the Layer 3 routing logic happens in the routers of the members/customers of the IXP. The IXP is just there to provide a neutral interconnect to everyone without any filtering or policy. The IXP runs router reflectors where people can choose to advertise themselves, and then other folks see who is in the neighbourhood and make their own routing policy choices. The folks at Network Startup Resource Center (NSRC) have a series of videos on "IXP Design and Implementation": * https://learn.nsrc.org/bgp https://learn.nsrc.org/bgp They're by Philip Smith, who helped set up LINX (London, UK) in the early days: * https://nsrc.org/bios/PhilipSmith.html https://nsrc.org/bios/PhilipSmith.html

by jpgvm 5y ago

kloch 5y ago

The last time I got a tour of the SIX, there were switches on two different floors in the Westin building. This is more for capacity than redundancy. The original location is literally a janitor's closet in a hallway and it's so tight that exhaust fans are built into the door. The second location was in proper four-post racks in a small room. It is possible they have ports on both switches for redundancy but more likely both are on the newer location. The SIX is also built into other buildings and they could connect to more than one, but there is typically only 200G backhaul to the other sites. https://www.seattleix.net/topology https://www.seattleix.net/topology 2x100G to an IX are usually connected as separate ports, each with their own assigned IP ont he IX subnet. BGP multipath can enable load balancing across both ports. Otherwise you might have some peers configured on one port and others on the second. These days I like to have all peers on both ports and use BGP multipath. If both ports are connected to a single switch on the IX side and the same router on the customer side then LACP can be used to combine them into a single 200G port with one IP. This is a less common configuration unless you are doing more than two ports (4x100G as 2x 2x100G lags for example). There is another consideration: what happens if there is an icmp or arp flood on the IX? It's not supposed to happen but sometimes it does, even on the most major and well-run exchanges. In that case you might want to have all of your IX ports on one of your routers, keeping the other one "clean", just in case your control plane policing doesn't mitigate it. Different networks place different priority on IX ports. In most cases it's ok if it's down for a short time (traffic will just shift to paid transit) so redundancy is not essential. In other cases it may be too much traffic to reroute so you care more about IX reliability and redundancy.

jpgvm 5y ago
Just want to say really great summary. I know these things but I bet many folks reading this glean an insight or two. Been a lot time since I have done all this stuff though. I sort of miss it sometimes (though not the pager that comes with the territory).

throw0101a 5y ago

> More importantly it's probably to 2 redundant routers on SIX side also so that both can handle HW failure. Strictly speaking, IXPs do not work at OSI Layer 3, but usually at Layer 2. All the Layer 3 routing logic happens in the routers of the members/customers of the IXP. The IXP is just there to provide a neutral interconnect to everyone without any filtering or policy. The IXP runs router reflectors where people can choose to advertise themselves, and then other folks see who is in the neighbourhood and make their own routing policy choices. The folks at Network Startup Resource Center (NSRC) have a series of videos on "IXP Design and Implementation": * https://learn.nsrc.org/bgp https://learn.nsrc.org/bgp They're by Philip Smith, who helped set up LINX (London, UK) in the early days: * https://nsrc.org/bios/PhilipSmith.html https://nsrc.org/bios/PhilipSmith.html