3 ms·

The thing I was imagining replacing here is the supervision card alone (the sup720-XL), not the entire box. The supervision card doesn't need 720Gbps bandwidth;

by derefr 3y ago

The thing I was imagining replacing here is the supervision card alone (the sup720-XL), not the entire box. The supervision card doesn't need 720Gbps bandwidth; it's just spitting routing decisions, not entire packets. (And I would bet that established L3 flows cache their routing decisions in per-ASIC channel descriptors for short TTLs; so it's probably not even being prompted for a routing decision for every single packet, either.) I assume the supervision card is prompted over some wire protocol by the ASICs in the switches for routing decisions, and responds back to them with a predictable delay. To achieve parity with the existing supervision card, it "only" needs to emit 30MM one-byte(!) decisions per second. I.e. a top-line input rate of 3840Mbps (for IPv6), and a top-line output rate of 240Mbps. Basically, it confuses me why you can't slap such a "supervision card" together by taking a modern 8-core single-board computer that can fit the entire routing table into L2 cache on each core, and has a PCI-e socket; plugging an Infiniband card or whatever into it; and then running an RTOS on it. Heck, when you think about it, SBCs are so cheap compared to a single used sup720-XL, that you could cluster them inside your router, with each supervision shard taking routing-decision load from 1/Nth of the ASICs.

phirephly 3y ago
256 next hops isn't enough. Typical ASICs support 20,000 to 160,000 next hops FECs. Cisco tried caching routing decisions from non-line rate routing engines in the 90s, and the industry learned the lesson that it's a bad idea. Caching works until you overflow the cache for some reason, and then the box completely falls over as it thrashes.
dfox 3y ago
There are different modes in which 6500 can operate. The original 15Mpps mode (30Mpps is architecturally the same thing at higher clock speed) involves the line card arbitrating for a shared parallel bus on the backplane, transfering relevant data through this bus to the sup and after getting the result it transmits the whole packet over another shared parallel bus to the egress line card. In this mode the actual data do not traverse the sup card. In the newer dCEF modes there is a switching fabric of point-to-point source synchronous links and the crossbar switch of this fabric is physically located on the supervisor cards, so the “720” in “sup720” really means that there is 720Gbps of switching bandwidth on the card (although it is not exactly tightly coupled to the rest of the card that implements the “supervisor” itself, to large extent it is colocated on the same card due to physical convenience of that design, IIRC it is actually implemented as a mezzanine board). At ~400Mpps doing the routing decisions in software is going to be a significant bottleneck even with some kind of modern SoC. But OTOH, today you can design the whole thing in bunch of other ways, as it is significantly easier to implement something with large amount of separate RAM blocks connected through some random arrangement of buses and crossbars controlled by few tightly coupled CPU cores as a one big ASIC (or even FPGA). But all that assumes that the thing you are building is functionally an L2 switch, maybe designed with ridiculous amount of L3 acceleration (which is exactly what 6500/7600 is). As a point of reference C7200 is a router roughly contemporary to 6500/7600 and architecturally it is just a MIPS-based SBC with six PCI slots.

by derefr 3y ago

phirephly 3y ago

256 next hops isn't enough. Typical ASICs support 20,000 to 160,000 next hops FECs. Cisco tried caching routing decisions from non-line rate routing engines in the 90s, and the industry learned the lesson that it's a bad idea. Caching works until you overflow the cache for some reason, and then the box completely falls over as it thrashes.

dfox 3y ago

There are different modes in which 6500 can operate. The original 15Mpps mode (30Mpps is architecturally the same thing at higher clock speed) involves the line card arbitrating for a shared parallel bus on the backplane, transfering relevant data through this bus to the sup and after getting the result it transmits the whole packet over another shared parallel bus to the egress line card. In this mode the actual data do not traverse the sup card. In the newer dCEF modes there is a switching fabric of point-to-point source synchronous links and the crossbar switch of this fabric is physically located on the supervisor cards, so the “720” in “sup720” really means that there is 720Gbps of switching bandwidth on the card (although it is not exactly tightly coupled to the rest of the card that implements the “supervisor” itself, to large extent it is colocated on the same card due to physical convenience of that design, IIRC it is actually implemented as a mezzanine board). At ~400Mpps doing the routing decisions in software is going to be a significant bottleneck even with some kind of modern SoC. But OTOH, today you can design the whole thing in bunch of other ways, as it is significantly easier to implement something with large amount of separate RAM blocks connected through some random arrangement of buses and crossbars controlled by few tightly coupled CPU cores as a one big ASIC (or even FPGA). But all that assumes that the thing you are building is functionally an L2 switch, maybe designed with ridiculous amount of L3 acceleration (which is exactly what 6500/7600 is). As a point of reference C7200 is a router roughly contemporary to 6500/7600 and architecturally it is just a MIPS-based SBC with six PCI slots.