16 ms·

6500/7600 is weird beast where the same HW architecture (and well 4 generations of it that are to some extent compatible although internally very different) are

by dfox 3y ago

6500/7600 is weird beast where the same HW architecture (and well 4 generations of it that are to some extent compatible although internally very different) are used as both high port density ethernet switch (6500) and as an router (7600, the only meaningful difference in the HW is that 7600 is painted white). In almost all configurations all the forwarding decisions are done in hardware, which involves somewhat expensive memory chip that Cisco calls TCAM (Ternary Content Addressable Memory), essentially an SRAM array interspersed with comparators that can find a FIB record matching the destination address with most specific mask in one cycle. In the actual implementation it is not a full TCAM and involves some kind of hashing and lookup takes multiple cycles, to make it not ridiculously expensive, but still it is not priced like your typical off the shelf 6T-SRAM (which by itself is orders of magnitude more expensive than random high volume DDRx SDRAM). Another thing is that the thing is somewhat old and designed by Cisco who will not go out of their way to produce something that makes it obvious that their product can be replaced by a x86 box running Linux And good luck making a x86 box that has 720Gbps bandwidth and 144 ethernet ports. There is a question of exactly what is the real world practical application for 48 port gigabit linecard (there even is a PoE option, 6500 in right config can prowide kilowatts of PoE) in a router that can speak BGP, but well, you can build such a thing from 6500.

derefr 3y ago
The thing I was imagining replacing here is the supervision card alone (the sup720-XL), not the entire box. The supervision card doesn't need 720Gbps bandwidth; it's just spitting routing decisions, not entire packets. (And I would bet that established L3 flows cache their routing decisions in per-ASIC channel descriptors for short TTLs; so it's probably not even being prompted for a routing decision for every single packet, either.) I assume the supervision card is prompted over some wire protocol by the ASICs in the switches for routing decisions, and responds back to them with a predictable delay. To achieve parity with the existing supervision card, it "only" needs to emit 30MM one-byte(!) decisions per second. I.e. a top-line input rate of 3840Mbps (for IPv6), and a top-line output rate of 240Mbps. Basically, it confuses me why you can't slap such a "supervision card" together by taking a modern 8-core single-board computer that can fit the entire routing table into L2 cache on each core, and has a PCI-e socket; plugging an Infiniband card or whatever into it; and then running an RTOS on it. Heck, when you think about it, SBCs are so cheap compared to a single used sup720-XL, that you could cluster them inside your router, with each supervision shard taking routing-decision load from 1/Nth of the ASICs.
- phirephly 3y ago
  256 next hops isn't enough. Typical ASICs support 20,000 to 160,000 next hops FECs. Cisco tried caching routing decisions from non-line rate routing engines in the 90s, and the industry learned the lesson that it's a bad idea. Caching works until you overflow the cache for some reason, and then the box completely falls over as it thrashes.
- dfox 3y ago
  There are different modes in which 6500 can operate. The original 15Mpps mode (30Mpps is architecturally the same thing at higher clock speed) involves the line card arbitrating for a shared parallel bus on the backplane, transfering relevant data through this bus to the sup and after getting the result it transmits the whole packet over another shared parallel bus to the egress line card. In this mode the actual data do not traverse the sup card. In the newer dCEF modes there is a switching fabric of point-to-point source synchronous links and the crossbar switch of this fabric is physically located on the supervisor cards, so the “720” in “sup720” really means that there is 720Gbps of switching bandwidth on the card (although it is not exactly tightly coupled to the rest of the card that implements the “supervisor” itself, to large extent it is colocated on the same card due to physical convenience of that design, IIRC it is actually implemented as a mezzanine board). At ~400Mpps doing the routing decisions in software is going to be a significant bottleneck even with some kind of modern SoC. But OTOH, today you can design the whole thing in bunch of other ways, as it is significantly easier to implement something with large amount of separate RAM blocks connected through some random arrangement of buses and crossbars controlled by few tightly coupled CPU cores as a one big ASIC (or even FPGA). But all that assumes that the thing you are building is functionally an L2 switch, maybe designed with ridiculous amount of L3 acceleration (which is exactly what 6500/7600 is). As a point of reference C7200 is a router roughly contemporary to 6500/7600 and architecturally it is just a MIPS-based SBC with six PCI slots.

by dfox 3y ago

derefr 3y ago

The thing I was imagining replacing here is the supervision card alone (the sup720-XL), not the entire box. The supervision card doesn't need 720Gbps bandwidth; it's just spitting routing decisions, not entire packets. (And I would bet that established L3 flows cache their routing decisions in per-ASIC channel descriptors for short TTLs; so it's probably not even being prompted for a routing decision for every single packet, either.) I assume the supervision card is prompted over some wire protocol by the ASICs in the switches for routing decisions, and responds back to them with a predictable delay. To achieve parity with the existing supervision card, it "only" needs to emit 30MM one-byte(!) decisions per second. I.e. a top-line input rate of 3840Mbps (for IPv6), and a top-line output rate of 240Mbps. Basically, it confuses me why you can't slap such a "supervision card" together by taking a modern 8-core single-board computer that can fit the entire routing table into L2 cache on each core, and has a PCI-e socket; plugging an Infiniband card or whatever into it; and then running an RTOS on it. Heck, when you think about it, SBCs are so cheap compared to a single used sup720-XL, that you could cluster them inside your router, with each supervision shard taking routing-decision load from 1/Nth of the ASICs.

phirephly 3y ago
256 next hops isn't enough. Typical ASICs support 20,000 to 160,000 next hops FECs. Cisco tried caching routing decisions from non-line rate routing engines in the 90s, and the industry learned the lesson that it's a bad idea. Caching works until you overflow the cache for some reason, and then the box completely falls over as it thrashes.
dfox 3y ago
There are different modes in which 6500 can operate. The original 15Mpps mode (30Mpps is architecturally the same thing at higher clock speed) involves the line card arbitrating for a shared parallel bus on the backplane, transfering relevant data through this bus to the sup and after getting the result it transmits the whole packet over another shared parallel bus to the egress line card. In this mode the actual data do not traverse the sup card. In the newer dCEF modes there is a switching fabric of point-to-point source synchronous links and the crossbar switch of this fabric is physically located on the supervisor cards, so the “720” in “sup720” really means that there is 720Gbps of switching bandwidth on the card (although it is not exactly tightly coupled to the rest of the card that implements the “supervisor” itself, to large extent it is colocated on the same card due to physical convenience of that design, IIRC it is actually implemented as a mezzanine board). At ~400Mpps doing the routing decisions in software is going to be a significant bottleneck even with some kind of modern SoC. But OTOH, today you can design the whole thing in bunch of other ways, as it is significantly easier to implement something with large amount of separate RAM blocks connected through some random arrangement of buses and crossbars controlled by few tightly coupled CPU cores as a one big ASIC (or even FPGA). But all that assumes that the thing you are building is functionally an L2 switch, maybe designed with ridiculous amount of L3 acceleration (which is exactly what 6500/7600 is). As a point of reference C7200 is a router roughly contemporary to 6500/7600 and architecturally it is just a MIPS-based SBC with six PCI slots.