4 ms·

I had always assumed MPLS (and the even more hideous, SRv6) were designed by software engineers. If you are designing a hardware network processor, traversing a

by nsteel 4y ago

I had always assumed MPLS (and the even more hideous, SRv6) were designed by software engineers. If you are designing a hardware network processor, traversing a variable stack of labels at line-rate is pretty horrible.

bcrl 4y ago
Eh? Each hop only looked at the outermost label, keeping it, doing a push or a pop. The label namespace is small, which made it possible to use it directly as an index into a small SRAM to do a lookup and determine what the next hop is. That's how MPLS made line rate possible at a time when there wasn't enough space on silicon to do full IPv4 route lookups. It was very much a design driven by hardware requirements for how to do hardware line rate forwarding with limited resources.
- nsteel 4y ago
  Correct, the (dumb, cheap) devices within the network have an easy time. The devices on the edge can have a lot of work to do, potentially popping multiple labels. And then there's a bunch of meta labels that have other meanings (I'm not actually sure if those are standardised or not).
  - bcrl 4y ago
    Edge routers already have to cope with adding and removing arbitrary sized headers (just look at tunneling protocols like IPIP, L2TP or the more modern VXLAN), so it's not like MPLS with multiple labels is a new requirement for the data path. And it's only at the originating site that you have to push multiple labels onto a packet -- the receiving end should only have to pop at most 1 label off as the transport network would consume the rest. I had to implement a software MLPS data path once upon a time maybe a decade ago. I've also written some Verilog code for adding and removing headers on packets on FPGAs. Once you have to support VLANs in hardware, you're already half way towards inserting an arbitrary number of bytes into a packet as higher data rates in FPGAs mean wide busses with all the corner cases of packets being aligned at various offsets into a 256 bit or 512 bit data path. Figuring out how to implement an ethernet CRC32 block that meets timing in a Kintex 7 was certainly fun!
    - nsteel 4y ago
      MPLS was originally designed to be simple and fast. As you (and another) have commented, that was to help hardware. But it's grown into something that can be more complicated and even the P switches may read further down the stack to find hash labels etc (but yes, still only lookup using the top label). One benefit was to avoid using CAM but that's getting harder to realise. We don't have just switching labels and these other types may not be consumed, leaving the egress PE popping multiple labels. This is fine and the datapath is designed to make these sorts of packet manipulations, it's just harder to do that at full speed as people get further from the nice, simple original MPLS cases. When using multiple layers of encapsulation, customers don't make the same performance assumptions. Things are worse with SRv6 where the need to read much further into packets is approaching the pain point. Most (all?) network processors have a limit of what you can manipulate at full speed. There's another painful case but I don't think I can discuss that.
noselasd 4y ago
It's the oposite, trying to route or switch on variable length prefixes(as you would with IPv4 or V6), is pretty horrible to do in hardware, thanks to CAM it's atleast doeable. Route or switch directly on MAC addresses is ok, but creating policies becomes painful as there's no hierarchy. With MPLS, you get simple fixed length headers with single integer values is quite easy to push/pop and look up in hardware, and you can have software to generate the policies and manage hierarchies, placing the complexities and hard work at the MPLS edges, and having simple/dumb (aka. cheap) devices in the core.

by nsteel 4y ago

bcrl 4y ago

Eh? Each hop only looked at the outermost label, keeping it, doing a push or a pop. The label namespace is small, which made it possible to use it directly as an index into a small SRAM to do a lookup and determine what the next hop is. That's how MPLS made line rate possible at a time when there wasn't enough space on silicon to do full IPv4 route lookups. It was very much a design driven by hardware requirements for how to do hardware line rate forwarding with limited resources.

nsteel 4y ago
Correct, the (dumb, cheap) devices within the network have an easy time. The devices on the edge can have a lot of work to do, potentially popping multiple labels. And then there's a bunch of meta labels that have other meanings (I'm not actually sure if those are standardised or not).
- bcrl 4y ago
  Edge routers already have to cope with adding and removing arbitrary sized headers (just look at tunneling protocols like IPIP, L2TP or the more modern VXLAN), so it's not like MPLS with multiple labels is a new requirement for the data path. And it's only at the originating site that you have to push multiple labels onto a packet -- the receiving end should only have to pop at most 1 label off as the transport network would consume the rest. I had to implement a software MLPS data path once upon a time maybe a decade ago. I've also written some Verilog code for adding and removing headers on packets on FPGAs. Once you have to support VLANs in hardware, you're already half way towards inserting an arbitrary number of bytes into a packet as higher data rates in FPGAs mean wide busses with all the corner cases of packets being aligned at various offsets into a 256 bit or 512 bit data path. Figuring out how to implement an ethernet CRC32 block that meets timing in a Kintex 7 was certainly fun!
  - nsteel 4y ago
    MPLS was originally designed to be simple and fast. As you (and another) have commented, that was to help hardware. But it's grown into something that can be more complicated and even the P switches may read further down the stack to find hash labels etc (but yes, still only lookup using the top label). One benefit was to avoid using CAM but that's getting harder to realise. We don't have just switching labels and these other types may not be consumed, leaving the egress PE popping multiple labels. This is fine and the datapath is designed to make these sorts of packet manipulations, it's just harder to do that at full speed as people get further from the nice, simple original MPLS cases. When using multiple layers of encapsulation, customers don't make the same performance assumptions. Things are worse with SRv6 where the need to read much further into packets is approaching the pain point. Most (all?) network processors have a limit of what you can manipulate at full speed. There's another painful case but I don't think I can discuss that.

noselasd 4y ago

It's the oposite, trying to route or switch on variable length prefixes(as you would with IPv4 or V6), is pretty horrible to do in hardware, thanks to CAM it's atleast doeable. Route or switch directly on MAC addresses is ok, but creating policies becomes painful as there's no hierarchy. With MPLS, you get simple fixed length headers with single integer values is quite easy to push/pop and look up in hardware, and you can have software to generate the policies and manage hierarchies, placing the complexities and hard work at the MPLS edges, and having simple/dumb (aka. cheap) devices in the core.