4 ms·

100x faster than Linux is certainly fast. IIRC, Linux packet processing is considered rather slow (though full featured, well behaved and configurable). VPP he

100x faster than Linux is certainly fast. IIRC, Linux packet processing is considered rather slow (though full featured, well behaved and configurable). VPP here seems to be a "user-mode network stack", as far as I can tell. I was kind of attracted to the title because I was hoping for SIMD / Vector compute maybe even GPUs, but that doesn't seem to be the case. Still, a usermode network stack is apparently a must-have for any very-high performance network application. I've never needed it, but a lot of optimizers talk about how "slow" Linux networking is when you actually benchmark it.

crotchfire 3y ago
> "user-mode network stack" Kernels typically cannot use vector instructions because if they did they would need to save and restore the vector register state when servicing interrupts. There is a very large performance cost to doing that. Moving packet processing into userspace means adding latency, including TLB pressure, in order to do the context switch. I imagine that we might get some innovation by allowing to configure the system such that the kernel owns the vector registers and userspace is not allowed to use them. If your primary interest in vector registers/instructions is packet processing, and you're doing that in kernelspace, you might not mind it if userspace can't use those registers.
- wmf 3y ago
  Moving packet processing into userspace means adding latency, including TLB pressure, in order to do the context switch. This isn't the case because VPP polls the NIC from userspace and never enters the kernel. There are no context switches.
  - vacuity 3y ago
    Is there something like an IOMMU to provide secure access to the NIC?
    - ADSSDA 3y ago
      Yes, DPDK (which VPP is built on) heavily utilizes IOMMU to provide host protection. https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html discusses it a bit.
- crest 3y ago
  The vector in vector packet processing has little to nothing to do with vector instruction sets (SSE/AVX, VMX, RVV, etc.). Only the very latest CPUs (and historical supercomputers) have scatter/gather instructions capable of efficiently extracting packet header fields from multiple packets in parallel and if you have a large enough batch of packets process switching the vector register file is worth it to the kernel. It's just that most kernels don't spill the userspace vector registers on every context switch because it's more common to switch back to the same thread (or an other userspace thread) than using the vector registers inside the kernel. Both Linux and *BSD can and do make limited use of vector registers inside the kernel e.g. for fast encryption/decryption because it's worth the start up cost. If I understood the VPP design and implementation details correctly they try to reduce the amortised cache misses for a batch of packets by running all packets in a batch through each software pipeline stage before processing the next stage. This should result in very good average instruction cache hit rates and should also help with data cache hit rates because packet headers are small and can be prefetched while the forwarding data structures e.g. 1 million IPv4 prefixes and their next hops can be hard to fit into L2 data caches and won't fit into L1 data caches. I assume a carefully tuned implementation can make further gains by dedicating cores to specific pipeline stages to keep the data caches hotter at the cost of copying processed packet headers their next stage in new (sub-)batches. The actual packet content is only relevant for a few operations like encryption/decryption and modern highend NICs have line rate crypto engines to help with IPsec or TLS.
  - benou 3y ago
    Batching packets bring several benefits: - amortizing cache misses are you mentioned - better use of out-of-order, superscalar processors: by processing multiple independent packets in parallel, the processor can fill more execution units - enable the use of vector instructions (SSE/AVX, VMX etc): again, processing multiple independent packets in parallel means you can leverage SIMD. SIMD instructions are used pervasively in VPP
- sweetjuly 3y ago
  > Kernels typically cannot use vector instructions because if they did they would need to save and restore the vector register state when servicing interrupts. There is a very large performance cost to doing that. While this is true, it definitely doesn't seem to stop people like Apple [1] from using SIMD extensions in the kernel anyways. On ARMv8, it's an extra 512 bytes (32 qword registers) or eight (and change, depending on alignment) dirty cache lines. Whether or not this causes a serious performance impact will depend on how well the kernel can actually make use of SIMD (saving and restoring might cost 1% perf but if speeding up the kernel wins 5%, who cares!). It could be interesting to play with this on other kernels to verify these assumptions, it might be worth enabling in the kernel on some devices! Kernels do tend to do a lot of moving data around, which these extensions (on ARM, anyways) excel at. [1] https://github.com/apple-oss-distributions/xnu/blob/main/osfmk/arm64/exception_asm.h#L162-L177 https://github.com/apple-oss-distributions/xnu/blob/main/osf...
reflexe 3y ago
Actually, in its root it is based on simd and prefetching. In short, each part of the packet processing graph is a node. It receives a vector of packets (represented as a vector of packet indexes), then the output is one or more vectors, each goes as an input to the next step in the processing graph. This architecture maximizes cache hits and heats the branch predictor (since we run the same small code for many packets instead of the whole graph for each packet). You can read more about it here: https://s3-docs.fd.io/vpp/24.02/aboutvpp/scalar-vs-vector-packet-processing.html https://s3-docs.fd.io/vpp/24.02/aboutvpp/scalar-vs-vector-pa...
- dragontamer 3y ago
  I can certainly imagine some SIMD concepts in that. Particularly stream-compaction (or in AVX512 case: VPCOMPRESSD and VPEXPANDD instructions) EDIT: I guess from a SIMD-perspective, I'd have expected an interleaved set of packets, a-la struct-of-arrays rather than array-of-structs. But maybe that doesn't make sense for packet formats.
  - wmf 3y ago
    The NIC gives you an array (ring buffer) of pointers to structs (packets). Interleaving them into SOA format would probably cost more than any speedup from SIMD.
    - dragontamer 3y ago
      Yeah, but its difficult to write a SIMD / AVX512 routine if things aren't in SOA format. I can see how this approach described is "vector-like", even if the vector is this... imaginary unit that's parallelizing over the branch predictor instead of an explicit SIMD-code. This "vector" organization probably has 99.999%+ branch prediction or something, effectively parallelizing the concept. But not in the SIMD-way. So still useful, but not what I thought originally based on the title.
      - xoranth 3y ago
        A ring buffer of pointers to structs is friendly to gather instructions. That said, the documentation shows a graph of operations applied to each packet. I'd expect that to lead to a lot of "divergence", and therefore being non-SIMD friendly. (also, x86-64 CPUs with good gather instructions are rare, and sibling comments show that this is aimed at lower end CPUs. That makes SIMD even less relevant.)

crotchfire 3y ago

> "user-mode network stack" Kernels typically cannot use vector instructions because if they did they would need to save and restore the vector register state when servicing interrupts. There is a very large performance cost to doing that. Moving packet processing into userspace means adding latency, including TLB pressure, in order to do the context switch. I imagine that we might get some innovation by allowing to configure the system such that the kernel owns the vector registers and userspace is not allowed to use them. If your primary interest in vector registers/instructions is packet processing, and you're doing that in kernelspace, you might not mind it if userspace can't use those registers.

wmf 3y ago
Moving packet processing into userspace means adding latency, including TLB pressure, in order to do the context switch. This isn't the case because VPP polls the NIC from userspace and never enters the kernel. There are no context switches.
- vacuity 3y ago
  Is there something like an IOMMU to provide secure access to the NIC?
  - ADSSDA 3y ago
    Yes, DPDK (which VPP is built on) heavily utilizes IOMMU to provide host protection. https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html discusses it a bit.
crest 3y ago
The vector in vector packet processing has little to nothing to do with vector instruction sets (SSE/AVX, VMX, RVV, etc.). Only the very latest CPUs (and historical supercomputers) have scatter/gather instructions capable of efficiently extracting packet header fields from multiple packets in parallel and if you have a large enough batch of packets process switching the vector register file is worth it to the kernel. It's just that most kernels don't spill the userspace vector registers on every context switch because it's more common to switch back to the same thread (or an other userspace thread) than using the vector registers inside the kernel. Both Linux and *BSD can and do make limited use of vector registers inside the kernel e.g. for fast encryption/decryption because it's worth the start up cost. If I understood the VPP design and implementation details correctly they try to reduce the amortised cache misses for a batch of packets by running all packets in a batch through each software pipeline stage before processing the next stage. This should result in very good average instruction cache hit rates and should also help with data cache hit rates because packet headers are small and can be prefetched while the forwarding data structures e.g. 1 million IPv4 prefixes and their next hops can be hard to fit into L2 data caches and won't fit into L1 data caches. I assume a carefully tuned implementation can make further gains by dedicating cores to specific pipeline stages to keep the data caches hotter at the cost of copying processed packet headers their next stage in new (sub-)batches. The actual packet content is only relevant for a few operations like encryption/decryption and modern highend NICs have line rate crypto engines to help with IPsec or TLS.
- benou 3y ago
  Batching packets bring several benefits: - amortizing cache misses are you mentioned - better use of out-of-order, superscalar processors: by processing multiple independent packets in parallel, the processor can fill more execution units - enable the use of vector instructions (SSE/AVX, VMX etc): again, processing multiple independent packets in parallel means you can leverage SIMD. SIMD instructions are used pervasively in VPP
sweetjuly 3y ago
> Kernels typically cannot use vector instructions because if they did they would need to save and restore the vector register state when servicing interrupts. There is a very large performance cost to doing that. While this is true, it definitely doesn't seem to stop people like Apple [1] from using SIMD extensions in the kernel anyways. On ARMv8, it's an extra 512 bytes (32 qword registers) or eight (and change, depending on alignment) dirty cache lines. Whether or not this causes a serious performance impact will depend on how well the kernel can actually make use of SIMD (saving and restoring might cost 1% perf but if speeding up the kernel wins 5%, who cares!). It could be interesting to play with this on other kernels to verify these assumptions, it might be worth enabling in the kernel on some devices! Kernels do tend to do a lot of moving data around, which these extensions (on ARM, anyways) excel at. [1] https://github.com/apple-oss-distributions/xnu/blob/main/osfmk/arm64/exception_asm.h#L162-L177 https://github.com/apple-oss-distributions/xnu/blob/main/osf...

reflexe 3y ago

Actually, in its root it is based on simd and prefetching. In short, each part of the packet processing graph is a node. It receives a vector of packets (represented as a vector of packet indexes), then the output is one or more vectors, each goes as an input to the next step in the processing graph. This architecture maximizes cache hits and heats the branch predictor (since we run the same small code for many packets instead of the whole graph for each packet). You can read more about it here: https://s3-docs.fd.io/vpp/24.02/aboutvpp/scalar-vs-vector-packet-processing.html https://s3-docs.fd.io/vpp/24.02/aboutvpp/scalar-vs-vector-pa...

dragontamer 3y ago
I can certainly imagine some SIMD concepts in that. Particularly stream-compaction (or in AVX512 case: VPCOMPRESSD and VPEXPANDD instructions) EDIT: I guess from a SIMD-perspective, I'd have expected an interleaved set of packets, a-la struct-of-arrays rather than array-of-structs. But maybe that doesn't make sense for packet formats.
- wmf 3y ago
  The NIC gives you an array (ring buffer) of pointers to structs (packets). Interleaving them into SOA format would probably cost more than any speedup from SIMD.
  - dragontamer 3y ago
    Yeah, but its difficult to write a SIMD / AVX512 routine if things aren't in SOA format. I can see how this approach described is "vector-like", even if the vector is this... imaginary unit that's parallelizing over the branch predictor instead of an explicit SIMD-code. This "vector" organization probably has 99.999%+ branch prediction or something, effectively parallelizing the concept. But not in the SIMD-way. So still useful, but not what I thought originally based on the title.
    - xoranth 3y ago
      A ring buffer of pointers to structs is friendly to gather instructions. That said, the documentation shows a graph of operations applied to each packet. I'd expect that to lead to a lot of "divergence", and therefore being non-SIMD friendly. (also, x86-64 CPUs with good gather instructions are rare, and sibling comments show that this is aimed at lower end CPUs. That makes SIMD even less relevant.)