3 ms·

That's a very interesting experiment in many levels. Haven't taken the time to look at the paper yet, but I'm curious of how you got your number of pps vs Gb/s

by molyss 7y ago

That's a very interesting experiment in many levels. Haven't taken the time to look at the paper yet, but I'm curious of how you got your number of pps vs Gb/s in the README : "full bidirectional load at 20 Gbit/s with 64 byte packets (29.76 Mpps)". sounds like 20Gb/s should be closer to 40Mpps than to 30Mpps. Did you hit CPU limits on the packet generator, or am I missing some packet header overhead ? Did you try bigger that 64-byte packets ? I'm curious how various runtimes would handle that. And how long did you run the benchmarks ? I couldn't really figure it out from the github or the paper. Mostly wondering if java and other Gc'd language showed improvement or degradation over time. I could see the JITs kicking in, but I could also see the GCs causing latency spikes.

benou 7y ago
> am I missing some packet header overhead ? Yes: Ethernet adds 20 bytes: 8 byte preamble/start of frame delimiter + 12 byte interframe gap => the "on-the-wire" size is actually 84-bytes => 20Gbps/84-bytes = 29.76Mpps > Did you try bigger that 64-byte packets ? I'm curious how various runtimes would handle that. In typical forwarding, packet size does not impact forwaring that much until you hit some bandwidth limit (PCIe, DDR and/or L3 cache) because you only touch the packet header (typically the 1st 64-bytes cacheline in the packet). The data transfer itself will be done by NIC DMA.
- emmericp 7y ago
  PCIe bandwidth also decreases with increasing packet size as there's a lot of overhead per packet. Memory isn't used, it's all handled in cache, hitting main memory is super slow.
yaantc 7y ago
You're only considering the MAC level size of 64 bytes but there is also the physical layer overhead, which pushes the effective size of a packet to 84 bytes (see [1]) with a 7 bytes preamble, a 1 byte start of frame delimiter and 12 bytes of inter-packets gap. If you use 84 bytes with 20 Gbps you get the 29.76 Mpps. [1] https://en.wikipedia.org/wiki/Ethernet_frame https://en.wikipedia.org/wiki/Ethernet_frame

by molyss 7y ago

benou 7y ago

> am I missing some packet header overhead ? Yes: Ethernet adds 20 bytes: 8 byte preamble/start of frame delimiter + 12 byte interframe gap => the "on-the-wire" size is actually 84-bytes => 20Gbps/84-bytes = 29.76Mpps > Did you try bigger that 64-byte packets ? I'm curious how various runtimes would handle that. In typical forwarding, packet size does not impact forwaring that much until you hit some bandwidth limit (PCIe, DDR and/or L3 cache) because you only touch the packet header (typically the 1st 64-bytes cacheline in the packet). The data transfer itself will be done by NIC DMA.

emmericp 7y ago
PCIe bandwidth also decreases with increasing packet size as there's a lot of overhead per packet. Memory isn't used, it's all handled in cache, hitting main memory is super slow.

yaantc 7y ago

You're only considering the MAC level size of 64 bytes but there is also the physical layer overhead, which pushes the effective size of a packet to 84 bytes (see [1]) with a 7 bytes preamble, a 1 byte start of frame delimiter and 12 bytes of inter-packets gap. If you use 84 bytes with 20 Gbps you get the 29.76 Mpps. [1] https://en.wikipedia.org/wiki/Ethernet_frame https://en.wikipedia.org/wiki/Ethernet_frame