5 ms·

> Ethernet is stuck on 1500 byte frames, with the result of limiting TCP as well. That's just nuts on 100G or 400G networks, even on 10G it's a bit of a stretch

> Ethernet is stuck on 1500 byte frames, with the result of limiting TCP as well. That's just nuts on 100G or 400G networks, even on 10G it's a bit of a stretch already. How limiting is 1500 for the end-nodes for most people? For access-, distribution-, and core-layer switches the ASICs process things at wire speed, so does it matter what the frame size is? I can see some server applications, but we've had offloading for years now: * https://en.wikipedia.org/wiki/TCP_offload_engine https://en.wikipedia.org/wiki/TCP_offload_engine Netflix is hitting (encrypted) 800 Gb/s: * https://www.youtube.com/watch?v=36qZYL5RlgY https://www.youtube.com/watch?v=36qZYL5RlgY * https://news.ycombinator.com/item?id=32519881 https://news.ycombinator.com/item?id=32519881 > TCP is also sticking around for too long. Almost nothing wants a stream these days, lots and lots of applications reinvent datagrams on top of it. Tallk to the vendors of middleware boxes (e.g., (CPE) firewalls): we've had (say) SCTP and DCCP for twenty years that offer variations, but they're never allowed to pass through. No new Layer 4 can really be allowed through because of ossification, and so kludging everything to TCP/UDP port 80/443 is what we're left with. NAT and a lack of end-to-end also doesn't help: one has to futz around with things like STUN and TURN.

dale_glass 3y ago
> How limiting is 1500 for the end-nodes for most people? For access-, distribution-, and core-layer switches the ASICs process things at wire speed, so does it matter what the frame size is?\ Well, here's my personal experience for instance. If you take a Linux server and try to do put a 10G interface into a bridge, that disables a good amount of the offload capability of the network card. As a result you don't get 10G anymore even on fairly decent hardware. So VMs in a bridge with a 10G interface is already a configuration that doesn't quite work. VFIO does the trick fortunately. Also, surely even for ASICs there's a cost that comes with having to handle 150M packets/s. And yeah, you can do those speeds on switches and devices with limited functionality, but maybe I want some iptables rules on my server hooked up to 100G. And all of those are really workarounds that shouldn't be necessary. Having more than a hundred million packets per second is something that only happens because things got fossilized on a tiny packet size, not because there's actually some point to splitting up a 2 GB download into more than a million tiny bits.
- 15155 3y ago
  150M x 1500 byte packets is only 225 gigabytes/sec: an easy task on any midsize to large FPGA and certainly any ASIC.
toast0 3y ago
> How limiting is 1500 for the end-nodes for most people? Not very. For most end users, connection speeds are much less than 1G and 1500 is a reasonable size. > For access-, distribution-, and core-layer switches the ASICs process things at wire speed, so does it matter what the frame size is? Yes, for these systems, one of the limiting factors is routing lookups per second. You'll see some systems that can only process wire speed with large packets, not with small packets, maybe with a mix. Increasing the size of packets reduces the packet count, reducing the routing lookups or other per packet work.

dale_glass 3y ago

> How limiting is 1500 for the end-nodes for most people? For access-, distribution-, and core-layer switches the ASICs process things at wire speed, so does it matter what the frame size is?\ Well, here's my personal experience for instance. If you take a Linux server and try to do put a 10G interface into a bridge, that disables a good amount of the offload capability of the network card. As a result you don't get 10G anymore even on fairly decent hardware. So VMs in a bridge with a 10G interface is already a configuration that doesn't quite work. VFIO does the trick fortunately. Also, surely even for ASICs there's a cost that comes with having to handle 150M packets/s. And yeah, you can do those speeds on switches and devices with limited functionality, but maybe I want some iptables rules on my server hooked up to 100G. And all of those are really workarounds that shouldn't be necessary. Having more than a hundred million packets per second is something that only happens because things got fossilized on a tiny packet size, not because there's actually some point to splitting up a 2 GB download into more than a million tiny bits.

15155 3y ago
150M x 1500 byte packets is only 225 gigabytes/sec: an easy task on any midsize to large FPGA and certainly any ASIC.

toast0 3y ago

> How limiting is 1500 for the end-nodes for most people? Not very. For most end users, connection speeds are much less than 1G and 1500 is a reasonable size. > For access-, distribution-, and core-layer switches the ASICs process things at wire speed, so does it matter what the frame size is? Yes, for these systems, one of the limiting factors is routing lookups per second. You'll see some systems that can only process wire speed with large packets, not with small packets, maybe with a mix. Increasing the size of packets reduces the packet count, reducing the routing lookups or other per packet work.