7 ms·
Going Big with TCP Packets
- codechad 5y agoInteresting development. However, I do wonder at some point if those desiring fast packet processing shouldn't just bite the bullet and adopt something DPDK-based (eg, FDIO/VPP). I think pure-interrupt packet processing is challenged (for the reasons laid out in the article). But getting people to dedicate a core/thread [to RX/TX polling] is also a hard sell. Either way this is good news - more pps/bps!
- jsnell 5y agoKernel bypass is not a panacea. Even with a totally userspace networking stack, there will be a lot of per-packet rather than per-byte processing, and often packets from different sources will be sufficiently interleaved that you can't even do any kind of useful batching fastpath .
- signa11 5y agotaking cpu-cores out of normal scheduling, and dedicating them for just packet-forwarding etc. is the first step. not sure how popular ddio is outside of intel ecosystem, but shunting packets between nic and cpu-cache (to and from) is very useful for such workloads imho.
- gjulianm 5y ago> But getting people to dedicate a core/thread [to RX/TX polling] is also a hard sell. I think that if you're at the level where the kernel network stack is not enough, core pinning is a no brainer. You're probably spending more than one core in receiving data, moving to a polling thread pinned to a single core will not only improve performance due to the polling, but also due to the scheduler not pushing your process out and to the possibility of putting the threads on the appropriate NUMA nodes.
- trasz 5y agoOr simply use network cards with full TCP offload (https://en.wikipedia.org/wiki/TCP_offload_engine https://en.wikipedia.org/wiki/TCP_offload_engine).
- deleted 5y ago[deleted]
- jeffbee 5y agoThe organization in question already bypasses the kernel with their “SNAP” scheme. Perhaps they are motivated to keep improving kernel IP stack simply because practically all GCP customers still use it. https://blog.acolyer.org/2019/11/11/snap-networking/ https://blog.acolyer.org/2019/11/11/snap-networking/
- codechad 5y ago
- touisteur 5y agoI wanted to pay someone to work on a dpdk layer for zeromq, I'm not sure anyone would be interested but I feel it'd help alleviate part of the dpdk pain.
- bogomipz 5y agoWhat's the's pain exactly with using DPDK?
- touisteur 5y agoYou have to be root/suid, have to implement your netstack yourself (ip, tcp, etc.), have to use the special threads which may or may not play nice with other threading libraries, locking and lock-free stuff, and there are subtle differences between controler behaviours that are smoothed over by the kernel and drivers, but hey you don't want those. For very simple forwarding and L2/L3 stuff it's easy to start and you can get to high perf quick enough. For anything applicative it can quickly become hard. I lost so much time with e.g. jumbo frames support, arp, frame re-ordering and low-level footguns, honestly sometimes I miss the pain of plain sockets. And in the end what I want is mostly giving a large reassembled 2-100MB buffer to userland to avx512-stuff all over - same as I do with zmq on a (bunch of) 10Gb link(s). Or send to gpu via gpudirect, and the joy of programming DMAs instead of 'normal' cudaMemcpyAsync. And don't forget to reserve a (bunch of) physical machines for CI. I'm not saying it's not useful, on the contrary. You get to send/receive 800Gbit/s of traffic on a single socket system! But it feels like a setback in usability, learning curve. I'm saying I'd like a simpler API over it, to transmit/receive pub/sub large messages.
- bogomipz 5y agoThanks for the wonderfully detailed response. I had a couple questions: >"And in the end what I want is mostly giving a large reassembled 2-100MB buffer to userland to avx512-stuff all over - same as I do with zmq" How does the avx512 come into play here? I know its mainly used for HPC type workloads. Is there a more general usage pattern for it as well? I know Linux was very vocal about his disdain for avx512 and mentioned people were using it for memcpy[1] Is that the case for DPDK? Second is "zmq" here ZeroMQ, the messaging library? I would type of application space you work in. [1] https://www.techradar.com/news/linux-founder-chastises-intel-for-using-power-virus-tech https://www.techradar.com/news/linux-founder-chastises-intel...
- baruch 5y agoPart of the pain with kernel bypass is memory management. For real performance gains you need zero copy and for that you need to work inside the packets. Perfectly doable for a green field project not so east I think for a large existing system.
- cyberge99 5y agoAren’t cat videos delivered with UDP?
- bragr 5y agoSeems pretty useful once all the drivers, utilities, network devices catch up. This will be all the rage in SAN and other performance network environments in a couple years. Would be great to see a more thorough performance evaluation. For example I wonder if the throughput and latency hold in different network conditions - e.g. network conditions with high latency, packet loss, and both. I presume after a certain point of degradation the larger packet size becomes of penalty because your retransmissions become very wasteful.
- joshbaptiste 5y agoWhat are the current MTU sizes of these performance backbone environments?
- jmyeet 5y agoI'm a big believer in Postel's Law (also known as the robustness principle) [1]. Basically be liberal in what you accept and conservative in what you send. The problem with TCP is that the actual Layer 2 and 3 infrastructure doesn't obey this principle. It's taken the opposite stance to simply reject anything that's weird or unexpected. There's a reason and history for why this is. The most defensible is security. Less defensible is things like deep packet inspection. This is a well-knwon problem and falls under the term (which I love) "ossification" [2]. As an example, much of the Internet stops working with MTUs above ~1500 (which this article mentions). Large packets (eg MTU ~9000) are almost a necessity for 10+ GbE but that often won't work outside of the LAN. So I guess my point is changes like this will take a long time (if ever) to get widespread support. This could still impact, say, data center and other "local" deployments but I guess you have to start somewhere. [1]: https://en.wikipedia.org/wiki/Robustness_principle https://en.wikipedia.org/wiki/Robustness_principle [2]: https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7738442 https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7738442
- treesknees 5y agoPart of this problem is there really isn't a layer 2 protocol for discovering the max MTU of a path. You have to use a higher protocol like ICMP to find it and then adjust your endpoint. The Internet will work with large L3/L4 packets assuming the path is able to create fragments. The article talked about the kernel's ability to hold fragments but it wasn't clear to me how Internet fragmentation comes into play or whether the performance gain is lost when using this on the Internet versus a LAN. Side tangent, this is one issue I have with the major cloud providers. Azure, AWS, et al all advertise high-bandwidth interfaces upwards of 100Gbps, but you can't saturate it anywhere near those rates with 1500 Byte packets from the Internet.
- jeffbee 5y agoSure, but the amount of traffic inside AWS is way bigger than the amount of traffic that crosses its boundary. This is true for a lot of operators. Netflix and YouTube and similar loads that are virtually all egress are special cases.
- 5y ago
- drewg123 5y agoThe article says "Enabling a packet size of 185,000 bytes increased network throughput by nearly 50%" So they essentially tripled the max TSO/GRO size from ~64k to 185k and that resulted in more throughput, presumably due to less overhead. That's a LOT of per-packet overheads, and seems a bit surprising to me. In tests I've done, I have not seen much difference between a 16k max tso and a 64k max TSO. Then again, my benchmark is generally a TCP_STREAM (throughput), and not a TCP_RR (ping pong) test. In general, outside a datacenter, the last thing you want to do is dump 185KB for the same connection on the wire in a giant burst, as you'll blow out router buffers when transiting the internet from 100GbE in your datacenter to some crappy DSL router at your users' houses. So I wonder how much the benefit is reduced when using some kind of software based packet pacing? I think linux has some kind of packet scheduler designed to avoid large bursts, etc. Does it break apart these giant TSOs? Or does using it just cap the max TSO size? At least in FreeBSD. the TCP stack controls pacing and one thing it does is to send down smaller TSOs to NICs that don't support hardware pacing. The goal is to reduce the burst size on the wire, so TCP will send down a few KB, wait several milliseconds, dribbled down more, etc. This reduces our average TSO size, and increases overhead.
- touisteur 5y agoOne of the things datacenter people want is to avoid dpdk-everything just because they put 4 200GbE in their server. I sure would like to just zmq everything... Especially for internal streaming applications, that would be a boon. I haven't read the patchset yet but I'm wondering whether it would work with UDP. TCP can be too much 'kitchensink' if you don't need reordering or many-resends.
- mprovost 5y agoI've played around with jumbo frames in the past. One issue is that none of the checksums at any layer were designed for packets this big. Both the Ethernet and TCP checksums don't have enough bits to reliably detect errors, even on 9k frames. This is ok if you're doing checksumming at the application layer, but often you're not, maybe it's something like NFS that assumes lower levels of the stack take care of message integrity. As transmission speeds have gone up, the bit error rate has stayed the same, so you can just do the math on how many errors you're going to have per day.
- brohee 5y agoThe NFS WG was adding Integrity Measurement Architecture that would address that and more, but the effort seems to have fizzled out, i assume COVID didn't help. https://datatracker.ietf.org/doc/html/draft-ietf-nfsv4-integrity-measurement-08 https://datatracker.ietf.org/doc/html/draft-ietf-nfsv4-integ...
- touisteur 5y agoThat's actually a very interesting observation. How much bits should a checksum take for such payloads? At least the 9000 jumbo things.
- mprovost 5y agoiSCSI uses a 32 bit CRC. (TCP is 16 bits). There was a draft to use 32 bit TCP checksums but it never got traction: https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-anumita-tcp-stronger-checksum https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-anumit...
- rjsw 5y agoSCTP uses a 32 bit checksum.
- Sesse__ 5y agoYou need to have application-level checksums anyway, since pretty much every router or switch these days will recalculate checksums on all layers (they need to if routing the packet, and since pretty much all COTS chipsets can do L3, they do so all the time even if the switch's software won't let you configure it), so if a bit flips in any router's processing, your checksums won't detect it.
- bogomipz 5y ago>"Imagine, for a second, that you are trying to keep up with a 100Gb/s network adapter. As networking developer Jesper Brouer described back in 2015, if one is using the longstanding maximum packet size of 1,538 bytes, running the interface at full speed means coping with over eight-million packets per second." Should this not be "frame size" instead of "packet size"? Unless I am reading this incorrectly the maximum TCP packet size is 1500(20 bytes for IP header, 20 bytes for TCP header and 1460 bytes for the actual payload.) I'm not trying to be pedantic but the other 38 bytes would all be considered part of the Ethernet and not TCP no?
- eklitzke 5y agoThis is wrong for a few reasons. It is common to try to fit TCP packets into 1500 bytes (minus the bytes for the IP and TCP headers) because 1500 bytes is a common MTU for Ethernet, but this isn't a limitation of TCP, it's just something done to avoid fragmentation. Without extensions TCP packets can be sized up to 64KB. For example, in an environment with jumbo Ethernet frames you might have 9000 byte MTU at the link layer which is just fine. But it's also fine to send a 9000 byte or even 64KB TCP packet over a network with a 1500 byte MTU, the packet will just be split up at the IP layer using IP fragmentation. And as the article points out, RFC 2675 allows creating TCP packets whose payload is even larger than 64KB.
- bogomipz 5y agoNo it is not wrong. In fact you are supporting my point exactly. My question was about the terminology the author was using. Packets are TCP terminology. Frames are ethernet terminology and the 1500 byte limit is because this is the MTU of an Ethernet interface, again a layer 2 concern not layer 4. See also elsewhere in this thread where I also asked what the benefit of this over Jumbo frames.
- bogomipz 5y agoCan someone say what the benefit of this is over just using existing Ethernet jumbo frames?
- justsomehnguy 5y ago> existing Ethernet jumbo frames 9000 / 1500 = 6 > Enabling a packet size of 185,000 bytes 185000 / 1500 = 123.3 Jumbo would give you [up to] 6 times less packets for the same bandwidth => 6 times less overhead on the packet processing. BIG TCP could give up to 123 times less packets (only theoretically, of course) but surely would dramatically lower the processing overhead for the big payloads.
- bogomipz 5y agoTank you for the clear break down. This puts it into perspective. That is indeed significant. Cheers.
- deleted 5y ago[deleted]
- sriram_sun 5y agoDo virtual adapters on VMs for communication between host and guest stand to benefit from this? For e.g. scp between host and guest. Current max mtu size on VirtualBox is 9000 right?
- lawrenceyan 5y agoOh, so this must be why the maximum size of a Solana transaction is 1232 bytes (to fit into the 1538 byte limit of a packet with some wiggle room).
- jhugo 5y agoSeems strange, there is no fundamental reason why a transaction should have to fit inside a single packet at a lower layer unless there is some nasty leaking of abstractions going on.