5 ms·
How to achieve low latency with 10Gbps Ethernet
- rasz_pl 11y agostep one should be 'we move drivers to userspace and bypass kernel completelly' edit: finished reading, its the last step :/ and ~halves best case they were able to achieve with tweaks.
- baruch 11y agoThat's still not easy enough to do. UDP is simple enough but doing TCP in those drivers is still a pain. There are DPDK with LWIP or with some BSD tacked on top but so far I didn't find something that I can just take for a ride in that area. For the extreme cases you really want to control the packet allocation for receive and make the buffers your app uses be also used for the network transmit so you get a real zero-copy.
- ghshephard 11y agoIt's an absolutely wonderful article - perhaps one of the best I've seen regarding latency tweaking on a linux stack - but I don't understand why so little about the final step, "Since we have a Solarflare network card handy, we can use the OpenOnload kernel bypass technology to skip the kernel network stack all together:" was discussed. That was like the major punch line of the entire article, and it was given a minor "oh, by the way, " sort of treatment. Great article regardless. I'm wondering what the latency would have been if they had started with that step.
- thrownaway2424 11y agoWhat are the tradeoffs? Is there any downside to this userspace network stack, or is it pure win? Edit: other than needing the thousand-dollar NIC, I mean.
- jnordwick 11y agoNo more local tcp dump. No more kernel routing, copying, logging, or filtering of packets. As someone who's hacked on OoL and other kernal bypass stacks a little, it is a huge win if you are latency sensitive especially for just thrashing the core by constant polling for absolute lowest latency.
- usefulcat 11y agoHaving used it, it will generally consume all available CPU cycles on the core(s) on which it runs. So presumably it is doing a lot of busy-waiting. Which, if low latency is your top priority, may be just fine.
- neomantra 11y agoOpenOnload's spinning behavior is highly tunable and disabled by default. The default profile might appear to take some more CPU cycles, but really it's the same CPU cycles that the kernel would be using to operate the network stack (but across the kernel/userspace barrier and possibly on another core/socket/numanode, etc, introducing latency). It's a great article, but I'm surprised that the author introduces spinning and task-setting without noting CPU isolation (either with cgroups or isolcpus). Not doing so could introduce problems.
- pjc50 11y agoIt's open-source but patent-encumbered, which is a strange state. It's not a user-space TCP stack but a "card-space" one: it's offloading all the work into the card.
- wmf 11y agoI don't think that's the case for Solarflare; they call it "onload" for a reason.
- neomantra 11y agoIt's not offloading ALL the work onto the card. With 'offload' people often think about feature like computing checksums in hardware, but that's not a big deal on modern hardware. OpenOnload definitely leverages the cards capabilities, notably packet buffer handling which is exposed in their ef_vi API (there's a comment about this on the article's page); the flow steering (mentioned in the article) is a big help too. OpenOnload is a user space network stack built on top of ef_vi. But so much the win come from being userspace and tunable. OpenOnload also accelerate pipes, loopback UDP/TCP and epoll, all of which are unrelated to their hardware. Indeed, having unaccelerated fds in the same epoll set as accelerated fds kills performance because it needs to ask the kernel for the status of the unaccelerated fds. That said, I think you need to have a SolarFlare card for OpenOnload to work at all, even if you just use it for local communication. I've not tried it though.
- thrownaway2424 11y agoThere have been a spate of articles lately that sum up to one fact: out of the box, all Linux distributions are tuned for nothing in particular. With some configs you can reduce your latency by an order of magnitude, increase your MySQL ops by an order of magnitude, etc. This is one reason why I, as an application-level programmer, am glad to work at a company big enough to have someone else dealing with the platform on my behalf. I don't want to spend ten years slowly discovering traps like default window size tuned for T1 speed, default min_rto tuned for interplanetary traffic, not enough PIDs to support a 96-core machine, etc.
- deleted 11y ago[deleted]
- fulafel 11y agoNitpick: Maybe you are thinking of Windows wrt buffer sizes? Linux TCP out of the box has window scaling enabled since 2004 or so. "sysctl net.ipv4.tcp_rmem" tells me max buffer size is 6+ MB on my old 4GB laptop, which should be good to saturate 10 a gigabit pipe with up to 50 ms RTT... Mininum retransmission tuning might get you something on your own network if it's congested but would probably be bad to crank down for general internet usage, given things like bufferbloat and large delays caused by wireless L2 retransmissions.
- acd 11y agoLinux TCP window tuning is in fact off, agree with thrownaway. It's by the way a trick that the CDN providers use since they tune their stacks to increase the download speed. But if you could as well do the same tuning yourself.
- superuser2 11y agoHow can one learn to do this? Are there good books/classes on the topic? Learning from mentors seems pretty vulnerable to cargo culting/bad science but I don't know where to even start.
- j1e 11y agohttp://www.brendangregg.com/ http://www.brendangregg.com/
- fulafel 11y agoThe extreme measures (bypassing kernel etc) allowed them to shave ~20 us off the latency, which corresponds to a few kilometers. So this could indeed make a difference for latency-bound apps talking to eachother inside a data center.
- wtallis 11y agoReally important point: this is measuring baseline latency of sending one packet at a time. They're not measuring latency under load where queueing delays are possible. The picture there usually looks pretty different and some of the settings they're changing here may be harmful or irrelevant to that case.
- majke 11y agoTotally. Throughput is an enemy of latency. If you have some other applications on that RX queues or CPU's you can't expect any latency numbers.
- brianwawok 11y agoAlso why upstream smart batching is so important. If you only have 1 packet shovel it off. But maybe wait 1-2 micros and see if you can combine 10 packets (up to MTU) to decrease overall latency.
- wtallis 11y agoIt depends some on what layer you're talking about. Nagle's algorithm makes a lot of sense in a lot of situations. But if you've got a complete packet that's ready to encapsulate in an ethernet frame and send over the wire, you only want to try for batching if you're actually hitting a CPU limit from interrupt frequency or something like that. Unless you're dealing with WiFi where extreme amounts of packet aggregation are necessary to get close to the theoretical throughput.
- brianwawok 11y agoWell if you use smart batching ala ( http://mechanical-sympathy.blogspot.com/2011/10/smart-batching.html http://mechanical-sympathy.blogspot.com/2011/10/smart-batchi... ), you don't really have the same kind of hit that you would with Nagle's or a more simple minded batching algo.
- StillBored 11y agoNUMA pin the DMA buffers too. I didn't see that in the article. With the app I worked on getting the PCI bus/CPU/memory/interrupt all pinned to the "right" node gave us a good 30% performance improvement. Put another way, identify which socket the PCIe bridge and device are attached to, then make sure the DMA buffers, and interrupt handler are pinned to that node. Also, don't touch the buffers from the other socket. Strangely enough touching the DMA buffers from the other node just once was enough to cause a performance fall off.
- monocasa 11y agoProbably because the cache line changes from Exclusive to Owned in MOESI.
- StillBored 11y agoYah, that is the obvious choice, but it wasn't as clear cut. Especially with the behavior we were seeing where the performance problems would remain long after the lines were flushed from the CPU caches. I spent a fair amount of time looking at Intel docs on snoop filters and the like and never really satisfied myself that I knew exactly what was causing the problem.
- wyldfire 11y agoYeah, with previous generations of chipsets/CPUs, I couldn't saturate quad-10GbE receive streams unless I bound the buffers to particular nodes and IRQs to corresponding cores.
- hpcadmin007 11y ago10 GbE is often good enough for most use-cases. For same datacenter same-floor gear and if there is a pressing requirement for low latency, deploying 1-2 generations old Infiniband (IB) gear is likely a smarter approach because the cost point approaches 10 GbE and the latency is sub-microsecond-scale (0.5-0.7 us vs 20-30 for 10 GbE, which is a factor of 40-60x). Deploying OFED can be a pain unless using a distro like Rocks, don't mix-match different vendors' gear (stick to all Mellanox or all QLogic) and be sure the specific bios versions of everything are supported. For Linux Rocks clusters support / deployment, we've used the cool & helpful folks over at StackIQ (was Cluster Corp).