6 ms·
Linux 6.8 Network Optimizations Can Boost TCP Performance by ~40%
- cdaringe 3y agoI looked at the references url for the email patches. There were drillions of comitters and commits. How was this initiative organized?
- j16sdiz 3y agoThey came from google
- isliguezze 3y agoamazing!
- MrBuddyCasino 3y agoHuge improvement for AMD EPYC, basically nothing for Intel. Anyone got an idea why?
- jeffbee 3y agoProbably the "machine of the day" for google websearch is an EPYC so they are focused on that. Possibly the issue didn't affect Xeon in the first place, leaving nothing to optimize. <strike>You can see in the absolute values it was already faster on Intel than it is on EPYC after the improvement.</strike> actually I am cross-eyed, I thought there was another zero there. I wonder if this set of patches was the result of visibility gained from recent `perf mem` features that show loads and stores for each struct member.
- TacticalCoder 3y ago> You can see in the absolute values it was already faster on Intel than it is on EPYC after the improvement I don't know because the table shows a 100 Gbit/s NIC on AMD and a 200 Gbit/s (!) NIC on Intel. Which for some benchmarks would now make AMD twice as fast as Intel. Note sure if the 100 Gbit vs 200 Gbit labels are correct though.
- wmf 3y agoEpyc has multiple L3 caches while Xeon has a unified L3. If the same TCP socket is accessed on multiple CCXes it would cause cache lines to bounce on Epyc which is pretty costly.
- briffle 3y agoI believe the new Xeon SP processors coming out are also based on the 'chiplet' design like the Epyc ones.
- wmf 3y agoThey are but Intel implements chiplets differently so they still have a single unified L3 cache.
- charleshn 3y agoOn that subject, it's worth knowing that with the default BIOS settings the whole socket shows up as a single NUMA node, which means that most workloads don't benefit from automatic NUMA balancing or manual NUMA-aware placement within a socket. One needs to set NPS "Nodes per socket" to e.g. 4 to have one NUMA node per quadrant, and potentially also "L3 cache as NUMA" to have one per CCX [0]. [0] https://www.amd.com/content/dam/amd/en/documents/epyc-technical-docs/tuning-guides/58002_amd-epyc-9004-tg-hpc.pdf https://www.amd.com/content/dam/amd/en/documents/epyc-techni...
- Fornax96 3y agoThis is exactly what I needed for pixeldrain. I have been struggling with TCP bottlenecks for months. I can't wait for 6.8 to be released.
- klysm 3y agoStruggling with TCP bottlenecks or performance bottlenecks at similarly lower layers has been some of the most fun so far in my career
- Fornax96 3y agoIt's definitely a very technical challenge. I have learned a lot since I made the switch to 100 GbE. My website regularly needs to handle more than 100k connections per server, the bottlenecks become very apparent at that scale.
- renewiltord 3y agoAbsolutely wild. We don't have that many simultaneous flows. Maybe like 256 through multiple 40 G cards, but it will be good to try this once it's in-repo on a Debian.
- aetherspawn 3y agoAnyone got any idea how to build your own router with PowerEdge, AMD EPYC and idk, pfsense?
- bauruine 3y agoIf you want to go down the rabbit hole have a look at VPP. [0] It's a very fast, like 100GBit/s+ and 100 Mpps+, user space routing engine for commodity x86 hardware. Pim van Pelt has many interesting Videos [1] and articles [2] about it. I only tested it briefly before abandoning it because I'm still stuck on measly ~300/120 g.fast for the foreseeable future so I don't need it yet but especially the Linux CP [3] makes it reasonably workable if you have some Linux networking knowledge. [0]: https://fd.io/ https://fd.io/ [1]: https://video.ipng.ch/search?search=vpp&searchTarget=local https://video.ipng.ch/search?search=vpp&searchTarget=local [2]: https://ipng.ch/s/articles/ https://ipng.ch/s/articles/ [3]: https://github.com/pimvanpelt/lcpng https://github.com/pimvanpelt/lcpng
- wmf 3y agoJust install PFSense and follow the docs?
- edward28 3y agoWell pfsense won't benefit from this as it is FreeBSD based.
- speed_spread 3y agoMaybe it's already faster than Linux?
- klooney 3y agoHave people benchmarked this recently? I feel like this flamewar is still based on forum posts from 1998
- wmf 3y agoI haven't seen any recent head-to-head benchmarks but Netflix is getting really good performance from FreeBSD networking and they publish detailed information about it. It gets discussed here every six months or so. Of course networking encompasses many different workloads (e.g. routing with PFSense is going to be a different code path than serving data over HTTPS) and each OS probably has its own strengths and weaknesses.
- hedgehog 3y agoPour one out for Orinoco support. Those old Wavelan cards were the mainstay of community wireless networks in the early 2000s as well as the card Apple rebranded as the first AirPort cards.
- maybeben 3y agoOof, Aironet, too. RIP kings. Seems like they're paving the way for removal of all 16-bit PC Card support like FreeBSD.
- xattt 3y agoLater Orinoco cards were the mainstays for wardriving with Netstumbler and Kismet. End of an era.
- bradfitz 3y agoI still have one! RIP.
- aforty 3y agoOrinoco was my first experience with WiFi back in my college days. I remember it being incredibly bad, with buggy drivers, and an overall horrendous experience. That was with Windows of course.
- hedgehog 3y agoInitially the Linux drivers and wireless stack were pretty rough too, I think OpenBSD had the first open support that worked well especially for running in AP mode.
- whatwhaaaaat 3y agoYes but they existed. I recall this being a time when half the wifi cards used wpa supplicant which was a damn scourge - tho a necessary one. Orinoco cards didn’t require that at least.
- kstrauser 3y agoWhat do those numbers mean?
- malux85 3y agoI’m curious can you elaborate on the problems you are having and how these improvements will help?
- wmf 3y ago"Efficiency is computed as cpu seconds / throughput (one tcp_rr round trip)."
- kstrauser 3y agoSorry, it's been a long day. My eyes kept moving past that. I wonder why they presented it that way instead of the inverse, like connections per CPU second.
- lakomen 3y agoHow does he know when 6.7 was just released?
- CaliforniaKarl 3y agoIt's the way the kernel-development process works: As soon as a release takes place, there's a period (the "merge window") where new things can be merged in for the next release. See https://www.kernel.org/doc/html/latest/process/2.Process.html https://www.kernel.org/doc/html/latest/process/2.Process.htm... for more details. To make merges go smoother, there's a set of trees that are designed for developers to stage things that are ready to merge during the next merge window. linux-next is a good example: https://www.kernel.org/doc/man-pages/linux-next.html https://www.kernel.org/doc/man-pages/linux-next.html The Linux Networking folks have their own trees that fulfill the same role for their stuff: https://www.kernel.org/doc/html/next/process/maintainer-netdev.html https://www.kernel.org/doc/html/next/process/maintainer-netd...
- deleted 3y ago[deleted]
- pajko 3y agoMore expected changes could be found at https://www.phoronix.com/search/Linux+6.8 https://www.phoronix.com/search/Linux+6.8
- ashvardanian 3y agoDoes anyone know a way to run recent kernels like 6.8 on AWS?
- mrintegrity 3y agoIs it likely there would be significant performance improvements on AWS graviton based instances?
- wmf 3y agohttps://docs.aws.amazon.com/AWSEC2/latest/UserGuide/UserProvidedKernels.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/UserProv...
- ashvardanian 3y agoCool, this may actually work! Thanks!