10 ms·
Can QUIC match TCP’s computational efficiency?
- enitihas 7y agoDisabling TCP offload to hardware makes this comparison much less useful. TCP has hardware offload right now, and QUIC will take years to get at that level.
- drewg123 7y agoIndeed. Given that the protocol itself is encrypted, doing things like TCP Large Send (TSO) in hardware gets much trickier.
- emmericp 7y agoIsn't a large part of the performance gain from TSO offloading on transmit due to avoiding working with way too many skb's in the kernel? Shouldn't GSO for UDP with proper NIC support give you most of the performance gain? I'm not sure about the implementation details of GSO and TSO in Linux. But I know a thing or two about the hardware: the Intel Niantic/82599 NICs support UDP transmit segmentation offloading, I've checked the datasheet real quick and it says "Note that current UDP segmentation offload is not supported by any standard OS." Not sure if this is just an outdated comment in a 10 year old datasheet or if there's some performance to be gained by using this UDP segmentation offload for quick. Edit: Should have clicked on the link in the article [1] before posting here, UDP segementation offloading is coming to lots of NIC drivers in 5.5. Always interesting how the hardware often supports way more features than the drivers expose and you suddenly get new hardware features on a 10+ year old NIC model :) (Similar story with IPsec offloading on the Niantic/82599) [1] https://kernelnewbies.org/Linux_5.5#Networking-1 https://kernelnewbies.org/Linux_5.5#Networking-1
- richardwhiuk 7y agoTSO and GSO improve network bandwidth, mainly.
- telotortium 7y agoHow much does TCP offload help in practice, in particular for TLS connections? Is it a 20% improvement, a 2X improvement, etc.? That would be good to know to evaluate the performance of QUIC in the article.
- unethical_ban 7y agoI don't understand how TLS makes a difference. They are different layers of the stack.
- gok 7y agoI think the question is: given that you're going to be performing pretty heavy symmetric encryption on every byte anyway, how much can TCP offload really help when all it's handling is the relatively trivial checksum and sliding window computation?
- geocar 7y agoHaving 40Gb/sec TCP is great but if you can't come up with 40Gb/sec to put into it, "what's the point?" Of course, you could just buy a TLS accelerator if you wanted one.
- ComputerGuru 7y agoCPU utilization is a zero-sum game. If you need to use the CPU to calculate checksums or do packet segmentation then it’s that much less CPU available for TLS.
- alfalfasprout 7y agoSo not quite... a lot of modern CPUs have crypto primitives implemented in hardware that allow other instructions to run simultaneously on the same core (taking advantage of hyperthreading and out of order execution).
- geocar 7y agoThat's one way to look at it. Another way is if it takes a couple CPUs to run my application, a CPU or two to drive the storage, and a CPU to do TLS, why would I also spend a CPU on TCP if my network card already has one that does that? That's a little bit like what programmers are talking about when they are talking about "multi-core programming", except those are general-purpose "cores", and (presumably) these "cores" can only handle doing TCP.
- PudgePacket 7y agoIf I understand correctly the whole point of QUIC is to not be in hardware level. That's why QUIC exists, because TCP can't be changed because it is baked in at such low, fundamental levels.
- oneplane 7y agoNo, the point of QUIC is to shave a few % of traffic off of the loads the likes of FAANG have, saving millions, while having a marginal benefit to pretty much everyone else. QUIC can be 'a bit better' in 'some scenarios', but most of the time it is totally irrelevant, except for the fact that it requires TLS 1.3 which is a good requirement overall.
- jeffbee 7y agoQUIC in the datacenter is hugely superior to Linux kernel TCP. Kernel TCP has been inappropriate for datacenter networking for decades and this has been a known issue mentioned many times in the literature (start with [1] if this is new to you). Datacenter operators have been forced to either patch the kernel or bypass it to get decent TCP performance. QUIC is a rational response to this situation. Instead of waiting a quarter century or more for Linux to get its TCP stack in order, switch to UDP and innovate in user space. 1: https://www.cs.cmu.edu/~dga/papers/incast-sigcomm2009.pdf https://www.cs.cmu.edu/~dga/papers/incast-sigcomm2009.pdf
- derefr 7y agoAre you talking about TCP over datacenter hardware Ethernet, or TCP over a software-defined network (potentially terminated on the same host)?
- jeffbee 7y agoNot sure these are opposing topics. My understanding of SDN is simply that a program may decide to drop certain frames, and this can run on top of ordinary Ethernet.
- toast0 7y agoI'm much less worried about tcp segmentation offload being disabled, than a single core, single threaded test. QUIC is going to hurt performance the most on multicore, multithreaded loads, where with TCP, you're getting the NIC to hash packets and deliver them to per-cpu rxqueues, and (hopefully) avoiding a lot of cross-cpu locking. That's going to be painful on QUIC unless/until you can get NICs to hash on the connection ids.
- jeffbee 7y agoHaving a receive queue on each CPU is usually counterproductive in my experience ... one per longest-latency-cache domain is better in most cases. Anyway these days many NICs multiqueue resources have been outstripped by the number of cores in the box. We have NICs with 32 or 64 queues in boxes having 88 or 128 CPUs. RFS flow tables are installed by sendmsg and recvmsg, so I fail to see why RFS doesn't benefit UDP just as much as TCP. If you are using RFS, your QUIC packets should still arrive on the correct CPU.
- derefr 7y ago> We have NICs with 32 or 64 queues in boxes having 88 or 128 CPUs. Doesn't that mean that you should have more NICs? From my recollection, the main thing that's made a computer a server or mainframe for the last 50 years has been that such systems have enough IO channels to saturate the CPU (i.e. they're intended to be CPU-bound, not IO-bound, under heavily-concurrent workloads.) If you take all the IO-offload cards out of a mainframe, it's just a regular computer.
- xxpor 7y agoPCIe Gen 3 is becoming a real problem here. There's only so many lanes you get per socket, it's completely possible today to end up being IO bound just due to PCIe bandwidth, not network speeds or CPU processing power. For example, this Cascade Lake Xeon has 48 lanes of PCIe gen 3: https://ark.intel.com/content/www/us/en/ark/compare.html?productIds=192458 https://ark.intel.com/content/www/us/en/ark/compare.html?pro... If you give yourself 32 of 48 lanes for networking, that's only 256 gbits of PCIe bandwidth. So each 16x slot is limited to a 100 gbit card. And that's a 22 core processor. With DPDK, coming up with a application that could process 200 gbit is almost trivial.
- petethepig 7y agoIs that common in consumer or even server hardware though?
- danudey 7y agoYes, quite. The i219 chipset in my laptop "includes advanced interrupt handling features to reduce CPU overheard. Other performance-enhancing features include offloading TCP/UDP (for both IPv4 and IPv6) checksum calculations and performing TCP segmentation. Advanced features such as Jumbo Frame support for extra-large packets and Receive Side Scaling (RSS) are also supported." It's also supported on the old chipset from the motherboard I bought more than half a decade ago, and every chipset I've used in the server space in as long as I can remember.
- alfalfasprout 7y agoWith Intel DPDK now in a fairly good state, it's actually fairly possible to get very attractive performance in userspace. You can leverage CPU instructions for crypto primitives when available to speed things up too. The amount of TCP hardware offload in linux right now is actually fairly limited. Kernel developers have largely rejected true in-hardware TCP stacks for a while now (although you can get this functionality with high end NICs and custom kernel builds). That said, since QUIC generally reduces the ack frequency it should be possible to match or exceed TCP with hardware offload. Especially when you look at total throughput with many streams open.
- aduitsis 7y agoAll well and good, but TCP has been around for decades and there are top quality network stacks providing TCP sockets on top of almost anything that can run on a CPU. Is there a (preferably BSD-licensed) QUIC piece of code that can be retrofitted where the TCP stack of an OS used to be? Or are we going to start linking a piece of the network stack into the applications?
- jabedude 7y ago>Is there a (preferably BSD-licensed) QUIC piece of code that can be retrofitted where the TCP stack of an OS used to be? No, and that's a current drawback of QUIC. But of course it's just a current drawback - the investment cost of integrating this into the operating system vice applications will certainly be eclipsed by the benefits as QUIC is deployed more throughout the internet
- PudgePacket 7y agoIt's intentionally userspace to avoid ossification.
- simias 7y agoWhat you call "ossification" others would call "stabilization" and "standardization". Not to say that the inertia of (some) kernels is not excessive sometimes but I'm not a fan of the "fuck it, we'll do it in userland" attitude. That creates big, opaque application blobs that nobody understands and knows how to debug with standard tools. I feel like Google in particular loves to do that stuff because while they control the internet and the browser, they don't (yet) control the OS so they have a very strong incentive to push stuff up in order to retain control. They don't have to ask anybody to add things into Chrome, they have to get third parties on board to extend the kernels (outside of Android at least).
- toast0 7y agoWhat's extra frustrating is that Google does control the OS for Android, but they don't use that to make TCP better. Things they could do: a) enable MTU blackhole probing, so people behind dumb networks can talk to servers that don't artificially reduce MSS. Apple has a pretty agressive implementation of MTU probing, and it's very effective. b) put in hooks for userspace congestion control. Make sending faster (or not) with updates from Google Play services (or whatever) c) the article mentions sending fewer acks. That could probably be done in TCP too --- it makes sense to make that a setting that they could roll out. d) some sort of name and shame program for bad networks / bad network devices. It's 2020 and people running PPPoE and sending MSS 1480 when it's really 1472 should be ashamed of themselves, but big companies just set their servers to MSS 1460 and call it a day. :(
- wmf 7y agoIntel Core m3-6Y30 ... USB Gigabit-Ethernet adapter Danger, non-server hardware detected. we limited the CPU’s clock from 2.2GHz down to 400MHz Danger danger, system imbalance approaching critical levels; please evacuate to a safe distance.
- emmericp 7y agoThe hardware selection and test setup is questionable, yeah. But downclocking the CPU is pretty standard for many benchmarks like that. Would have been better to evaluate the effects of CPU frequency changes separately. But there's nothing wrong with downclocking in general if you want to evaluate effects of CPU bottlenecks when you don't have high-speed links available.
- wmf 7y agoThanks to the cloud, everyone has access to 10G/25G/100G. It shouldn't be taken for granted that a CPU bottleneck exists, although from Netflix's work we know that serving >100G from one server is CPU-intensive.
- dirtydroog 7y agoBut is there any point it profiling on a laptop? Thermal throttling is a bigger issue than on a desktop, where even there you have to faff about with C states if you want somewhat believable stats.
- deleted 7y ago[deleted]
- vardump 7y agoIf the intention was to artificially reduce performance for testing their code, it would have made sense also to run it with just memory module (50% bandwidth loss) and to underclock memory.
- bullen 7y agoEverything past HTTP 1.1 has been a waste of time and energy. Everyone involved in making these protocols (HTTPS/WebSocket/HTTP2/HTTP3/etc.) are just victims of: “It is difficult to get a man to understand something, when his salary depends upon his not understanding it!” - Upton Sinclair
- AlEinstein 7y agoHave fun in your HTTP 1.1 world.
- gear54rus 7y agoBold claim. Let me guess: it's not gonna be backed up is it?
- vardump 7y agoHTTP 1.1 is somewhat broken. For example pipelining and keeping connection open, data integrity issues like no content length followed by unexpected connection loss, etc. It also allows ISP shenanigans like content modification to add unwanted advertising and unlimited tracking, even spying. So could you please explain your reasoning?
- acdha 7y agoCan you back that up? HTTPS is a requirement for doing anything securely. It seems like a lazy troll to just dismiss it with no explanation. Similarly, even a cursory amount of research into WebSockets or HTTP/2 will turn up real engineering problems which are solved by those protocols. Can you address any of them?
- bullen 7y agoI hash the password with a one-time server salt in the client (instead of HTTPS) and I use simple HTTP comet-stream for real-time data (instead of WebSockets): http://fuse.rupy.se http://fuse.rupy.se HTTP/2 has the TCP head of line problem which makes it completely useless. HTTP/3 is trying to be more nimble, but it's a protocol! Ossification is a _feature_ of protocols; for TCP it allows the backbone to evolve without everyone re-implementing everything over and over again at the edges! The solution is to make TCP like UDP instead, by allowing sockets to not resend missed packets. Another real solution is to remove the network kernel on linux.
- kev009 7y agoTheir implementation of congestion control (reno) is 85 lines including a big copyright block comment. That's cute but it's completely incomparable to the transmission intelligence in the Linux TCP stack, especially for the line of business Fastly is engaged in (maybe the technical staff doesn't even know what that is due to poor leadership?) in trying to get goodput to devices across congested and channel lossly links. This article is highly comical. I've yet to be impressed by fastly aside from offloading itself to retail investors with another nonsensical IPO.
- vardump 7y agoThey use Reno? I don't know what's their business, but hopefully it doesn't involve lossy links like mobile data, which often drop packets even when uncongested... Reno can work nicely for low packet loss links, though.
- kev009 7y agoJust pertinent to this article, take a look at the links to their QUIC implementation. The reason I bring this up is because it's pretty easy to make a simplified fast path TCP stack that doesn't have all the branch heavy code necessary to do what the Linux TCP stack does well. So this is like a Phoronix style "benchmark". An undergrad level of understanding of systems will net you that you can get equal performance in userspace or kernel space with sufficient holistic thinking and accounting of data movement. I would caution that very few people are qualified to do this in userspace though, which is paradoxical because you will find a lot more people showing up claiming they know how. So basically this isn't a competent test; it isn't the production grade QUIC from google, it's a toy and the result is complete nonsense.
- vardump 7y agoYeah, looked at their code [0]. You're right. I'm no TCP (or even congestion control) expert, but I wouldn't have needed to test their code in real life to tell exponentially reducing send window on a lost packet is just not going work in real life in 2020. They should have at least tested their code with simulated jittery latencies and packet loss typical for modern 4G networks. I bet Linux TCP would have left their implementation in dust. Another thing that caught my eye was their assumption that MTU is same on return path... probably often true, but definitely wouldn't count on it! Huge respect for them for putting their necks out and releasing code! Maybe this is their intention as well, to gain understanding through criticism. Also, perhaps this was just their MVP? It would certainly explain weird choice for congestion control. [0]: https://github.com/h2o/quicly/blob/master/lib/cc-reno.c https://github.com/h2o/quicly/blob/master/lib/cc-reno.c
- campuscodi 7y agoShouldn't we be waiting for QUIC to pass IETF before we go diving into performance benchmarks?
- wmf 7y agoGiven that performance is one of the motivations for QUIC, benchmarking needs to be done at every stage of development. Also, QUIC is close to being finalized so I don't think it's too premature.