4 ms·
I feel like this article misses why people avoid TCP like the plague, and why people use UDP for many applications 1. Routers do all kinds of terrible things w
by 20k 2y ago
I feel like this article misses why people avoid TCP like the plague, and why people use UDP for many applications
1. Routers do all kinds of terrible things with TCP, causing high latency, and poor performance. Routers do not do this to nearly the same extent with UDP
2. Operating systems have a tendency to buffer for high lengths of time, resulting in very poor performance due to high latency. TCP is often seriously unusable for deployment on a random clients default setup. Getting caught out by Nagle is a classic mistake, its one of the first things to look for in a project suffering from tcp issues
3. TCP is stream based, which I don't think has ever been what I want. You have to reimplement your own protocol on top of TCP anyway to introduce message frames
4. The model of network failures that TCP works well for is a bit naive, network failures tend to cluster together making the reliability guarantees not that useful a lot of the time. Failures don't tend to be statistically independent, and your connection will drop requiring you to start again anyway
5. TCP's backoff model on packet failures is both incredibly aggressive, and mismatched for a flaky physical layer. Even a tiny % of packet loss can make your performance unusable, to the point where the concept of using TCP is completely unworkable
Its also worth noting that people use "unreliable" to mean UDP for its promptness guarantees, because reliable = TCP, and unreliable = UDP
QUIC and TCP actively don't meet the needs of certain applications - its worth examining a use case that's kind of glossed over in the article: Videogames
I think this article misses the point strongly here by ignoring this kind of use case, because in many domains you have a performance and fault model that are simply not well matched by a protocol like TCP or QUIC. None of the features on the protocol list are things that you especially need or even can implement for videogames (you really want to encrypt player positions?). In a game, your update rate might be 1KB/s - absolutely tiny. If more than N packets get dropped - under TCP or UDP (or quic) - because games are a hard realtime system you're screwed, and there's nothing you can do about it no matter what protocol you're using. If you use QUIC, the server will attempt to send the packet again which.... is completely pointless, and now you're stuck waiting for potentially a whole queue of packets to send if your network hiccups for a second, with presumably whatever congestion control QUIC implements, so your game lags even more once your network recovers. Ick! Should we have a separate queue for every packet?
Videogame networking protocols are built to tolerate the loss of a certain number of packets within a certain timeframe (eg 1 every 200ms), and this system has to be extremely tightly integrated into the game architecture to maintain your hard realtime guarantees. Adding quic is just overhead, because the reliability that QUIC provides, and the reliability that games need, are not the same kind of reliability
Congestion in a videogame with low bandwidths is extremely unlikely. The issue is that network protocols have no way to know if a dropped packet is because of congestion, or because of a flaky underlying connection. Videogames assume a priori that you do not have congestion (otherwise your game is unplayable), so all recoverable networking failures are 1 off transient network failures of less than a handful of packets by definition. When you drop a packet in a videogame, the server may increase its update rate to catch you up via time dilation, rather than in a protocol like TCP/QUIC which will reduce its update rate. A well designed game built on UDP tolerates a slightly flakey connection. If you use TCP or QUIC, you'll run into problems. QUIC isn't terrible, but its not good for this kind of application, and we shouldn't pretend its fine
For more information about a good game networking system, see this video: https://www.youtube.com/watch?v=odSBJ49rzDo https://www.youtube.com/watch?v=odSBJ49rzDo, and it goes over pretty in detail why you shouldn't use something like QUIC
- lxgr 2y ago> 1. Routers do all kinds of terrible things with TCP, causing high latency, and poor performance. Routers do not do this to nearly the same extent with UDP What things are you thinking of here? > 2. Operating systems have a tendency to buffer for high lengths of time, resulting in very poor performance due to high latency. Do they? I can only think of Nagle's algorithm causing any potential buffering delay on the OS level, and you can deactivate that via TCP_NODELAY on most OSes. > 5. TCP's backoff model on packet failures is both incredibly aggressive, and mismatched for a flaky physical layer. Even a tiny % of packet loss can make your performance unusable, to the point where the concept of using TCP is completely unworkable That's a property of your specific TCP implementation's congestion control algorithm, nothing inherent to TCP. TCP BBR is quite resistant to non-congestion-induced packet loss, for example [1]. > If you use QUIC, the server will attempt to send the packet again which.... is completely pointless Isn't one of QUIC's features that you can cancel pending streams, which explicitly solves that issue? In your implementation, if you send update type x for frame n+1, you could just cancel all pending updates of type x if every update contains your complete new state. [1] https://atoonk.medium.com/tcp-bbr-exploring-tcp-congestion-control-84c9c11dc3a9 https://atoonk.medium.com/tcp-bbr-exploring-tcp-congestion-c...
- bvrmn 2y ago> 3. TCP is stream based, which I don't think has ever been what I want. You have to reimplement your own protocol on top of TCP anyway to introduce message frames I'm always curious where frame proponents are going to buffer in-progress frames. OS? I have huge memory concerns. Also take note you want to drop in-order requirement and it means multiple in-progress frames. It's a very nice DoS vector.
- kwindla 2y agoI've been having this "you don't want to use TCP" conversation a lot lately with people who are building real-time voice + LLM applications. Almost everybody who hacks together a voice + LLM prototype starts with WebSockets. Which I totally understand. WebSockets seem like they should do what you want. And WebSockets do work fine while you're testing. But WebSockets are TCP, so when you roll things out to real-world users, latency is higher than it should be a lot of the time. Connections randomly drop. You have to try to figure out why different OS configurations are behaving differently. You start building your own keepalive logic. You start trying to figure out how to build metrics and monitoring for audio-over-WebSockets. The answer is to use UDP, and use a protocol built on top of UDP designed for real-time media (WebRTC). It's been interesting to me that I've had this exact same conversation with several dozen engineers over the past few months. It's a good reminder that things that seem obvious when you've been doing something a long time aren't obvious to people new to a domain. Even if they are very experienced in some adjacent domain. (In this case, mobile app and web app development.) It's made me think about where my knowledge boundaries are and what "obvious" things I'm ignorant about. (Lots of them, I'm sure.) If you're interested in WebSockets, WebRTC, and why UDP is the right low-level approach for real-time media, I wrote a a primer about that a few months ago here: https://www.daily.co/blog/how-to-talk-to-an-llm-with-your-voice/ https://www.daily.co/blog/how-to-talk-to-an-llm-with-your-vo...