14 ms·

Timeliness without datagrams using QUIC

224 points by wofo 2y ago

blueflow 2y ago
What should be based on datagrams: - Local discovery (DHCP, slaac, UPnP, mDNS, tinc, bittorrent) - Broadcasts (Local network streaming) - Package encapsulation (wireguard, IPSec, OpenVPN, vlan)
- promiseofbeans 2y ago
  Hey I'm sure you're right, but as someone with less low-level networking expertise, I was wondering if you could explain why you'd want datagrams for these use-cases?
  - deleted 2y ago
    [deleted]
  - swiftcoder 2y ago
    Local discovery is itself based on broadcast/multicast, both of which only work over UDP (TCP doesn't provide any mechanism for broadcast streams).
  - blueflow 2y ago
    TCP is a back & forth between two hosts and thus cant do "shouting blindly into the network" things. That rules out local discovery and media broadcast. Package encapsulation is bad over TCP because if the encapsulated data is TCP itself, you have congestion control twice. On congested networks, this results in extra slowdowns that can make the connection unusable.
- deleted 2y ago
  [deleted]
- Hikikomori 2y ago
  And games.
  - swiftcoder 2y ago
    The whole point of the article is that you don't need to go all the way down to raw datagrams to achieve the kind of low latency that is needed for things like games/VOIP/livestreaming
  - eru 2y ago
    It totally depends on the kind of game.
- ajb 2y ago
  One missing from your list is real-time media. Retransmission, or even just buffering for reordering, adds latency so it's better to take the hit of a loss with error correction or packet-loss-concealment
  - tfyoung 2y ago
    Yeah exactly. With realtime, a dropped packet is often too old by the time it's re-transmitted. It's actually harmful to wait for it when you'd rather just skip it and move on and stay as close to realtime as possible. It's a very different usecase from most uses. I do wish QUIC allowed carrying streams that were useful for realtime in conjunction with allowing reliable streams. Using MPEG-TS over SRT to have it just spam metadata to handle the unreliableness is janky. It would be far nicer to have a reliable stream for metadata, then an unreliable one for realtime streaming.
forrestthewoods 2y ago
It mentions that the video game industry uses UDP but then fails to further address that use case. So, should competitive shooter video games switch to QUIC? Is that even supported across all the various gaming platforms?
- swiftcoder 2y ago
  > It mentions that the video game industry uses UDP but then fails to further address that use case Video games tend to use UDP for the same reason everyone else mentioned does: timeliness. You want the most recent position of the various game objects now, and you don't give a shit about where they were 100ms ago. The proposed solution of segmenting data into QUIC streams and mucking with priorities should work just fine for a game. > Is that even supported across all the various gaming platforms? QUIC itself is implemented in terms of datagrams, so if you have datagrams, you can have QUIC.
  - Tuna-Fish 2y ago
    But the proposed solution provides no value over datagrams, while it has plenty of downsides. Let me rephrase the problem here for a simple FPS game: The entire world state (or, the subset that the client is supposed to know about) is provided in each packet sent from the server. The last few actions with timestamps are in each packet sent from the client. You always want to have the latest of each, with lowest possible latency. Both sides send one packet per tick. You do not want retransmission based on the knowledge that a packet was lost (the RTT is way too long for the information that a packet was lost to ever be useful, you just retransmit everything every tick), you do not want congestion control (total bandwidth is negligible, and if there is too much packet loss to maintain what is required, there is no possible solution to maintain sufficient performance and you shouldn't even try), and none of the other features talked about in the post add anything of value, either. It reads like someone really likes QUIC, it fit well into their problems, and they are a bit too enthusiastic about evangelizing it.
    - swiftcoder 2y ago
      In practice most games also need reliable side channels for control messages/chat/etc, and so they end up building optional reliable streams over UDP... and at the end of this path lies something that looks a lot like a (less thoroughly designed/tested) version of QUIC
promiseofbeans 2y ago
We use straight UDP datagrams for streaming high-frequency sensor data. One of our R&D people built a new system that uses quic and solves most of our problems with out-of-order delivery. We still use datagrams over UDP for everything because we have to support some 3rd party sensors out of the box without adapters, and UDP is all they can do.
- jcelerier 2y ago
  Also pretty much every media art system in the world just uses OSC over UDP
  - jesprenj 2y ago
    And ArtNet also over UDP.
- Almondsetat 2y ago
  Yeah, high frequency sensor data was on my mind too. Do you have any figures on the power saved wrt TCP?
  - Joel_Mckay 2y ago
    In general, the wireless handshake on G5/LTE/Wifi/Starlink will dwarf the traffic of sensors for power use. For old-style satellite mailboxes, the traffic is better batched in 45min chunks anyway (the 5W transmitters are usually only active for well under 20 seconds per message, and use a super-cap to handle the RF power draw spike.) Naive UDP protocols have reachability problems in numerous scenarios, variable costs can balloon (concurrency with polling links is dumb), and can open a hole in your infrastructure at scale without special equipment mods. Only if you have a tier 3 or better WAN trunk or cloud-center should you even consider something silly like QUIC. YMMV, and I love the AstroTurf on YC... lol ;)
- songbird23 2y ago
  what language was the system build with? thats super cool!
asdefghyk 2y ago
Maybe ... never use on a congested network. or Never use on a network where congestion is above a certain level or Never us on a network where this parameter is above a certain level - like network latency or only use on a LAN not a WAN ....?
- swiftcoder 2y ago
  Unless you are building software for a pre-established intranet, how do you predict ahead of time whether your software will run on congested networks? Most end-user software ends up running on a variety of oversubscribed wifi and cell networks
  - smcameron 2y ago
    > how do you predict ahead of time whether your software will run on congested networks? One thing you can do is build in "bad network emulation" into your software, allowing the various pieces to drop packets, delay packets, reorder packets, etc. and make sure it still behaves reasonably with all this garbage turned up.
    - asdefghyk 2y ago
      RE ".... build in "bad network emulation" into your software....." I expect there would be network emulators that would be able to emulate these situations. How good these emulators are would be the next question.
  - marcosdumay 2y ago
    You tell the user "hey, don't use on congested networks". And yeah, since most internet software has this exact disclaimer, it works kinda well. But people do appreciate software that doesn't have this restriction, and even go out of their way to talk about it (more on mobile networks than on wired connections).
- ot 2y ago
  LAN can be oversubscribed too, even if your switches have sufficient capacity you can have short packet bursts that fill up the queues. You cannot assume reliable delivery.
FpUser 2y ago
I use UDP for my desktop game like software with clients all over the world to propagate anonymized state. It needs no encryption as the transmitted data has zero value to any third party. It needs no reliability since dropped packets would be handled by predictive filter with practical reliability that is way more than needed. So why the F.. would I bother with anything else?
- aarmot 2y ago
  Because author of the article likes to be edgy and parrot the deep truth he learned yesterday - that TCP is "reliable" and UDP is not. The reality, of course, is somewhat different and muddy. For example, if you have network outage following later by a software crash or a reboot, then all the TCP buffer worth of data (several kilobytes or upto some megabytes - depends on your tuning) is mercilessly dropped. And your application thinks that just because you used TCP the data must have been reliably delieved. To combat this you have to implement some kind of serializing and acking - but the article scoffs us that we are too dumb to implement anything besides basic stream /s I'm not arguing that TCP is useless, just that UDP has its place and we - mere mortals - can use is too. Where appropriate.
  - Lerc 2y ago
    The thing is, when you need that low latency, you don't want most of the features of more robust streams. When you are using UDP the correct way to handle out of order delivery is to just ignore the older packet. It's old and consequently out of date. Figuring out how to solve any mess caused by mistransmission necessarily has to be done at the application level because that's where the most up to date data is.
  - ben0x539 2y ago
    The author (disclaimer: former coworker) has likely learned about UDP before yesterday because he's been complaining about TCP for years.
  - vitus 2y ago
    > Because author of the article likes to be edgy and parrot the deep truth he learned yesterday I will point out that "author of the article" is one of the core contributors in the IETF Media-over-QUIC working group (which is an effort to standardize how one might build these real-time applications over QUIC) and has been working in the real-time media protocols space for 10+ years. The author recognizes that the title is clickbait, but the key point of the article is that you probably don't want to use raw UDP in most cases. Not that UDP is inherently bad (otherwise he wouldn't be working on improving QUIC).
ggm 2y ago
Quic is implemented over UDP. It's literally running over datagrams.
- ot 2y ago
  Everything at some layer has to run over datagrams, since that's what IP is. This is literally the point of the article: if you want to create a protocol over raw datagrams, you have to implement a lot of things that are very hard to get right, so you should just use QUIC instead, which does them for you.
  - ggm 2y ago
    "You" is doing all the hard work. Apps should use reliable sessions and transport, almost all the time? No disagree. The exceptions are understood. But don't pretend the substrate is that reliable data stream. "We" have to construct it almost always. We do reliable for you, over UDP. I don't do this stuff any more, but I worked on OSI transport and remote operations service mapped to UDP and other protocols back in the 80s
PhilipRoman 2y ago
IMO stream abstractions make it too convenient to write fragile programs which are slow to recover from disconnections (if they do at all) and generally place too many restrictions on the transport layer. Congestion control is definitely needed but everything else seems questionable. In a datagram-first world we would have no issue bonding any number of data links with very high efficiency or seamlessly roaming across network boundaries without dropping connections. Many types of applications can handle out-of-order frames with zero overhead and would work much faster if written for the UDP model.
- CJefferson 2y ago
  Out of interest, what kind of applications are you thinking of? What systems are common written using TCP, that could switch to UDP? Clearly websites, audio and video generally don't work with out-of-order frames -- most people don't want dropped audio and video. Some video games are happy to ignore missed packets, but when they can they are already written in UDP.
  - ansgri 2y ago
    Any data block that must be fully received before processing comes to mind (e.g. websites’ html), just request retransmission of parts that didn’t make it in first pass. Funnily enough it’s (realtime) video and audio that already uses UDP due to preferring timely data to complete data. On the contrary, for me it’s hard to imagine video game with missed packets as the state can get out of sync too easily, you’ll need eventual consistency via retransmission or some clever data structures (I know least about this domain though)
    - smcameron 2y ago
      I'm not expert in this area, but my understanding is that for latency sensitive video games, on the client side, state is predicted between updates, e.g. the state may be updated via the network, at say, roughly 10Hz (probably faster, but probably less than 60Hz), but updated locally at 60Hz via interpolation, dead reckoning and other predictive/smoothing heuristics, a few missed updates just means a bit more prediction is used. "State" is not usually transmitted as one lump "frame" at a time, but rather per-game unit (per spaceship, asteroid, robot, or whatever is in the game) or per some-small-group of units. When some updates are delayed too long, you might get some visible artifacts, "rubber-banding", "teleportation" or "warping", etc. often lumped together by players under the umbrella term "lag". For out-of-order packets, older packets might be dropped as newer packets may already have been applied to the state of some game units (typically there's some timestamps or monotonically increasing counter associated with state updates for each unit used by the prediction/interpolation/smoothing heuristics) and the state is usually represented in absolute terms rather than relative terms (e.g. (x,y) = (100,100), rather than x+=10, y+=10, so that any update may be applied in isolation.)
bitcharmer 2y ago
This is a narrow way of looking at UDP applications. The whole HFT, low-latency fintech world is built on top of datagrams. Using TCP would be pure the worst choice possible.
- AtlasBarfed 2y ago
  Is that because they mostly use dedicated links and intranets? A lot of Enterprise messaging is based on UDP, I think on the presumption that corporate networks are just simply going to be a lot more reliable
  - bitcharmer 2y ago
    You are correct, it's either point-to-point, like direct cross-connects with the exchange or it's a relatively small intranet. Additionally, an overwhelming majority of packets flying around are market data. You don't want to re-transmit lost ones because by the time you do, the data is already outdated.
adunk 2y ago
This may seem like a minor nit, but I think there is a problem with using the term "unreliable" to describe UDP. The more commonly used term, and IMHO better term, is "best-effort" [1]. UDP makes its best effort to deliver the datagrams, but the datagrams may be dropped anyway. But it does not make UDP inherently unreliable. [1] https://en.wikipedia.org/wiki/Best-effort_delivery https://en.wikipedia.org/wiki/Best-effort_delivery
- vitus 2y ago
  IMO "best-effort" is euphemistic and confusing to people outside of this space, even (especially?) to native English speakers. (I would probably have described UDP as "reasonable effort" and TCP as "best effort", if not for the existing terminology.) I recall my computer networking professor describing this as "Best-effort means never having to say you're sorry". In practice, best-effort does not mean you try your hardest to make sure the message gets from A to B, it means that you made an effort. Router in the path was congested? Link flap leading to blackholing on the order of 50ms before fast reroute kicks in? Oh well, we tried. Meanwhile, TCP's reliable delivery will retry several times and will present an in-order data stream to the application. Reliable vs unreliable might be bad terminology, but I don't think best-effort is any better. My experience with unreliable systems is that they're great something like 95% of the time, and they're great for raw throughput, but there are many cases where that last 5% makes a huge difference.
  - eru 2y ago
    The question is 'who deals with dropped packages'? In TCP, the answer is: 'the protocol'. In UDP the answer is 'the next layer of abstraction' (eg the app or some library). You can build a 'reliable' protocol on top of UDP, and still not get TCP. Eg if you want to transfer a large file that you know up front, then TCP's streaming mechanism doesn't make too much sense. You could use something like UDP to send the whole file from A to B in little chunks once, and at the end B can tell A what (numbered) chunks she's missing. There's no reason to hold off on sending chunk n+1 of the file, just because chunk n hasn't arrived yet.
    - 2y ago
JackSlateur 2y ago
Network is nothing but datagrams.
dale_glass 2y ago
IMO one of the worst mistakes made in IP development was not having made a standard protocol for the one thing that pretty much everyone seems to want: A reliable, unlimited-length, message-based protocol. With TCP there's a million users that throw out the stream aspect and implement messages on top of it. And with UDP people implement reliability and the ability to transmit >1 MTU. So much time wasted reinventing the wheel.
- supriyo-biswas 2y ago
  There’s Homa[1]. [1] https://github.com/PlatformLab/Homa https://github.com/PlatformLab/Homa
- H8crilA 2y ago
  Sorry for a useless comment but now I feel so stupid for never noticing that pretty much everything done on TCP is actually message exchange, not streams. The mismatch is indeed so unfortunate.
  - eru 2y ago
    Well, you often have some dependency between messages. Some re-orderings don't matter, but some do. If you just keep everything in the same order, then you never have to worry about any re-ordering. I can see why people picked streams as the one-size-fits-all-(but-badly) abstraction.
    - tliltocatl 2y ago
      Because worse is better and if you try to be everything you (don't) get OSI. Also because everyone loves serial ports and dialup modems and TCP just emulates a serial port over packet network. And it wasn't supposed to be one-size-fits-all, rather «let us have telnet while we figuring out the rest» but then ossification happend.
  - iainmerrick 2y ago
    Is it really that unfortunate? What’s the big downside? There’s a bit of reinvention needed, sure, but adding a framing layer is about the easiest task in networking. If you want a standard way of doing it, these days you can use a WebSocket. Edit to add: oh, I see, you want reliable but unordered messages. That would definitely be useful sometimes, but other times you do want ordering. If you don’t need ordering, isn’t that pretty much what QUIC does?
thomashabets2 2y ago
> The bytes within each stream are ordered, reliable, and can be any size; it’s nice and convenient. Each stream could be a video frame […] But you can tell the QUIC stack to focus on delivering important streams first. The low priority streams will be starved, and can be closed to avoid wasting bandwidth. Is the author saying that with QUIC I can send a "score update" for my game (periodic update) on a short-lived stream, and prevent retransmissions? I'll send an updated "score update" in a few seconds, so if the first one got lost, then I don't want it to waste bandwidth retransmitting. Especially I don't want it retransmitted after I've sent a newer update.
- gnfargbl 2y ago
  I think you're looking for RFC 9221, "An Unreliable Datagram Extension to QUIC."
  - swiftcoder 2y ago
    You don't need datagrams for this. You can set the stream to discard itself as soon as packet loss is detected
    - crote 2y ago
      That doesn't quite provide the same experience, does it? For something like a scoreboard you don't want the message to be discarded once any packet loss is detected. As long as message N is the most recent one, message N should be retried until it succeeds. You just want to stop retrying N once N+1 becomes available - or ideally even successfully delivered.
      - swiftcoder 2y ago
        In a game context, where we're spewing state updates out 10-60x per second, there is always a newer packet in flight already, so discard-on-loss produces the same result. In your scoreboard example, you probably just want an actually reliable stream for that - it's not like scores are rolling in so fast that it's beneficial to deal with it as unreliable.
mjw_byrne 2y ago
Silly clickbait title, which the author even admits up front. UDP and TCP have different behaviour and different tradeoffs, you have to understand them before choosing one for your use case. That's basically it. No need for "Never do X" gatekeeping.
- g15jv2dp 2y ago
  My apologies if the star in front of "never" was added to the title in the last ten minutes after you posted your comment. But that star is clearly there to indicate "fine print ahead"; in a title, it's pretty obvious that it means that the article is not about to "gatekeep" people out of UDP. (In fact, at the end of the article, the author suggests to use QUIC which, you guessed it, is based on UDP.)
- deleted 2y ago
  [deleted]
tliltocatl 2y ago
Most of TCP woes comes from high-bandwith latency-sensitive stuff like HFT and video, but TCP isn't particularly good for low-bandwidth high-latency networks either (e. g. NB-IoT with 10 seconds worst case RTT): - TCP will waste roundtrips on handshakes. And then some extra on MTU discovery. - TCP will keep trying to transmit data even if it's no longer useful (same issue as with real-time multimedia). - If you move into a location with worse coverage, your latency increases, but TCP will assume packet loss due to congestion and reduce bandwidth. And in general, loss-based congestion control just doesn't work at this point. - Load balancers and middleboxes (and HTTP servers, but that's another story) may disconnect you randomly because hey, you haven't responded for four seconds, you are probably no longer there anyway, right? - You can't interpret the data you've got until you have all of it - because TCP will split packets with no regards to data structure. Which is twice as sad when all of your data would actually fit in 1200 bytes.
- eru 2y ago
  And TCP wants to pretend that all your data arrives as a linear stream one after the other; and keeps you from seeing package n+1, if you haven't received package n, yet. That's useful for some things, but often you can make use of package n+1, even when package n hasn't arrived, yet. For example, when you are transferring a large file, you could use erasure encoding to just automatically deal with 5% package loss. (Or you could use a fountain code to deal with variable packet loss, and the sender just keeps sending until the receiver says "I'm done" for the whole file, instead of ack-ing individual packages. Fountain codes are how deep space probes send their data back. Latency is pretty terrible out to Jupiter or Mars.)
  - Aurornis 2y ago
    > And TCP wants to pretend that all your data arrives as a linear stream one after the other; That’s one of the reasons people use TCP. > For example, when you are transferring a large file, you could use erasure encoding to just automatically deal with 5% package loss. It’s never that simple. You can’t just add some erasure coding and have it automatically solve your problems. You now have to build an entire protocol around those packets to determine and track their order. You also need mechanisms to handle the case where packet loss exceeds what you can recover, which involves either restarting the transfer or a retransmission mechanism. The number of little details you have to handle quickly explodes in complexity. Even in the best case scenario, you’d be paying a price to handle erasure coding on one end, the extra bandwidth of the overhead, and then decoding on the receiving end. That’s a lot of complexity, engineering, debugging, and opportunities to introduce bugs, and for what gain? In the file transfer example, what would you actually gain by rolling your own entire transmission protocol with all this overhead? > Fountain codes are how deep space probes send their data back. Latency is pretty terrible out to Jupiter or Mars.) Fountain codes and even general erasure codes are not the right tool for this job. The loss of a digital packetized channel across the internet is very different than a noisy analog channel sent through space.
nabla9 2y ago
The choice is not UDP vs TCP. UDP adds minimum over raw sockets so that you don't need root privileges. Other protocols are build on top of UDP. It's better to use existing not-TCP protocols instead of UDP when the need arises instead of making your own. Especially for streaming.
ozim 2y ago
Great in depth article from what seems really a person who knows that stuff.
nyc_pizzadev 2y ago
One thing not mentioned often is that a lot of networks will drop UDP packets first when encountering congestion. The thinking is that those packets will not re-transmit, so it’s an effective means to shed excess traffic. Given we now have protocols that aggressively re-transmit on UDP, I wonder how that has changed things. I do seem to remember QUIC having re-transmit issues (vs HTTP1/2) years ago because of this.
cenriqueortiz 2y ago
Nahhh. While most of the applications/cases will be using session-based connections, there are uses for using datagrams directly — don’t be afraid. Yes, you will have to take care of many more details yourself. And as a side line, it is a great way of learning the low level aspects of networking.
- ddtaylor 2y ago
  The SteamNetworkingMessages API for game development does a good job of making this available if you want it without caring about the internals for that use case.
Anon_Admirer 2y ago
Hope this one gets captured by quackernews - can’t wait to see its description.
TuringNYC 2y ago
>> The common wisdom is: >> >> use TCP if you want reliable delivery >> >> use UDP if you want unreliable delivery >> What the *(& does that mean? Who wants unreliability? I dont agree with the premise of this article, UDP isnt for unreliability, it provides a tradeoff which trades speed and efficiency and provides best-efforts instead of guarantees. It makes sense depending on your application. For example, if I have a real-time multi-player video game, and things fall behind, the items which fell behind no longer matter because the state of the game changed. Same thing for a high-speed trading application -- I only care about the most recent market data in some circumstances, not what happened 100ms ago.
- alexey-salmin 2y ago
  That's basically what the article says if you keep on reading. It states the "common wisdom" at first to disagree with it.
  - hoseja 2y ago
    Well then it's clickbait.
- opheliate 2y ago
  That isn’t the premise of the article, that is the “common wisdom” the author corrects as the article goes on. The author goes on to list video games as an example of where UDP makes sense, as well as live video.
20k 2y ago
I feel like this article misses why people avoid TCP like the plague, and why people use UDP for many applications 1. Routers do all kinds of terrible things with TCP, causing high latency, and poor performance. Routers do not do this to nearly the same extent with UDP 2. Operating systems have a tendency to buffer for high lengths of time, resulting in very poor performance due to high latency. TCP is often seriously unusable for deployment on a random clients default setup. Getting caught out by Nagle is a classic mistake, its one of the first things to look for in a project suffering from tcp issues 3. TCP is stream based, which I don't think has ever been what I want. You have to reimplement your own protocol on top of TCP anyway to introduce message frames 4. The model of network failures that TCP works well for is a bit naive, network failures tend to cluster together making the reliability guarantees not that useful a lot of the time. Failures don't tend to be statistically independent, and your connection will drop requiring you to start again anyway 5. TCP's backoff model on packet failures is both incredibly aggressive, and mismatched for a flaky physical layer. Even a tiny % of packet loss can make your performance unusable, to the point where the concept of using TCP is completely unworkable Its also worth noting that people use "unreliable" to mean UDP for its promptness guarantees, because reliable = TCP, and unreliable = UDP QUIC and TCP actively don't meet the needs of certain applications - its worth examining a use case that's kind of glossed over in the article: Videogames I think this article misses the point strongly here by ignoring this kind of use case, because in many domains you have a performance and fault model that are simply not well matched by a protocol like TCP or QUIC. None of the features on the protocol list are things that you especially need or even can implement for videogames (you really want to encrypt player positions?). In a game, your update rate might be 1KB/s - absolutely tiny. If more than N packets get dropped - under TCP or UDP (or quic) - because games are a hard realtime system you're screwed, and there's nothing you can do about it no matter what protocol you're using. If you use QUIC, the server will attempt to send the packet again which.... is completely pointless, and now you're stuck waiting for potentially a whole queue of packets to send if your network hiccups for a second, with presumably whatever congestion control QUIC implements, so your game lags even more once your network recovers. Ick! Should we have a separate queue for every packet? Videogame networking protocols are built to tolerate the loss of a certain number of packets within a certain timeframe (eg 1 every 200ms), and this system has to be extremely tightly integrated into the game architecture to maintain your hard realtime guarantees. Adding quic is just overhead, because the reliability that QUIC provides, and the reliability that games need, are not the same kind of reliability Congestion in a videogame with low bandwidths is extremely unlikely. The issue is that network protocols have no way to know if a dropped packet is because of congestion, or because of a flaky underlying connection. Videogames assume a priori that you do not have congestion (otherwise your game is unplayable), so all recoverable networking failures are 1 off transient network failures of less than a handful of packets by definition. When you drop a packet in a videogame, the server may increase its update rate to catch you up via time dilation, rather than in a protocol like TCP/QUIC which will reduce its update rate. A well designed game built on UDP tolerates a slightly flakey connection. If you use TCP or QUIC, you'll run into problems. QUIC isn't terrible, but its not good for this kind of application, and we shouldn't pretend its fine For more information about a good game networking system, see this video: https://www.youtube.com/watch?v=odSBJ49rzDo https://www.youtube.com/watch?v=odSBJ49rzDo, and it goes over pretty in detail why you shouldn't use something like QUIC
dicroce 2y ago
I wonder if this guy thinks video GOPs should all be hundreds of frames long because P frames are so much smaller than I frames and since in his world we NEVER use an unreliable network you might as well.
deleted 2y ago
[deleted]
karmakaze 2y ago
TL;DR - use QUIC (should have just looked at the domain name)
dragonfax 2y ago
I've seen UDP used for great effect in video streaming. Especially timely video streaming such as cloud gaming. When waiting a late packet is no longer useful.
- Joel_Mckay 2y ago
  Not as popular as it once was, but it is still in use: https://en.wikipedia.org/wiki/Real-Time_Streaming_Protocol https://en.wikipedia.org/wiki/Real-Time_Streaming_Protocol Cheers =3
  - kwindla 2y ago
    RTSP is the control protocol. Some other protocol is needed for the actual audio/video streaming. That's usually RTP, these days. RTP is a core part of WebRTC, for example. When you're doing a video call in a web browser, you're using WebRTC, including RTP. In fact, this RTP-via-WebRTC is the only way to send UDP packets from JavaScript! RTSP is still used by older streaming systems and hardware ecosystems that are slow to change, such as network-connected security cameras. But in newer applications, WebRTC has mostly replaced it. Of course, the QUIC effort is in part an attempt to replace WebRTC, so the wheel continues to turn!
    - Joel_Mckay 2y ago
      WebRTC still has its own set of issues, and I found it only slightly improved over other options compiling the ARM64 port: https://github.com/mpromonet/webrtc-streamer.git https://github.com/mpromonet/webrtc-streamer.git I remain unconvinced UDP based streams will ultimately remain in the long-term, but webRTC certainly made it easier to peer a connection. ;)
deleted 2y ago
[deleted]
gary_0 2y ago
I wonder if the DiffServ[0] bits in IPv6 could be another way to prevent bufferbloat from affecting real-time datagrams? Or are they like IPv4's ToS[1] bits, which I think were never implemented widely (or properly) enough for any software to bother with? [0] https://en.wikipedia.org/wiki/Differentiated_services https://en.wikipedia.org/wiki/Differentiated_services [1] https://en.wikipedia.org/wiki/Type_of_service https://en.wikipedia.org/wiki/Type_of_service
- vitus 2y ago
  DSCP / ToS is used for traffic classification within a network (which can be used to select router policy), but generally not across autonomous systems. You'd have to get everyone to agree on what various traffic markings mean, and, more critically, which traffic deserves to be marked in certain ways. For instance, suppose that you wanted a special traffic class that's prioritized over everything else. You really don't want just anyone to be able to send you that kind of traffic, since a DoS attack can easily starve all the other queues. Your best bet is usually to match traffic based on other characteristics, e.g. port × protocol.
  - gary_0 2y ago
    Ah, I see. Still, it seems like it would be a useful idea to have an IP bit for "minimize buffering and drop instead, packet is time-sensitive", and that seems safe since the user is requesting fewer resources, not more.
    - vitus 2y ago
      You would think so, but buffers are actually a core part of regular router function. The problem is not inherently router buffers, but rather buffers that don't fully drain (bufferbloat) or that fill up (congestive drops). Think about it this way: a 100Gbps link can process a 1500 byte packet in about 0.12 microsecond. If you have an average of 1,000 packets in that buffer at steady-state, that buffer is contributing a fraction of a millisecond to your overall latency. Meanwhile, if your home router's buffer has the same 1,000 packets for a 1Gbps link, that's 12 milliseconds of latency. If you had a way to tell a router to never buffer packets, you'd encounter packet loss much sooner, and for no good reason. Buffers are great for smoothing out (small) bursts of traffic (although if you have large bursts on very short timescales, you can easily overwhelm these buffers -- this is why you typically have pacing on the OS level). If instead you want the router to add no more than X amount of latency, that's suddenly much harder to dictate. (The "correct" value of X depends on your application requirements as well as the number of hops through the network between you and the server you're talking to.) (Also, while you might think that this should be strictly beneficial for network operators since the user wants fewer resources, encoding exceptions like this into router policy ends up using more of a specialized and expensive kind of memory called TCAM. Assuming that it's even possible to do such a thing.)
ta1243 2y ago
Used high numbers of UDP packets over intercontinental internet links for mission critical applications for 15 years. 20mbit of UDP carrying RTP. Loss on a given flow is quite rare, and the application helps (via duplication, or retransmits) As time has progressed increased from nothing to fec to dual-streaming and offset-streaming to RIST and SRT depending on the criticality. On the other hand I've seen people try to use TCP (with rtmp) and fail miserably. Never* use TCP. Or you know, use the right tool for the right job.
dang 2y ago
I've attempted to replace the clickbait title* using terms from the article itself, but if someone can suggest a more representative phrase from the article, we can change it again. (* in keeping with the HN's title guideline: "Please use the original title, unless it is misleading or linkbait" - https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html)
kierank 2y ago
Imagine Ethernet was designed like this and you had to implement mandatory congestion control and other cruft. The layer of the stack that has knowledge of the content should be implementing the congestion control.
- ay 2y ago
  Ethernet was designed like this :-) https://en.m.wikipedia.org/wiki/IEEE_802.2 https://en.m.wikipedia.org/wiki/IEEE_802.2 Back in the olden times, the Windows 3.11 and Windows 95 could even run services atop it using another protocol: https://en.m.wikipedia.org/wiki/NetBIOS_Frames https://en.m.wikipedia.org/wiki/NetBIOS_Frames In fact, the link layer protocols that dealt with congestion and access to media more meticulously, eg Token Ring or FDDI, were generally much more efficient with the use of the media - using near 100% of the potential bandwidth, whereas Ethernet on a shared medium already gets quite inefficient at 30-40% utilization due to retries caused by collisions. However, the trade off between additional complexity (and thus bigger cost and difficulty of troubleshooting) was such that the much simpler and dumber Ethernet has won. However, a lot of similar principles are there in Fibre Channel family protocols, which are still used in some data center-specific protocols.

224 points by wofo 2y ago

What should be based on datagrams: - Local discovery (DHCP, slaac, UPnP, mDNS, tinc, bittorrent) - Broadcasts (Local network streaming) - Package encapsulation (wireguard, IPSec, OpenVPN, vlan)

promiseofbeans 2y ago
Hey I'm sure you're right, but as someone with less low-level networking expertise, I was wondering if you could explain why you'd want datagrams for these use-cases?
- deleted 2y ago
  [deleted]
- swiftcoder 2y ago
  Local discovery is itself based on broadcast/multicast, both of which only work over UDP (TCP doesn't provide any mechanism for broadcast streams).
- blueflow 2y ago
  TCP is a back & forth between two hosts and thus cant do "shouting blindly into the network" things. That rules out local discovery and media broadcast. Package encapsulation is bad over TCP because if the encapsulated data is TCP itself, you have congestion control twice. On congested networks, this results in extra slowdowns that can make the connection unusable.
deleted 2y ago
[deleted]
Hikikomori 2y ago
And games.
- swiftcoder 2y ago
  The whole point of the article is that you don't need to go all the way down to raw datagrams to achieve the kind of low latency that is needed for things like games/VOIP/livestreaming
- eru 2y ago
  It totally depends on the kind of game.
ajb 2y ago
One missing from your list is real-time media. Retransmission, or even just buffering for reordering, adds latency so it's better to take the hit of a loss with error correction or packet-loss-concealment
- tfyoung 2y ago
  Yeah exactly. With realtime, a dropped packet is often too old by the time it's re-transmitted. It's actually harmful to wait for it when you'd rather just skip it and move on and stay as close to realtime as possible. It's a very different usecase from most uses. I do wish QUIC allowed carrying streams that were useful for realtime in conjunction with allowing reliable streams. Using MPEG-TS over SRT to have it just spam metadata to handle the unreliableness is janky. It would be far nicer to have a reliable stream for metadata, then an unreliable one for realtime streaming.

forrestthewoods 2y ago

It mentions that the video game industry uses UDP but then fails to further address that use case. So, should competitive shooter video games switch to QUIC? Is that even supported across all the various gaming platforms?

swiftcoder 2y ago
> It mentions that the video game industry uses UDP but then fails to further address that use case Video games tend to use UDP for the same reason everyone else mentioned does: timeliness. You want the most recent position of the various game objects now, and you don't give a shit about where they were 100ms ago. The proposed solution of segmenting data into QUIC streams and mucking with priorities should work just fine for a game. > Is that even supported across all the various gaming platforms? QUIC itself is implemented in terms of datagrams, so if you have datagrams, you can have QUIC.
- Tuna-Fish 2y ago
  But the proposed solution provides no value over datagrams, while it has plenty of downsides. Let me rephrase the problem here for a simple FPS game: The entire world state (or, the subset that the client is supposed to know about) is provided in each packet sent from the server. The last few actions with timestamps are in each packet sent from the client. You always want to have the latest of each, with lowest possible latency. Both sides send one packet per tick. You do not want retransmission based on the knowledge that a packet was lost (the RTT is way too long for the information that a packet was lost to ever be useful, you just retransmit everything every tick), you do not want congestion control (total bandwidth is negligible, and if there is too much packet loss to maintain what is required, there is no possible solution to maintain sufficient performance and you shouldn't even try), and none of the other features talked about in the post add anything of value, either. It reads like someone really likes QUIC, it fit well into their problems, and they are a bit too enthusiastic about evangelizing it.
  - swiftcoder 2y ago
    In practice most games also need reliable side channels for control messages/chat/etc, and so they end up building optional reliable streams over UDP... and at the end of this path lies something that looks a lot like a (less thoroughly designed/tested) version of QUIC

promiseofbeans 2y ago

We use straight UDP datagrams for streaming high-frequency sensor data. One of our R&D people built a new system that uses quic and solves most of our problems with out-of-order delivery. We still use datagrams over UDP for everything because we have to support some 3rd party sensors out of the box without adapters, and UDP is all they can do.

jcelerier 2y ago
Also pretty much every media art system in the world just uses OSC over UDP
- jesprenj 2y ago
  And ArtNet also over UDP.
Almondsetat 2y ago
Yeah, high frequency sensor data was on my mind too. Do you have any figures on the power saved wrt TCP?
- Joel_Mckay 2y ago
  In general, the wireless handshake on G5/LTE/Wifi/Starlink will dwarf the traffic of sensors for power use. For old-style satellite mailboxes, the traffic is better batched in 45min chunks anyway (the 5W transmitters are usually only active for well under 20 seconds per message, and use a super-cap to handle the RF power draw spike.) Naive UDP protocols have reachability problems in numerous scenarios, variable costs can balloon (concurrency with polling links is dumb), and can open a hole in your infrastructure at scale without special equipment mods. Only if you have a tier 3 or better WAN trunk or cloud-center should you even consider something silly like QUIC. YMMV, and I love the AstroTurf on YC... lol ;)
songbird23 2y ago
what language was the system build with? thats super cool!

asdefghyk 2y ago

Maybe ... never use on a congested network. or Never use on a network where congestion is above a certain level or Never us on a network where this parameter is above a certain level - like network latency or only use on a LAN not a WAN ....?

swiftcoder 2y ago
Unless you are building software for a pre-established intranet, how do you predict ahead of time whether your software will run on congested networks? Most end-user software ends up running on a variety of oversubscribed wifi and cell networks
- smcameron 2y ago
  > how do you predict ahead of time whether your software will run on congested networks? One thing you can do is build in "bad network emulation" into your software, allowing the various pieces to drop packets, delay packets, reorder packets, etc. and make sure it still behaves reasonably with all this garbage turned up.
  - asdefghyk 2y ago
    RE ".... build in "bad network emulation" into your software....." I expect there would be network emulators that would be able to emulate these situations. How good these emulators are would be the next question.
- marcosdumay 2y ago
  You tell the user "hey, don't use on congested networks". And yeah, since most internet software has this exact disclaimer, it works kinda well. But people do appreciate software that doesn't have this restriction, and even go out of their way to talk about it (more on mobile networks than on wired connections).
ot 2y ago
LAN can be oversubscribed too, even if your switches have sufficient capacity you can have short packet bursts that fill up the queues. You cannot assume reliable delivery.

FpUser 2y ago

I use UDP for my desktop game like software with clients all over the world to propagate anonymized state. It needs no encryption as the transmitted data has zero value to any third party. It needs no reliability since dropped packets would be handled by predictive filter with practical reliability that is way more than needed. So why the F.. would I bother with anything else?

aarmot 2y ago
Because author of the article likes to be edgy and parrot the deep truth he learned yesterday - that TCP is "reliable" and UDP is not. The reality, of course, is somewhat different and muddy. For example, if you have network outage following later by a software crash or a reboot, then all the TCP buffer worth of data (several kilobytes or upto some megabytes - depends on your tuning) is mercilessly dropped. And your application thinks that just because you used TCP the data must have been reliably delieved. To combat this you have to implement some kind of serializing and acking - but the article scoffs us that we are too dumb to implement anything besides basic stream /s I'm not arguing that TCP is useless, just that UDP has its place and we - mere mortals - can use is too. Where appropriate.
- Lerc 2y ago
  The thing is, when you need that low latency, you don't want most of the features of more robust streams. When you are using UDP the correct way to handle out of order delivery is to just ignore the older packet. It's old and consequently out of date. Figuring out how to solve any mess caused by mistransmission necessarily has to be done at the application level because that's where the most up to date data is.
- ben0x539 2y ago
  The author (disclaimer: former coworker) has likely learned about UDP before yesterday because he's been complaining about TCP for years.
- vitus 2y ago
  > Because author of the article likes to be edgy and parrot the deep truth he learned yesterday I will point out that "author of the article" is one of the core contributors in the IETF Media-over-QUIC working group (which is an effort to standardize how one might build these real-time applications over QUIC) and has been working in the real-time media protocols space for 10+ years. The author recognizes that the title is clickbait, but the key point of the article is that you probably don't want to use raw UDP in most cases. Not that UDP is inherently bad (otherwise he wouldn't be working on improving QUIC).

ggm 2y ago

Quic is implemented over UDP. It's literally running over datagrams.

ot 2y ago
Everything at some layer has to run over datagrams, since that's what IP is. This is literally the point of the article: if you want to create a protocol over raw datagrams, you have to implement a lot of things that are very hard to get right, so you should just use QUIC instead, which does them for you.
- ggm 2y ago
  "You" is doing all the hard work. Apps should use reliable sessions and transport, almost all the time? No disagree. The exceptions are understood. But don't pretend the substrate is that reliable data stream. "We" have to construct it almost always. We do reliable for you, over UDP. I don't do this stuff any more, but I worked on OSI transport and remote operations service mapped to UDP and other protocols back in the 80s

PhilipRoman 2y ago

IMO stream abstractions make it too convenient to write fragile programs which are slow to recover from disconnections (if they do at all) and generally place too many restrictions on the transport layer. Congestion control is definitely needed but everything else seems questionable. In a datagram-first world we would have no issue bonding any number of data links with very high efficiency or seamlessly roaming across network boundaries without dropping connections. Many types of applications can handle out-of-order frames with zero overhead and would work much faster if written for the UDP model.

CJefferson 2y ago
Out of interest, what kind of applications are you thinking of? What systems are common written using TCP, that could switch to UDP? Clearly websites, audio and video generally don't work with out-of-order frames -- most people don't want dropped audio and video. Some video games are happy to ignore missed packets, but when they can they are already written in UDP.
- ansgri 2y ago
  Any data block that must be fully received before processing comes to mind (e.g. websites’ html), just request retransmission of parts that didn’t make it in first pass. Funnily enough it’s (realtime) video and audio that already uses UDP due to preferring timely data to complete data. On the contrary, for me it’s hard to imagine video game with missed packets as the state can get out of sync too easily, you’ll need eventual consistency via retransmission or some clever data structures (I know least about this domain though)
  - smcameron 2y ago
    I'm not expert in this area, but my understanding is that for latency sensitive video games, on the client side, state is predicted between updates, e.g. the state may be updated via the network, at say, roughly 10Hz (probably faster, but probably less than 60Hz), but updated locally at 60Hz via interpolation, dead reckoning and other predictive/smoothing heuristics, a few missed updates just means a bit more prediction is used. "State" is not usually transmitted as one lump "frame" at a time, but rather per-game unit (per spaceship, asteroid, robot, or whatever is in the game) or per some-small-group of units. When some updates are delayed too long, you might get some visible artifacts, "rubber-banding", "teleportation" or "warping", etc. often lumped together by players under the umbrella term "lag". For out-of-order packets, older packets might be dropped as newer packets may already have been applied to the state of some game units (typically there's some timestamps or monotonically increasing counter associated with state updates for each unit used by the prediction/interpolation/smoothing heuristics) and the state is usually represented in absolute terms rather than relative terms (e.g. (x,y) = (100,100), rather than x+=10, y+=10, so that any update may be applied in isolation.)

bitcharmer 2y ago

This is a narrow way of looking at UDP applications. The whole HFT, low-latency fintech world is built on top of datagrams. Using TCP would be pure the worst choice possible.

AtlasBarfed 2y ago
Is that because they mostly use dedicated links and intranets? A lot of Enterprise messaging is based on UDP, I think on the presumption that corporate networks are just simply going to be a lot more reliable
- bitcharmer 2y ago
  You are correct, it's either point-to-point, like direct cross-connects with the exchange or it's a relatively small intranet. Additionally, an overwhelming majority of packets flying around are market data. You don't want to re-transmit lost ones because by the time you do, the data is already outdated.

adunk 2y ago

This may seem like a minor nit, but I think there is a problem with using the term "unreliable" to describe UDP. The more commonly used term, and IMHO better term, is "best-effort" [1]. UDP makes its best effort to deliver the datagrams, but the datagrams may be dropped anyway. But it does not make UDP inherently unreliable. [1] https://en.wikipedia.org/wiki/Best-effort_delivery https://en.wikipedia.org/wiki/Best-effort_delivery

vitus 2y ago
IMO "best-effort" is euphemistic and confusing to people outside of this space, even (especially?) to native English speakers. (I would probably have described UDP as "reasonable effort" and TCP as "best effort", if not for the existing terminology.) I recall my computer networking professor describing this as "Best-effort means never having to say you're sorry". In practice, best-effort does not mean you try your hardest to make sure the message gets from A to B, it means that you made an effort. Router in the path was congested? Link flap leading to blackholing on the order of 50ms before fast reroute kicks in? Oh well, we tried. Meanwhile, TCP's reliable delivery will retry several times and will present an in-order data stream to the application. Reliable vs unreliable might be bad terminology, but I don't think best-effort is any better. My experience with unreliable systems is that they're great something like 95% of the time, and they're great for raw throughput, but there are many cases where that last 5% makes a huge difference.
- eru 2y ago
  The question is 'who deals with dropped packages'? In TCP, the answer is: 'the protocol'. In UDP the answer is 'the next layer of abstraction' (eg the app or some library). You can build a 'reliable' protocol on top of UDP, and still not get TCP. Eg if you want to transfer a large file that you know up front, then TCP's streaming mechanism doesn't make too much sense. You could use something like UDP to send the whole file from A to B in little chunks once, and at the end B can tell A what (numbered) chunks she's missing. There's no reason to hold off on sending chunk n+1 of the file, just because chunk n hasn't arrived yet.
  - 2y ago

JackSlateur 2y ago

Network is nothing but datagrams.

dale_glass 2y ago

IMO one of the worst mistakes made in IP development was not having made a standard protocol for the one thing that pretty much everyone seems to want: A reliable, unlimited-length, message-based protocol. With TCP there's a million users that throw out the stream aspect and implement messages on top of it. And with UDP people implement reliability and the ability to transmit >1 MTU. So much time wasted reinventing the wheel.

supriyo-biswas 2y ago
There’s Homa[1]. [1] https://github.com/PlatformLab/Homa https://github.com/PlatformLab/Homa
H8crilA 2y ago
Sorry for a useless comment but now I feel so stupid for never noticing that pretty much everything done on TCP is actually message exchange, not streams. The mismatch is indeed so unfortunate.
- eru 2y ago
  Well, you often have some dependency between messages. Some re-orderings don't matter, but some do. If you just keep everything in the same order, then you never have to worry about any re-ordering. I can see why people picked streams as the one-size-fits-all-(but-badly) abstraction.
  - tliltocatl 2y ago
    Because worse is better and if you try to be everything you (don't) get OSI. Also because everyone loves serial ports and dialup modems and TCP just emulates a serial port over packet network. And it wasn't supposed to be one-size-fits-all, rather «let us have telnet while we figuring out the rest» but then ossification happend.
- iainmerrick 2y ago
  Is it really that unfortunate? What’s the big downside? There’s a bit of reinvention needed, sure, but adding a framing layer is about the easiest task in networking. If you want a standard way of doing it, these days you can use a WebSocket. Edit to add: oh, I see, you want reliable but unordered messages. That would definitely be useful sometimes, but other times you do want ordering. If you don’t need ordering, isn’t that pretty much what QUIC does?

thomashabets2 2y ago

> The bytes within each stream are ordered, reliable, and can be any size; it’s nice and convenient. Each stream could be a video frame […] But you can tell the QUIC stack to focus on delivering important streams first. The low priority streams will be starved, and can be closed to avoid wasting bandwidth. Is the author saying that with QUIC I can send a "score update" for my game (periodic update) on a short-lived stream, and prevent retransmissions? I'll send an updated "score update" in a few seconds, so if the first one got lost, then I don't want it to waste bandwidth retransmitting. Especially I don't want it retransmitted after I've sent a newer update.

gnfargbl 2y ago
I think you're looking for RFC 9221, "An Unreliable Datagram Extension to QUIC."
- swiftcoder 2y ago
  You don't need datagrams for this. You can set the stream to discard itself as soon as packet loss is detected
  - crote 2y ago
    That doesn't quite provide the same experience, does it? For something like a scoreboard you don't want the message to be discarded once any packet loss is detected. As long as message N is the most recent one, message N should be retried until it succeeds. You just want to stop retrying N once N+1 becomes available - or ideally even successfully delivered.
    - swiftcoder 2y ago
      In a game context, where we're spewing state updates out 10-60x per second, there is always a newer packet in flight already, so discard-on-loss produces the same result. In your scoreboard example, you probably just want an actually reliable stream for that - it's not like scores are rolling in so fast that it's beneficial to deal with it as unreliable.

mjw_byrne 2y ago

Silly clickbait title, which the author even admits up front. UDP and TCP have different behaviour and different tradeoffs, you have to understand them before choosing one for your use case. That's basically it. No need for "Never do X" gatekeeping.

g15jv2dp 2y ago
My apologies if the star in front of "never" was added to the title in the last ten minutes after you posted your comment. But that star is clearly there to indicate "fine print ahead"; in a title, it's pretty obvious that it means that the article is not about to "gatekeep" people out of UDP. (In fact, at the end of the article, the author suggests to use QUIC which, you guessed it, is based on UDP.)
deleted 2y ago
[deleted]

tliltocatl 2y ago

Most of TCP woes comes from high-bandwith latency-sensitive stuff like HFT and video, but TCP isn't particularly good for low-bandwidth high-latency networks either (e. g. NB-IoT with 10 seconds worst case RTT): - TCP will waste roundtrips on handshakes. And then some extra on MTU discovery. - TCP will keep trying to transmit data even if it's no longer useful (same issue as with real-time multimedia). - If you move into a location with worse coverage, your latency increases, but TCP will assume packet loss due to congestion and reduce bandwidth. And in general, loss-based congestion control just doesn't work at this point. - Load balancers and middleboxes (and HTTP servers, but that's another story) may disconnect you randomly because hey, you haven't responded for four seconds, you are probably no longer there anyway, right? - You can't interpret the data you've got until you have all of it - because TCP will split packets with no regards to data structure. Which is twice as sad when all of your data would actually fit in 1200 bytes.

eru 2y ago
And TCP wants to pretend that all your data arrives as a linear stream one after the other; and keeps you from seeing package n+1, if you haven't received package n, yet. That's useful for some things, but often you can make use of package n+1, even when package n hasn't arrived, yet. For example, when you are transferring a large file, you could use erasure encoding to just automatically deal with 5% package loss. (Or you could use a fountain code to deal with variable packet loss, and the sender just keeps sending until the receiver says "I'm done" for the whole file, instead of ack-ing individual packages. Fountain codes are how deep space probes send their data back. Latency is pretty terrible out to Jupiter or Mars.)
- Aurornis 2y ago
  > And TCP wants to pretend that all your data arrives as a linear stream one after the other; That’s one of the reasons people use TCP. > For example, when you are transferring a large file, you could use erasure encoding to just automatically deal with 5% package loss. It’s never that simple. You can’t just add some erasure coding and have it automatically solve your problems. You now have to build an entire protocol around those packets to determine and track their order. You also need mechanisms to handle the case where packet loss exceeds what you can recover, which involves either restarting the transfer or a retransmission mechanism. The number of little details you have to handle quickly explodes in complexity. Even in the best case scenario, you’d be paying a price to handle erasure coding on one end, the extra bandwidth of the overhead, and then decoding on the receiving end. That’s a lot of complexity, engineering, debugging, and opportunities to introduce bugs, and for what gain? In the file transfer example, what would you actually gain by rolling your own entire transmission protocol with all this overhead? > Fountain codes are how deep space probes send their data back. Latency is pretty terrible out to Jupiter or Mars.) Fountain codes and even general erasure codes are not the right tool for this job. The loss of a digital packetized channel across the internet is very different than a noisy analog channel sent through space.

nabla9 2y ago

The choice is not UDP vs TCP. UDP adds minimum over raw sockets so that you don't need root privileges. Other protocols are build on top of UDP. It's better to use existing not-TCP protocols instead of UDP when the need arises instead of making your own. Especially for streaming.

ozim 2y ago

Great in depth article from what seems really a person who knows that stuff.

nyc_pizzadev 2y ago

One thing not mentioned often is that a lot of networks will drop UDP packets first when encountering congestion. The thinking is that those packets will not re-transmit, so it’s an effective means to shed excess traffic. Given we now have protocols that aggressively re-transmit on UDP, I wonder how that has changed things. I do seem to remember QUIC having re-transmit issues (vs HTTP1/2) years ago because of this.

cenriqueortiz 2y ago

Nahhh. While most of the applications/cases will be using session-based connections, there are uses for using datagrams directly — don’t be afraid. Yes, you will have to take care of many more details yourself. And as a side line, it is a great way of learning the low level aspects of networking.

ddtaylor 2y ago
The SteamNetworkingMessages API for game development does a good job of making this available if you want it without caring about the internals for that use case.

Anon_Admirer 2y ago

Hope this one gets captured by quackernews - can’t wait to see its description.

TuringNYC 2y ago

>> The common wisdom is: >> >> use TCP if you want reliable delivery >> >> use UDP if you want unreliable delivery >> What the *(& does that mean? Who wants unreliability? I dont agree with the premise of this article, UDP isnt for unreliability, it provides a tradeoff which trades speed and efficiency and provides best-efforts instead of guarantees. It makes sense depending on your application. For example, if I have a real-time multi-player video game, and things fall behind, the items which fell behind no longer matter because the state of the game changed. Same thing for a high-speed trading application -- I only care about the most recent market data in some circumstances, not what happened 100ms ago.

alexey-salmin 2y ago
That's basically what the article says if you keep on reading. It states the "common wisdom" at first to disagree with it.
- hoseja 2y ago
  Well then it's clickbait.
opheliate 2y ago
That isn’t the premise of the article, that is the “common wisdom” the author corrects as the article goes on. The author goes on to list video games as an example of where UDP makes sense, as well as live video.

20k 2y ago

I feel like this article misses why people avoid TCP like the plague, and why people use UDP for many applications 1. Routers do all kinds of terrible things with TCP, causing high latency, and poor performance. Routers do not do this to nearly the same extent with UDP 2. Operating systems have a tendency to buffer for high lengths of time, resulting in very poor performance due to high latency. TCP is often seriously unusable for deployment on a random clients default setup. Getting caught out by Nagle is a classic mistake, its one of the first things to look for in a project suffering from tcp issues 3. TCP is stream based, which I don't think has ever been what I want. You have to reimplement your own protocol on top of TCP anyway to introduce message frames 4. The model of network failures that TCP works well for is a bit naive, network failures tend to cluster together making the reliability guarantees not that useful a lot of the time. Failures don't tend to be statistically independent, and your connection will drop requiring you to start again anyway 5. TCP's backoff model on packet failures is both incredibly aggressive, and mismatched for a flaky physical layer. Even a tiny % of packet loss can make your performance unusable, to the point where the concept of using TCP is completely unworkable Its also worth noting that people use "unreliable" to mean UDP for its promptness guarantees, because reliable = TCP, and unreliable = UDP QUIC and TCP actively don't meet the needs of certain applications - its worth examining a use case that's kind of glossed over in the article: Videogames I think this article misses the point strongly here by ignoring this kind of use case, because in many domains you have a performance and fault model that are simply not well matched by a protocol like TCP or QUIC. None of the features on the protocol list are things that you especially need or even can implement for videogames (you really want to encrypt player positions?). In a game, your update rate might be 1KB/s - absolutely tiny. If more than N packets get dropped - under TCP or UDP (or quic) - because games are a hard realtime system you're screwed, and there's nothing you can do about it no matter what protocol you're using. If you use QUIC, the server will attempt to send the packet again which.... is completely pointless, and now you're stuck waiting for potentially a whole queue of packets to send if your network hiccups for a second, with presumably whatever congestion control QUIC implements, so your game lags even more once your network recovers. Ick! Should we have a separate queue for every packet? Videogame networking protocols are built to tolerate the loss of a certain number of packets within a certain timeframe (eg 1 every 200ms), and this system has to be extremely tightly integrated into the game architecture to maintain your hard realtime guarantees. Adding quic is just overhead, because the reliability that QUIC provides, and the reliability that games need, are not the same kind of reliability Congestion in a videogame with low bandwidths is extremely unlikely. The issue is that network protocols have no way to know if a dropped packet is because of congestion, or because of a flaky underlying connection. Videogames assume a priori that you do not have congestion (otherwise your game is unplayable), so all recoverable networking failures are 1 off transient network failures of less than a handful of packets by definition. When you drop a packet in a videogame, the server may increase its update rate to catch you up via time dilation, rather than in a protocol like TCP/QUIC which will reduce its update rate. A well designed game built on UDP tolerates a slightly flakey connection. If you use TCP or QUIC, you'll run into problems. QUIC isn't terrible, but its not good for this kind of application, and we shouldn't pretend its fine For more information about a good game networking system, see this video: https://www.youtube.com/watch?v=odSBJ49rzDo https://www.youtube.com/watch?v=odSBJ49rzDo, and it goes over pretty in detail why you shouldn't use something like QUIC

dicroce 2y ago

I wonder if this guy thinks video GOPs should all be hundreds of frames long because P frames are so much smaller than I frames and since in his world we NEVER use an unreliable network you might as well.

deleted 2y ago

[deleted]

karmakaze 2y ago

TL;DR - use QUIC (should have just looked at the domain name)

dragonfax 2y ago

I've seen UDP used for great effect in video streaming. Especially timely video streaming such as cloud gaming. When waiting a late packet is no longer useful.

Joel_Mckay 2y ago
Not as popular as it once was, but it is still in use: https://en.wikipedia.org/wiki/Real-Time_Streaming_Protocol https://en.wikipedia.org/wiki/Real-Time_Streaming_Protocol Cheers =3
- kwindla 2y ago
  RTSP is the control protocol. Some other protocol is needed for the actual audio/video streaming. That's usually RTP, these days. RTP is a core part of WebRTC, for example. When you're doing a video call in a web browser, you're using WebRTC, including RTP. In fact, this RTP-via-WebRTC is the only way to send UDP packets from JavaScript! RTSP is still used by older streaming systems and hardware ecosystems that are slow to change, such as network-connected security cameras. But in newer applications, WebRTC has mostly replaced it. Of course, the QUIC effort is in part an attempt to replace WebRTC, so the wheel continues to turn!
  - Joel_Mckay 2y ago
    WebRTC still has its own set of issues, and I found it only slightly improved over other options compiling the ARM64 port: https://github.com/mpromonet/webrtc-streamer.git https://github.com/mpromonet/webrtc-streamer.git I remain unconvinced UDP based streams will ultimately remain in the long-term, but webRTC certainly made it easier to peer a connection. ;)

deleted 2y ago

[deleted]

gary_0 2y ago

I wonder if the DiffServ[0] bits in IPv6 could be another way to prevent bufferbloat from affecting real-time datagrams? Or are they like IPv4's ToS[1] bits, which I think were never implemented widely (or properly) enough for any software to bother with? [0] https://en.wikipedia.org/wiki/Differentiated_services https://en.wikipedia.org/wiki/Differentiated_services [1] https://en.wikipedia.org/wiki/Type_of_service https://en.wikipedia.org/wiki/Type_of_service

vitus 2y ago
DSCP / ToS is used for traffic classification within a network (which can be used to select router policy), but generally not across autonomous systems. You'd have to get everyone to agree on what various traffic markings mean, and, more critically, which traffic deserves to be marked in certain ways. For instance, suppose that you wanted a special traffic class that's prioritized over everything else. You really don't want just anyone to be able to send you that kind of traffic, since a DoS attack can easily starve all the other queues. Your best bet is usually to match traffic based on other characteristics, e.g. port × protocol.
- gary_0 2y ago
  Ah, I see. Still, it seems like it would be a useful idea to have an IP bit for "minimize buffering and drop instead, packet is time-sensitive", and that seems safe since the user is requesting fewer resources, not more.
  - vitus 2y ago
    You would think so, but buffers are actually a core part of regular router function. The problem is not inherently router buffers, but rather buffers that don't fully drain (bufferbloat) or that fill up (congestive drops). Think about it this way: a 100Gbps link can process a 1500 byte packet in about 0.12 microsecond. If you have an average of 1,000 packets in that buffer at steady-state, that buffer is contributing a fraction of a millisecond to your overall latency. Meanwhile, if your home router's buffer has the same 1,000 packets for a 1Gbps link, that's 12 milliseconds of latency. If you had a way to tell a router to never buffer packets, you'd encounter packet loss much sooner, and for no good reason. Buffers are great for smoothing out (small) bursts of traffic (although if you have large bursts on very short timescales, you can easily overwhelm these buffers -- this is why you typically have pacing on the OS level). If instead you want the router to add no more than X amount of latency, that's suddenly much harder to dictate. (The "correct" value of X depends on your application requirements as well as the number of hops through the network between you and the server you're talking to.) (Also, while you might think that this should be strictly beneficial for network operators since the user wants fewer resources, encoding exceptions like this into router policy ends up using more of a specialized and expensive kind of memory called TCAM. Assuming that it's even possible to do such a thing.)

ta1243 2y ago

Used high numbers of UDP packets over intercontinental internet links for mission critical applications for 15 years. 20mbit of UDP carrying RTP. Loss on a given flow is quite rare, and the application helps (via duplication, or retransmits) As time has progressed increased from nothing to fec to dual-streaming and offset-streaming to RIST and SRT depending on the criticality. On the other hand I've seen people try to use TCP (with rtmp) and fail miserably. Never* use TCP. Or you know, use the right tool for the right job.

dang 2y ago

I've attempted to replace the clickbait title* using terms from the article itself, but if someone can suggest a more representative phrase from the article, we can change it again. (* in keeping with the HN's title guideline: "Please use the original title, unless it is misleading or linkbait" - https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html)

kierank 2y ago

Imagine Ethernet was designed like this and you had to implement mandatory congestion control and other cruft. The layer of the stack that has knowledge of the content should be implementing the congestion control.

ay 2y ago
Ethernet was designed like this :-) https://en.m.wikipedia.org/wiki/IEEE_802.2 https://en.m.wikipedia.org/wiki/IEEE_802.2 Back in the olden times, the Windows 3.11 and Windows 95 could even run services atop it using another protocol: https://en.m.wikipedia.org/wiki/NetBIOS_Frames https://en.m.wikipedia.org/wiki/NetBIOS_Frames In fact, the link layer protocols that dealt with congestion and access to media more meticulously, eg Token Ring or FDDI, were generally much more efficient with the use of the media - using near 100% of the potential bandwidth, whereas Ethernet on a shared medium already gets quite inefficient at 30-40% utilization due to retries caused by collisions. However, the trade off between additional complexity (and thus bigger cost and difficulty of troubleshooting) was such that the much simpler and dumber Ethernet has won. However, a lot of similar principles are there in Fibre Channel family protocols, which are still used in some data center-specific protocols.