14 ms·

UDP vs TCP

453 points by colinprince 10y ago

dozzie 10y ago
I thought that it was somebody who has recently discovered the existence of UDP protocol and brags about that to the world, but from skimming the article, it actually has some non-trivial remarks about UDP and TCP. BTW, the article's view angle is multiplayer game programming.
- exDM69 10y ago
  This guy (Glenn Gaffer) has been writing network protocols for games for years and shipped popular titles. On his site you can also find an irate rant about people with no experience dismissing his claims as "reimplementing TCP". It has good points but it's a frustrated rant (reader beware). But it addresses a lot of the commonly believed fallacies. His latest project is an open source library for game networking over UDP: https://github.com/networkprotocol/libyojimbo https://github.com/networkprotocol/libyojimbo And he is correct, for "real time" games you must use UDP or you will be in trouble in real world networking conditions. When everything runs smoothly, TCP and UDP work almost identically but when packet loss occurs, TCP will make things worse.
  - jay_kyburz 10y ago
    Glenn _Fiedler_ - Gaffer is his online name / handle. Worked with Glenn on Freedom Force and Tribes. He knows his stuff!
techman9 10y ago
I appreciate the first user comment on this article: "I didn’t care for this article 7 months ago. I regret it. My whole game is useless now. I should have go with udp."
aakilfernandes 10y ago
Another important difference: UDP can't be used in the browser =(
- oszione 10y ago
  With the exception of the QUIC protocol. * TLS over 443/udp
- rdw 10y ago
  WebRTC is UDP. It's not easy, but it's there!
  - foota 10y ago
    nitpick: It can be either
    - arkitectual 10y ago
      It is possible to use only one or the other.
- semicolon_storm 10y ago
  Couldn't you potentially use something like WebRTC to get UDP functionality in the browser?
- simplehuman 10y ago
  Well, neither can TCP.
  - jensvdh 10y ago
    HTTP is built on top of TCP. Websockets are TCP connections.
    - simplehuman 10y ago
      So? TCP cannot be used directly in the browser by applications authors. It's an abstraction.
mjevans 10y ago
The main difference between TCP and UDP, as this programmer discovered, relates to quality of realtime service. Times to use UDP over TCP * When you need the lowest latency * When LATE data is worse than GAPS (loss of) in data. * When you want to implement your own form of error correction to handle late/missing/mangled data. TCP is best when * You need all of the data to arrive, period. * You want to automatically make a rough best estimate use of the available connection for /rate/ of transfer. For a videogame, having a general control channel to the co-ordination server in TCP is fine. Having interactive asset downloads (level setup) over TCP is fine. Interactive player movements /probably/ should be UDP. Very likely with a mix of forward error correction and major snapshot syncs for critical data (moving entity absolute location, etc).
- exDM69 10y ago
  Another article on this site addresses TCP plus UDP mix. The tl;dr is that they should not be used at the same time because TCP retransmission will interfere with UDP (and routers give TCP priority). As long as TCP happens outside of the "hot" loop, it's fine.
  - wtallis 10y ago
    > (and routers give TCP priority). What routers? Is this something that stupid routers did in the '90s and people are still worried about, or are modern routers actually behaving this way? (And if so, why?) TCP retransmission interfering with UDP sounds like something that shouldn't be a problem on any sane network, especially if the application authors are smart and use something like LEDBAT for their TCP traffic when they're also doing latency-sensitive UDP communication.
    - vvanders 10y ago
      You'd be shocked at the things home routers do. There's a reason most people don't try to implement NAT traversal and use common middleware. Since UDP isn't as common of a protocol I can certainly see some routers treating it a second class traffic when it comes to packet shaping.
mikepurvis 10y ago
The recommendation to avoid TCP altogether is surprising to me. Having encountered a number of video-conferencing systems which are in a similar space, it seems pretty standard to have separate real-time and control sockets on UDP and TCP respectively. I skimmed the linked paper and didn't find it conclusive; can someone summarize how it is that having a TCP socket can affect UDP traffic on the same interface? All that said, I certainly see the argument for an all-UDP protocol in terms of defining your own retransmission approach, or attempting to avoid it altogether with forward error correction or whatever.
- blackflame7000 10y ago
  I too was wondering how TCP and UDP can affect one another on the same interface and the explanation I read is that at the lowest level both UDP and TCP segment data into packets and place them onto the same network adapter's queue which can affect they way they are transmitted under congestion depending on the device/drivers.
  - wtallis 10y ago
    That's not an explanation. That's just pointing in the general direction of where to look for an explanation.
    - blackflame7000 10y ago
      Do you have a better explanation to offer because I'd love to learn? Not sure why I deserve a downvote for offering a potential solution when you with 6000 karma offer nothing.
      - wtallis 10y ago
        I think the claim was far too broad to actually be entirely true, and the evidence was both narrow and weak. The 1997 paper only shows that certain simulation parameters can produce bad packet loss for UDP traffic. But those parameters are almost all irrelevant to today's Internet: MTU is now 1500B not 512B, TCP congestion control is usually something like CUBIC rather than RENO, buffers are never a mere 16 packets long, and TCP traffic is more often relatively short-lived HTTP connections, not long-running FTP. The paper is just another piece of evidence that TCP synchronization sucks. You are very unlikely to encounter such perfect TCP synchronization on today's Internet, and even if you did, the dynamics of introducing some UDP flows to that situation would be different.
milansuk 10y ago
> they may arrive out of order, be duplicated, or not arrive at all! This is first time what I read that datagram can be duplicated. Is it true? It's duplicated by network or does it mean that peer send it again?
- noway421 10y ago
  Packets can be duplicated by faulty routers and loops in the topology. Happens all the time to me when I'm lost with iptables.
  - ra1n85 10y ago
    Observed this recently with a bug in Cisco's VSS. Some packets were being duplicated, which destroyed TCP throughput.
- azdle 10y ago
  It can also be "duplicated" if you have some sort of acknowledge and retransmission on top of UDP itself. If the ack gets lost on the way back to the original sender it'll send another (probably) identical packet.
- simplehuman 10y ago
  At network level, this is common for broadcast addresses. And also if a switch floods the packet across all its interfaces if it cannot make a routing decision. At app level, this is because the app decided to retry I.e app logic.
- sytelus 10y ago
  A classic example is FPV video from drone. There may be reflections in surroundings and you may get packets multiple times!
  - other_herbert 10y ago
    Right or a problem in something that could retransmit your packet again.. I did that in my office in the early 00s and took down a 100mbps lan from a few boxes retransmitting packets that weren't meant for them <eg>
peterburkimsher 10y ago
Do you want to hear a joke about UDP? You might not get it, but I don't really care.
- colejohnson66 10y ago
  Here's a joke about TCP. Here's a joke about TCP. Here's a joke about TCP. Here's a joke about TCP. Here's a joke about TCP. > ACK
  - deleted 10y ago
    [deleted]
  - deleted 10y ago
    [deleted]
- x0 10y ago
  Do you a hear joke UDP? about
- DonHopkins 10y ago
  OSI jokes work on too many layers.
  - vacri 10y ago
    My favourite version of "stupid user error"/PEBKAC/ID-10-T is 'layer 8 error'...
sydd 10y ago
Ah I love Glen's articles. Ive learned so much from them: game physics, integrators, game loops, game networking,..
- goodells 10y ago
  Yes! His writeups on networked physics were extremely lucid and easy to follow. The way that he focuses on minimizing the bandwidth with bitpacking and goes from something absurdly large to equally absurdly small was a mesmerizing read for me as a programmer just getting started because of an interest in games.
cpncrunch 10y ago
The reality is that these days there generally isn't any packet loss, so UDP vs TCP isn't such an issue as it might have been in the past. In fact TCP has a number of advantages these days such as easier firewall traversal, WebSockets, etc.
- ra1n85 10y ago
  >The reality is that these days there generally isn't any packet loss Do you have any sources/data on that? Genuinely interested where you drew that conclusion from. From my vantage point (anecdotal), ISPs and carriers routinely under-provision, congest peerings, and don't find a problem well until after a number of customers complain.
  - wtallis 10y ago
    Bufferbloat means that at many potential bottlenecks, you don't get any packet loss until well after the point at which your latency-sensitive application (be it TCP or UDP) has given up hope. Major peering points are pretty much the only place where you won't see >1second queues when congested, simply because they can't afford buffers that deep for the rates they operate at. But for ordinary everyday occurrences, your congestion will be somewhere around the last mile, where you cannot expect packets to be dropped within a reasonable timeframe.
    - DonHopkins 10y ago
      https://www.bufferbloat.net/projects/bloat/wiki/Introduction https://www.bufferbloat.net/projects/bloat/wiki/Introduction Introduction Bufferbloat is the undesirable latency that comes from a router or other network equipment buffering too much data. It is a huge drag on Internet performance created, ironically, by previous attempts to make it work better. The one-sentence summary is “Bloated buffers lead to network-crippling latency spikes.” The bad news is that bufferbloat is everywhere, in more devices and programs than you can shake a stick at. The good news is, bufferbloat is now, after 4 years of research, development and deployment, relatively easy to fix. See: fq_codel: wiki. The even better news is that fixing it may solve a lot of the service problems now addressed by bandwidth caps and metering, making the Internet faster and less expensive for both users and providers. The introduction below to the problem is extremely old, and we recommend learning about bufferbloat via van jacobson’s fountain model instead. Although the traffic analogy is close to what actually happens… in the real world, you can’t evaporate the excessive cars on the road, which is what we actually do with systems like fq_codel wiki. Still, onward to the blow. [...]
dogismycopilot 10y ago
That's a pretty obnoxious 'donate' button. They can't make it even larger?
- _coldfire 10y ago
  Only if you donate more
- bschwindHN 10y ago
  The guy is providing valuable information for free, he can make his donate button as large as he wants.
  - Kiro 10y ago
    Yeah, sure. He can put the Donate button all over the screen if he wants but then no-one would donate. I also think it's obnoxious.
dpeck 10y ago
reminds me of this old article we had as assigned reading during a networking class in school. The Internet Sucks: Or, What I Learned Coding X-Wing vs. TIE Fighter (http://www.gamasutra.com/view/feature/131781/the_internet_sucks_or_what_i_.php?print=1 http://www.gamasutra.com/view/feature/131781/the_internet_su...).
josephg 10y ago
Does anyone know if SCTP is suitable / in use by any games? It supports streams to work around the head-of-the-line blocking problem TCP runs into and it also supports opt-in unreliable delivery for game data. On the surface seems ideal for games, though I don't know if its getting much actual use.
- wtallis 10y ago
  Nobody who's in it for the money wants to be the first one to prove that SCTP can be deployed at scale. Application developers don't want to have to sort out getting a working SCTP implementation installed and configured on the proprietary operating systems they target, and past the firewalls of all the bottom of the barrel consumer routers.
  - fulafel 10y ago
    You wouldn't use SCTP on top of IP, you would use on top of UDP and implemented in userspace. Browsers do it this way (in WebRTC data channels).
z3t4 10y ago
I think tcp have an unfair reputation. Our networks are better now then 30 years ago ... Worst case latency for tcp is like 3 seconds, compared to the packet never arriving. The trick is to hide the lag with animations. I think google, facebook, and world of warcraft use tcp for their real time apps !?
- amaranth 10y ago
  3 seconds is 2900ms too long, a game would typically rather lose the packet rather than have it delay all transmission for that long. World of Warcraft is a relatively simple game that as far as I know is heavily client authoritative so the server is mostly just serving content and telling other clients the result of your actions. In a game like Call of Duty, while they have some form of lag compensation, if you had all the players in your game freeze for 3 seconds while you shoot them the server will disregard your actions as impossible as the players weren't actually there. It would have been better for you to see the players slightly out of sync or have them teleport a short distance by just losing that one update and continuing to get the rest on time.
- Thaxll 10y ago
  Google is no real-time VS a FPS game running @60hz.
- bschwindHN 10y ago
  > Our networks are better now then 30 years ago You mean the networks in particular that you use? There's a lot of varied network equipment out there. Glenn wrote this article at least 8 years ago and for the types of games he works on (realtime 64-player online FPSes), I'm sure TCP still wouldn't work as well as a UDP solution. > The trick is to hide the lag with animations That's an understatement. A significant amount of work needs to be done with client-side prediction in order to give the appearance of playing in real time consistently with other players. > I think google, facebook, and world of warcraft use tcp for their real time apps Which real time apps are you referring to? In the context of this article, Glenn is talking about soft realtime systems (30 hz updates or more). There aren't any user-facing apps from facebook or google that I'm aware of that I would call "real time". For WoW, timing is less critical than an FPS and obviously TCP works well for them.
the_arun 10y ago
There was a similar discussion in Hackernews earlier - https://news.ycombinator.com/item?id=7507377 https://news.ycombinator.com/item?id=7507377
j_s 10y ago
This feels like a nice introduction. Anyone here have any experience using QUIC in any application of their own? The custom congestion control makes me wonder if it only works alongside TCP traffic - once everything goes QUIC, then what happens? I looked for a bit about the story in ancient history of some blazing fast server OS TCP implementation that broke the rules so it fell over when more than one server was on the network, but couldn't find it.
- CobrastanJorji 10y ago
  QUIC seems like it'd be kind of neat for videogame networking. Does anyone know whether one can get a QUIC channel to deliver datagram-style, as in unreliable and no retransmitting? It'd be really cool if you could set up one QUIC connection that could work across IP address changes and then use it for N reliable TCP-like streams and M datagram-style game update packet channels. If you could set up that sort of thing from JavaScript in a web browser, that'd be straight up amazing.
- deleted 10y ago
  [deleted]
savad 10y ago
good introduction about tcp and udp
ozy 10y ago
"TCP has an option you can set that fixes this behavior called TCP_NODELAY" That fixes nothing. Now you are sending too many small packets using too many syscalls. Just like UDP, buffer in user space, send in one go. If you do that, TCP_NODELAY makes no difference. (The exception is user input, if you want to send those as they happen, use TCP_NODELAY, but think about the why ... it has little to do with what this article is talking about.) Games likely send data only around 25 times per second, and ping is likely < 50ms. Waiting on a dropped packet and the delay it causes is unnoticeable. Added that clients will need some kind of latency compensation and prediction, independent of the TCP/UDP choice. Delays and then bursts of 100ms or such are doable. The problem starts when the connection stalls for more than 100ms, especially in high bandwidth games. During the stall both behave the same. After the stall, TCP will be playing catchup and wasting more time receiving outdated data, and handing it to user space in order. UDP just passes on what is received, with a lot less catching up, and maybe some dropping of packets. But gameplay has been degraded in both cases. UDP just has a higher chance of masking and shortening degradation more. Anything more than that is basically cargo-culting, like this article.
- deburo 10y ago
  I will preface this with the fact that the author worked on at least one such FPS game (Titanfall 2) and probably know what he is talking about. He offered one solution to fix one behavior of TCP, the behavior being that if the data written to the buffer is not big enough, TCP might hold it. Setting TCP_NODELAY forces the protocol to avoid holding data. But this is only written as a note to coders who might think TCP_NODELAY will fix TCP for action games, but it doesn't actually, because the protocol has other characteristics that are undesirable in that type of games. Moreover, you write this: waiting on a dropped packet and the delay it causes is unnoticeable. Unless you have a FPS game with TCP as its network protocol to validate this claim, I call bull. Many network programmers recommend against TCP. This is probably not simple cargo-culting.
  - ozy 10y ago
    "TCP_NODELAY forces the protocol to avoid holding data" That is the misunderstanding. It will send one non full packet, and only then "hold on to data" until the next packet is a full packet. But that condition will reset when all data has been acknowledged. If you buffer in userspace and write data in one go, at 25 network fps, you basically never trigger the condition where TCP_NODELAY makes a difference. It is not nagle's algorithm, but a variant of minshall-nagle, what modern systems use. There is a reason why FPS games use UDP, but that discussion should not start with TCP_NODELAY. That one is misused enough ... often when it fixes anything, it is because it masks a real underlying problem, had you fixed that problem, your system would react much better under stress or bad networks.
rstuart4133 10y ago
The entire article makes me shudder in disbelief. It's aimed at people who don't know the difference between UDP and TCP (and possibly wet string). Yet he recommends they implement their own reliably protocol over UDP, and they avoid TCP because it's better to implement your own QoS? Why not add obtaining PhD in quantum mechanics just to round it out? It wouldn't alter the odds of pulling it off overly.
- coffeeaddicted 10y ago
  You don't have to invent a protocol when using UDP, you can still use a library like enet or raknet which already do that for you. And if it's the better choice it's the better choice and it's not something a programmer can't learn to do.
- user5994461 10y ago
  Agree. For people who are not familiar with TCP and UDP, the answer should be to ALWAYS use TCP. The only exception is some subset of games (FPS, RTS), that this article is exclusively intended for. Even there, for the average user who just wanna make a test game project, TCP may be easier to use and program around. For simple games (e.g. Connect 4), TCP is a better option either way.
  - theWatcher37 10y ago
    In the article he clearly states this is for things like syncing physics in BF4 or halo.
- greyskull 10y ago
  It's framed in the context of building networking for action games that are sensitive to networking. Fourth paragraph: The choice you make depends entirely on what sort of game you want to network. So from this point on, and for the rest of this article series, I’m going to assume you want to network an action game. You know games like Halo, Battlefield 1942, Quake, Unreal, CounterStrke, Team Fortress and so on.
- exDM69 10y ago
geokon 10y ago
What are some alternative protocols? It's been a while since my networking class, but if I remember correctly with UDP you have some serious issues where you can end up clobbering your network, filling up buffers in the middle and dropping tons of packets. The lack of congestion control is a huge no-no. For instance in the example he gives, sure you can tolerate dropped packets for player-position data, but how do you know if you can tolerate sending at 10Hz 100Hz 1000Hz? Even with TCP you can't (I think....) programmatically adapt to the size of your pipe. That's kinda abstracted away for you so that you just say "send file A to B" and it does it for you Are you supposed to write your own congestion control in userland???? Seems like this should be a solved problem
- daurnimator 10y ago
  DCCP and SCTP are the two main alternative transport protocols on top of IP - UDP: unordered unreliable datagrams - DCCP: unordered unreliable datagrams with congestion control - SCTP: optionally-ordered reliable datagrams with congestion control - TCP: ordered reliable stream with congestion control SCTP is the most widely used after TCP and UDP: it's often the transport used in phone networks (very popular with Ericsson); but also now with WebRTC (which uses SCTP over DTLS over UDP). The main issue being that consumer firewalls usually only support NAT for TCP, and limitedly for UDP (this is one of the 'other' reasons to look forward to IPv6)
jokoon 10y ago
In short, TCP will work hard to deliver 100% of the packets. So when a packet is lost, TCP asks to re-send the packet. This is fine to display a webpage or send a file, but it can't be tolerated in games where time continuity matters. I think it's the same issue in VOIP and video conferencing too.
zubspace 10y ago
I tried to build an UDP library once in C# with different methods (BeginReceiveFrom, ReceiveFromAsync, ReceiveFrom). You learn a ton and it's quite interesting. My goal was to recreate something similar to .NET remoting based on UDP. But be aware that it's a daunting task, because there are so many things you need to handle all together: lost packets, reordered packets, duplicate packets, connection handshakes, session handling, reliable/unreliable channels, packet resends, random disconnects, reconnects, network congestion, spoofing, protocol hacking attempts, dos-attacks, banning, encryption, etc... If you're writing an UDP library, you also need to think of performance, object pooling, connection buffers, threading/async issues and on top of that you also want to provide a nice API to the outside world for the client and server... Well, it gets messy... If you're into this thing, I can advice you to look at haxe libraries. Learned a lot of them. There are very simple, idiomatic server/client-side implementations which are easy to follow, even if you don't know haxe [1][2]. [1]: https://github.com/svn2github/haxe/blob/master/std/neko/net/ServerLoop.hx https://github.com/svn2github/haxe/blob/master/std/neko/net/... [2]: https://github.com/svn2github/haxe/blob/master/std/neko/net/ThreadServer.hx https://github.com/svn2github/haxe/blob/master/std/neko/net/...
JohnStrange 10y ago
What about UDT? I never tried it but always wondered about it's benefits/disadvantages.
bsaul 10y ago
Anyone here knows what protocol should be used for mobile multiplayer games , provided they are real time and not turn by turn ? (think people playing on a decent 3g connexion). I think you can't send a udp paquet to a phone because the carrier will block it, but i'm not sure.
- jfries 10y ago
  Yes an update on this topic for the mobile world would be very interesting.
fivesigma 10y ago
A huge problem with TCP wrt gaming is the default ACK frequency on Windows which is set to 2. This effectively almost doubles the latency of game connections (sending/receiving a lot of time-sensitive small packets). It can be changed with a registry setting (TcpAckFrequency) but you can't expect even a significant fraction of your users to do that. Why this isn't a per-connection option sort of like TCP_NODELAY is beyond me.
johnbellone 10y ago
If you want to get familiar with TCP/UDP and are not gun-shy with C I would suggest the Pocket Socket Guide (now Practical Guide to TCP sockets)[1]. I have a really old edition, but it ages extremely well, and its one of the books I always use to refresh myself on network programming basics. [1]: https://www.amazon.com/gp/product/0123745403/ref=ox_sc_act_title_1?ie=UTF8&psc=1&smid=ATVPDKIKX0DER https://www.amazon.com/gp/product/0123745403/ref=ox_sc_act_t...

453 points by colinprince 10y ago

I thought that it was somebody who has recently discovered the existence of UDP protocol and brags about that to the world, but from skimming the article, it actually has some non-trivial remarks about UDP and TCP. BTW, the article's view angle is multiplayer game programming.

exDM69 10y ago
This guy (Glenn Gaffer) has been writing network protocols for games for years and shipped popular titles. On his site you can also find an irate rant about people with no experience dismissing his claims as "reimplementing TCP". It has good points but it's a frustrated rant (reader beware). But it addresses a lot of the commonly believed fallacies. His latest project is an open source library for game networking over UDP: https://github.com/networkprotocol/libyojimbo https://github.com/networkprotocol/libyojimbo And he is correct, for "real time" games you must use UDP or you will be in trouble in real world networking conditions. When everything runs smoothly, TCP and UDP work almost identically but when packet loss occurs, TCP will make things worse.
- jay_kyburz 10y ago
  Glenn _Fiedler_ - Gaffer is his online name / handle. Worked with Glenn on Freedom Force and Tribes. He knows his stuff!

techman9 10y ago

I appreciate the first user comment on this article: "I didn’t care for this article 7 months ago. I regret it. My whole game is useless now. I should have go with udp."

aakilfernandes 10y ago

Another important difference: UDP can't be used in the browser =(

oszione 10y ago
With the exception of the QUIC protocol. * TLS over 443/udp
rdw 10y ago
WebRTC is UDP. It's not easy, but it's there!
- foota 10y ago
  nitpick: It can be either
  - arkitectual 10y ago
    It is possible to use only one or the other.
semicolon_storm 10y ago
Couldn't you potentially use something like WebRTC to get UDP functionality in the browser?
simplehuman 10y ago
Well, neither can TCP.
- jensvdh 10y ago
  HTTP is built on top of TCP. Websockets are TCP connections.
  - simplehuman 10y ago
    So? TCP cannot be used directly in the browser by applications authors. It's an abstraction.

mjevans 10y ago

The main difference between TCP and UDP, as this programmer discovered, relates to quality of realtime service. Times to use UDP over TCP * When you need the lowest latency * When LATE data is worse than GAPS (loss of) in data. * When you want to implement your own form of error correction to handle late/missing/mangled data. TCP is best when * You need all of the data to arrive, period. * You want to automatically make a rough best estimate use of the available connection for /rate/ of transfer. For a videogame, having a general control channel to the co-ordination server in TCP is fine. Having interactive asset downloads (level setup) over TCP is fine. Interactive player movements /probably/ should be UDP. Very likely with a mix of forward error correction and major snapshot syncs for critical data (moving entity absolute location, etc).

exDM69 10y ago
Another article on this site addresses TCP plus UDP mix. The tl;dr is that they should not be used at the same time because TCP retransmission will interfere with UDP (and routers give TCP priority). As long as TCP happens outside of the "hot" loop, it's fine.
- wtallis 10y ago
  > (and routers give TCP priority). What routers? Is this something that stupid routers did in the '90s and people are still worried about, or are modern routers actually behaving this way? (And if so, why?) TCP retransmission interfering with UDP sounds like something that shouldn't be a problem on any sane network, especially if the application authors are smart and use something like LEDBAT for their TCP traffic when they're also doing latency-sensitive UDP communication.
  - vvanders 10y ago
    You'd be shocked at the things home routers do. There's a reason most people don't try to implement NAT traversal and use common middleware. Since UDP isn't as common of a protocol I can certainly see some routers treating it a second class traffic when it comes to packet shaping.

mikepurvis 10y ago

The recommendation to avoid TCP altogether is surprising to me. Having encountered a number of video-conferencing systems which are in a similar space, it seems pretty standard to have separate real-time and control sockets on UDP and TCP respectively. I skimmed the linked paper and didn't find it conclusive; can someone summarize how it is that having a TCP socket can affect UDP traffic on the same interface? All that said, I certainly see the argument for an all-UDP protocol in terms of defining your own retransmission approach, or attempting to avoid it altogether with forward error correction or whatever.

blackflame7000 10y ago
I too was wondering how TCP and UDP can affect one another on the same interface and the explanation I read is that at the lowest level both UDP and TCP segment data into packets and place them onto the same network adapter's queue which can affect they way they are transmitted under congestion depending on the device/drivers.
- wtallis 10y ago
  That's not an explanation. That's just pointing in the general direction of where to look for an explanation.
  - blackflame7000 10y ago
    Do you have a better explanation to offer because I'd love to learn? Not sure why I deserve a downvote for offering a potential solution when you with 6000 karma offer nothing.
    - wtallis 10y ago
      I think the claim was far too broad to actually be entirely true, and the evidence was both narrow and weak. The 1997 paper only shows that certain simulation parameters can produce bad packet loss for UDP traffic. But those parameters are almost all irrelevant to today's Internet: MTU is now 1500B not 512B, TCP congestion control is usually something like CUBIC rather than RENO, buffers are never a mere 16 packets long, and TCP traffic is more often relatively short-lived HTTP connections, not long-running FTP. The paper is just another piece of evidence that TCP synchronization sucks. You are very unlikely to encounter such perfect TCP synchronization on today's Internet, and even if you did, the dynamics of introducing some UDP flows to that situation would be different.

milansuk 10y ago

> they may arrive out of order, be duplicated, or not arrive at all! This is first time what I read that datagram can be duplicated. Is it true? It's duplicated by network or does it mean that peer send it again?

noway421 10y ago
Packets can be duplicated by faulty routers and loops in the topology. Happens all the time to me when I'm lost with iptables.
- ra1n85 10y ago
  Observed this recently with a bug in Cisco's VSS. Some packets were being duplicated, which destroyed TCP throughput.
azdle 10y ago
It can also be "duplicated" if you have some sort of acknowledge and retransmission on top of UDP itself. If the ack gets lost on the way back to the original sender it'll send another (probably) identical packet.
simplehuman 10y ago
At network level, this is common for broadcast addresses. And also if a switch floods the packet across all its interfaces if it cannot make a routing decision. At app level, this is because the app decided to retry I.e app logic.
sytelus 10y ago
A classic example is FPV video from drone. There may be reflections in surroundings and you may get packets multiple times!
- other_herbert 10y ago
  Right or a problem in something that could retransmit your packet again.. I did that in my office in the early 00s and took down a 100mbps lan from a few boxes retransmitting packets that weren't meant for them <eg>

peterburkimsher 10y ago

Do you want to hear a joke about UDP? You might not get it, but I don't really care.

colejohnson66 10y ago
Here's a joke about TCP. Here's a joke about TCP. Here's a joke about TCP. Here's a joke about TCP. Here's a joke about TCP. > ACK
- deleted 10y ago
  [deleted]
- deleted 10y ago
  [deleted]
x0 10y ago
Do you a hear joke UDP? about
DonHopkins 10y ago
OSI jokes work on too many layers.
- vacri 10y ago
  My favourite version of "stupid user error"/PEBKAC/ID-10-T is 'layer 8 error'...

sydd 10y ago

Ah I love Glen's articles. Ive learned so much from them: game physics, integrators, game loops, game networking,..

goodells 10y ago
Yes! His writeups on networked physics were extremely lucid and easy to follow. The way that he focuses on minimizing the bandwidth with bitpacking and goes from something absurdly large to equally absurdly small was a mesmerizing read for me as a programmer just getting started because of an interest in games.

cpncrunch 10y ago

The reality is that these days there generally isn't any packet loss, so UDP vs TCP isn't such an issue as it might have been in the past. In fact TCP has a number of advantages these days such as easier firewall traversal, WebSockets, etc.

ra1n85 10y ago
>The reality is that these days there generally isn't any packet loss Do you have any sources/data on that? Genuinely interested where you drew that conclusion from. From my vantage point (anecdotal), ISPs and carriers routinely under-provision, congest peerings, and don't find a problem well until after a number of customers complain.
- wtallis 10y ago
  Bufferbloat means that at many potential bottlenecks, you don't get any packet loss until well after the point at which your latency-sensitive application (be it TCP or UDP) has given up hope. Major peering points are pretty much the only place where you won't see >1second queues when congested, simply because they can't afford buffers that deep for the rates they operate at. But for ordinary everyday occurrences, your congestion will be somewhere around the last mile, where you cannot expect packets to be dropped within a reasonable timeframe.
  - DonHopkins 10y ago
    https://www.bufferbloat.net/projects/bloat/wiki/Introduction https://www.bufferbloat.net/projects/bloat/wiki/Introduction Introduction Bufferbloat is the undesirable latency that comes from a router or other network equipment buffering too much data. It is a huge drag on Internet performance created, ironically, by previous attempts to make it work better. The one-sentence summary is “Bloated buffers lead to network-crippling latency spikes.” The bad news is that bufferbloat is everywhere, in more devices and programs than you can shake a stick at. The good news is, bufferbloat is now, after 4 years of research, development and deployment, relatively easy to fix. See: fq_codel: wiki. The even better news is that fixing it may solve a lot of the service problems now addressed by bandwidth caps and metering, making the Internet faster and less expensive for both users and providers. The introduction below to the problem is extremely old, and we recommend learning about bufferbloat via van jacobson’s fountain model instead. Although the traffic analogy is close to what actually happens… in the real world, you can’t evaporate the excessive cars on the road, which is what we actually do with systems like fq_codel wiki. Still, onward to the blow. [...]

dogismycopilot 10y ago

That's a pretty obnoxious 'donate' button. They can't make it even larger?

_coldfire 10y ago
Only if you donate more
bschwindHN 10y ago
The guy is providing valuable information for free, he can make his donate button as large as he wants.
- Kiro 10y ago
  Yeah, sure. He can put the Donate button all over the screen if he wants but then no-one would donate. I also think it's obnoxious.

dpeck 10y ago

reminds me of this old article we had as assigned reading during a networking class in school. The Internet Sucks: Or, What I Learned Coding X-Wing vs. TIE Fighter (http://www.gamasutra.com/view/feature/131781/the_internet_sucks_or_what_i_.php?print=1 http://www.gamasutra.com/view/feature/131781/the_internet_su...).

josephg 10y ago

Does anyone know if SCTP is suitable / in use by any games? It supports streams to work around the head-of-the-line blocking problem TCP runs into and it also supports opt-in unreliable delivery for game data. On the surface seems ideal for games, though I don't know if its getting much actual use.

wtallis 10y ago
Nobody who's in it for the money wants to be the first one to prove that SCTP can be deployed at scale. Application developers don't want to have to sort out getting a working SCTP implementation installed and configured on the proprietary operating systems they target, and past the firewalls of all the bottom of the barrel consumer routers.
- fulafel 10y ago
  You wouldn't use SCTP on top of IP, you would use on top of UDP and implemented in userspace. Browsers do it this way (in WebRTC data channels).

z3t4 10y ago

I think tcp have an unfair reputation. Our networks are better now then 30 years ago ... Worst case latency for tcp is like 3 seconds, compared to the packet never arriving. The trick is to hide the lag with animations. I think google, facebook, and world of warcraft use tcp for their real time apps !?

amaranth 10y ago
3 seconds is 2900ms too long, a game would typically rather lose the packet rather than have it delay all transmission for that long. World of Warcraft is a relatively simple game that as far as I know is heavily client authoritative so the server is mostly just serving content and telling other clients the result of your actions. In a game like Call of Duty, while they have some form of lag compensation, if you had all the players in your game freeze for 3 seconds while you shoot them the server will disregard your actions as impossible as the players weren't actually there. It would have been better for you to see the players slightly out of sync or have them teleport a short distance by just losing that one update and continuing to get the rest on time.
Thaxll 10y ago
Google is no real-time VS a FPS game running @60hz.
bschwindHN 10y ago
> Our networks are better now then 30 years ago You mean the networks in particular that you use? There's a lot of varied network equipment out there. Glenn wrote this article at least 8 years ago and for the types of games he works on (realtime 64-player online FPSes), I'm sure TCP still wouldn't work as well as a UDP solution. > The trick is to hide the lag with animations That's an understatement. A significant amount of work needs to be done with client-side prediction in order to give the appearance of playing in real time consistently with other players. > I think google, facebook, and world of warcraft use tcp for their real time apps Which real time apps are you referring to? In the context of this article, Glenn is talking about soft realtime systems (30 hz updates or more). There aren't any user-facing apps from facebook or google that I'm aware of that I would call "real time". For WoW, timing is less critical than an FPS and obviously TCP works well for them.

the_arun 10y ago

There was a similar discussion in Hackernews earlier - https://news.ycombinator.com/item?id=7507377 https://news.ycombinator.com/item?id=7507377

j_s 10y ago

This feels like a nice introduction. Anyone here have any experience using QUIC in any application of their own? The custom congestion control makes me wonder if it only works alongside TCP traffic - once everything goes QUIC, then what happens? I looked for a bit about the story in ancient history of some blazing fast server OS TCP implementation that broke the rules so it fell over when more than one server was on the network, but couldn't find it.

CobrastanJorji 10y ago
QUIC seems like it'd be kind of neat for videogame networking. Does anyone know whether one can get a QUIC channel to deliver datagram-style, as in unreliable and no retransmitting? It'd be really cool if you could set up one QUIC connection that could work across IP address changes and then use it for N reliable TCP-like streams and M datagram-style game update packet channels. If you could set up that sort of thing from JavaScript in a web browser, that'd be straight up amazing.
deleted 10y ago
[deleted]

savad 10y ago

good introduction about tcp and udp

ozy 10y ago

"TCP has an option you can set that fixes this behavior called TCP_NODELAY" That fixes nothing. Now you are sending too many small packets using too many syscalls. Just like UDP, buffer in user space, send in one go. If you do that, TCP_NODELAY makes no difference. (The exception is user input, if you want to send those as they happen, use TCP_NODELAY, but think about the why ... it has little to do with what this article is talking about.) Games likely send data only around 25 times per second, and ping is likely < 50ms. Waiting on a dropped packet and the delay it causes is unnoticeable. Added that clients will need some kind of latency compensation and prediction, independent of the TCP/UDP choice. Delays and then bursts of 100ms or such are doable. The problem starts when the connection stalls for more than 100ms, especially in high bandwidth games. During the stall both behave the same. After the stall, TCP will be playing catchup and wasting more time receiving outdated data, and handing it to user space in order. UDP just passes on what is received, with a lot less catching up, and maybe some dropping of packets. But gameplay has been degraded in both cases. UDP just has a higher chance of masking and shortening degradation more. Anything more than that is basically cargo-culting, like this article.

deburo 10y ago
I will preface this with the fact that the author worked on at least one such FPS game (Titanfall 2) and probably know what he is talking about. He offered one solution to fix one behavior of TCP, the behavior being that if the data written to the buffer is not big enough, TCP might hold it. Setting TCP_NODELAY forces the protocol to avoid holding data. But this is only written as a note to coders who might think TCP_NODELAY will fix TCP for action games, but it doesn't actually, because the protocol has other characteristics that are undesirable in that type of games. Moreover, you write this: waiting on a dropped packet and the delay it causes is unnoticeable. Unless you have a FPS game with TCP as its network protocol to validate this claim, I call bull. Many network programmers recommend against TCP. This is probably not simple cargo-culting.
- ozy 10y ago
  "TCP_NODELAY forces the protocol to avoid holding data" That is the misunderstanding. It will send one non full packet, and only then "hold on to data" until the next packet is a full packet. But that condition will reset when all data has been acknowledged. If you buffer in userspace and write data in one go, at 25 network fps, you basically never trigger the condition where TCP_NODELAY makes a difference. It is not nagle's algorithm, but a variant of minshall-nagle, what modern systems use. There is a reason why FPS games use UDP, but that discussion should not start with TCP_NODELAY. That one is misused enough ... often when it fixes anything, it is because it masks a real underlying problem, had you fixed that problem, your system would react much better under stress or bad networks.

rstuart4133 10y ago

The entire article makes me shudder in disbelief. It's aimed at people who don't know the difference between UDP and TCP (and possibly wet string). Yet he recommends they implement their own reliably protocol over UDP, and they avoid TCP because it's better to implement your own QoS? Why not add obtaining PhD in quantum mechanics just to round it out? It wouldn't alter the odds of pulling it off overly.

coffeeaddicted 10y ago
You don't have to invent a protocol when using UDP, you can still use a library like enet or raknet which already do that for you. And if it's the better choice it's the better choice and it's not something a programmer can't learn to do.
user5994461 10y ago
Agree. For people who are not familiar with TCP and UDP, the answer should be to ALWAYS use TCP. The only exception is some subset of games (FPS, RTS), that this article is exclusively intended for. Even there, for the average user who just wanna make a test game project, TCP may be easier to use and program around. For simple games (e.g. Connect 4), TCP is a better option either way.
- theWatcher37 10y ago
  In the article he clearly states this is for things like syncing physics in BF4 or halo.
greyskull 10y ago
It's framed in the context of building networking for action games that are sensitive to networking. Fourth paragraph: The choice you make depends entirely on what sort of game you want to network. So from this point on, and for the rest of this article series, I’m going to assume you want to network an action game. You know games like Halo, Battlefield 1942, Quake, Unreal, CounterStrke, Team Fortress and so on.
exDM69 10y ago

geokon 10y ago

What are some alternative protocols? It's been a while since my networking class, but if I remember correctly with UDP you have some serious issues where you can end up clobbering your network, filling up buffers in the middle and dropping tons of packets. The lack of congestion control is a huge no-no. For instance in the example he gives, sure you can tolerate dropped packets for player-position data, but how do you know if you can tolerate sending at 10Hz 100Hz 1000Hz? Even with TCP you can't (I think....) programmatically adapt to the size of your pipe. That's kinda abstracted away for you so that you just say "send file A to B" and it does it for you Are you supposed to write your own congestion control in userland???? Seems like this should be a solved problem

daurnimator 10y ago
DCCP and SCTP are the two main alternative transport protocols on top of IP - UDP: unordered unreliable datagrams - DCCP: unordered unreliable datagrams with congestion control - SCTP: optionally-ordered reliable datagrams with congestion control - TCP: ordered reliable stream with congestion control SCTP is the most widely used after TCP and UDP: it's often the transport used in phone networks (very popular with Ericsson); but also now with WebRTC (which uses SCTP over DTLS over UDP). The main issue being that consumer firewalls usually only support NAT for TCP, and limitedly for UDP (this is one of the 'other' reasons to look forward to IPv6)

jokoon 10y ago

In short, TCP will work hard to deliver 100% of the packets. So when a packet is lost, TCP asks to re-send the packet. This is fine to display a webpage or send a file, but it can't be tolerated in games where time continuity matters. I think it's the same issue in VOIP and video conferencing too.

zubspace 10y ago

I tried to build an UDP library once in C# with different methods (BeginReceiveFrom, ReceiveFromAsync, ReceiveFrom). You learn a ton and it's quite interesting. My goal was to recreate something similar to .NET remoting based on UDP. But be aware that it's a daunting task, because there are so many things you need to handle all together: lost packets, reordered packets, duplicate packets, connection handshakes, session handling, reliable/unreliable channels, packet resends, random disconnects, reconnects, network congestion, spoofing, protocol hacking attempts, dos-attacks, banning, encryption, etc... If you're writing an UDP library, you also need to think of performance, object pooling, connection buffers, threading/async issues and on top of that you also want to provide a nice API to the outside world for the client and server... Well, it gets messy... If you're into this thing, I can advice you to look at haxe libraries. Learned a lot of them. There are very simple, idiomatic server/client-side implementations which are easy to follow, even if you don't know haxe [1][2]. [1]: https://github.com/svn2github/haxe/blob/master/std/neko/net/ServerLoop.hx https://github.com/svn2github/haxe/blob/master/std/neko/net/... [2]: https://github.com/svn2github/haxe/blob/master/std/neko/net/ThreadServer.hx https://github.com/svn2github/haxe/blob/master/std/neko/net/...

JohnStrange 10y ago

What about UDT? I never tried it but always wondered about it's benefits/disadvantages.

bsaul 10y ago

Anyone here knows what protocol should be used for mobile multiplayer games , provided they are real time and not turn by turn ? (think people playing on a decent 3g connexion). I think you can't send a udp paquet to a phone because the carrier will block it, but i'm not sure.

jfries 10y ago
Yes an update on this topic for the mobile world would be very interesting.

fivesigma 10y ago

A huge problem with TCP wrt gaming is the default ACK frequency on Windows which is set to 2. This effectively almost doubles the latency of game connections (sending/receiving a lot of time-sensitive small packets). It can be changed with a registry setting (TcpAckFrequency) but you can't expect even a significant fraction of your users to do that. Why this isn't a per-connection option sort of like TCP_NODELAY is beyond me.

johnbellone 10y ago

If you want to get familiar with TCP/UDP and are not gun-shy with C I would suggest the Pocket Socket Guide (now Practical Guide to TCP sockets)[1]. I have a really old edition, but it ages extremely well, and its one of the books I always use to refresh myself on network programming basics. [1]: https://www.amazon.com/gp/product/0123745403/ref=ox_sc_act_title_1?ie=UTF8&psc=1&smid=ATVPDKIKX0DER https://www.amazon.com/gp/product/0123745403/ref=ox_sc_act_t...