13 ms·
Game servers: UDP vs TCP
- lerno 13y agoA brief follow-up here: http://1024monkeys.wordpress.com/2014/04/08/udp-vs-tcp-a-follow-up/ http://1024monkeys.wordpress.com/2014/04/08/udp-vs-tcp-a-fol...
- ColinWright 13y agoWe use TCP for some things, UDP for some things, and our own variant on Reliable UDP for others. We use TCP for sending real-time vessel track data to the central correlation servers, because they need to know if the connection has dropped and change their behavior accordingly. Traffic volume is low, so congestion throttling and re-tries are not a problem. The system is able to cope with reports being delayed and then coming in much later (and catching up) because reports are time-stamped and the correlator works in fuzzy 6D phase-space. We use UDP for sending live radar data because, frankly, if you've missed one sector of data you really, really don't want to delay subsequent sectors while waiting for the data that will be irrelevant in 2 seconds anyway. The compression and encryption schemes are specially crafted to allow for lost packets while still using solid implementations of off-the-shelf algorithms where appropriate, and lovingly hand-crafted algorithms for the non-security critical aspects. And I can't tell you what we use our locally defined reliable UDP for.
- deleted 13y ago[deleted]
- JoeAltmaier 13y agoUsually that's for state messages. The kind of message that defines the meaning of subsequent messages. Like codec parameters, or framing data or something.
- mark_l_watson 13y agoI have used UDP for two major projects: a game engine while working at Angel Studios (networking was tightly coupled with camera and vehicle AI) and for the NMRD project in the late 1980s (NMRD could detect nuclear tests based on seismic waves). I am surprised that UDP is not used more often when the business cost of loss data is small.
- JabavuAdams 13y agoJust use RakNet http://www.jenkinssoftware.com http://www.jenkinssoftware.com I mean if you want to ship a product. If your goal is to learn networking, then by all means, roll your own.
- zurn 13y agoImpressive that he managed to say so much without mentioning head-of-line blocking, which is the main TCP problem with realtime apps. This is not due to congestion control, but because of the ordered byte stream semantics of TCP. You can't deliver a packet until the retransmission dance is done for any previous lost packets. As an aside, these days it's worth noting that a 3G, WLAN or 4G link layer will try very hard not to drop any packets. So when you're doing UDP, watch out for those 40000 millisecond old packets that will burst to your lap once the rain cloud moves away from the cell tower or your co-worker's cocoa blings in the microwave.
- lucb1e 13y agoHe didn't mention the term head-of-line blocking, but the issue was apparent to me as I read it. Perhaps because I have a bit of previous experience with tcp vs udp, but still.
- yvdriess 13y agoI was under the impression that packets behind the blocked TCP packet still sent through the network, relying on the client-side or last-hop to maintain the sequential stream illusion and thus reducing the delay of the subsequent frames to the delay blocked/resent frame. Could with a deeper knowledge confirm or deny this?
- mikeash 13y agoThat's true, but the delay from waiting for the lost packet to be retransmitted is the problem, not a delay for subsequent data after retransmitting the lost packet. Retransmission can take a substantial amount of time.
- eggnet 13y agoYes that is true but maintaining the illusion means that the application can't read the data after the lost packet until the packet is retransmitted and received. So yes, selective acks preserve network bandwidth but don't solve the real time problems. With UDP the client application can receive all packets when they arrive even if there are some missing.
- kayoone 13y agoBiggest problem with UDP: No support at all in modern browsers without plugin. So when doing web games, you can't really get around using Websockets which are TCP based.
- stcredzero 13y agoOr, the biggest problem with web games: no UDP.
- bryanpaluch 13y agoThis will no longer be the case once you can use WebRTC data channels. Its just a matter of making sure your server can support the protocol.
- k__ 13y agoThe data channels run on UDP per default?
- zurn 13y agoYes, DTLS over SCTP over UDP. TCP doesn't really work if you want to do NAT traversal, which WebRTC needs to do because it's aimed at P2P.
- MichaelGG 13y agoAnd NAT traversal only works if you focus on broken NAT implementations. If you have a "secure" NAT like a firewall may implement, then there is no such thing as workable NAT traversal.
- zurn 13y agoYou have that backwards! NATs are tasked to facilitate communications (same as routers). Yes, firewalls can be configured to restrict communications. And you can have a combined NAT+firewall that is so configured. Just likes routers vs firewalls. Routers forward packets as well as they know how, firewalls selectively drop them, router-FW combos can configured either way. Reliance on restrictive centralized firewalls is a pretty 1990s mindset, and doesn't lead to good security outcomes in the current world where users constantly suffle devices between networks, and vpn in to your network, etc. You just end up making your internal network unusable for production work.
- crucio 13y agoI've been hoping someone makes a decent server-side WebRTC peer, meaning that we can use UDP in the browser and really push HTML5 gaming forward. It seems to go completely under the radar with all this talk of audio/video P2P streaming.
- BatFastard 13y agoIf you absolutely positively need to have that lowest lag time between points then UDP is the way to go. However 1) UDP will drop the occasional packet. 2) UDP will NOT work behind many corporate firewalls. 3) UDP connections between users on separate networks will fail due to random configuration issues (I am sure there is a reason, just don't know what it is after years of looking). 4) UDP connections between users on the same DHCP servers requires a different addresses then the global address to make connections. 5) UDP connections can time out if you don't send anything for a few minutes. Easy to fix, hard to find. In the virtual world platform my team wrote, we started using UDP exclusively, then moved to mixed TCP and UDP, then moved to using TCP for everything except P2P voice connections. with UDP I had always dreaded going into see a potential corporate client, and not knowing if my networking was going to make it through the firewall. This is a good discussion of the subject. http://stackoverflow.com/questions/992069/ace-vs-boost-vs-poco http://stackoverflow.com/questions/992069/ace-vs-boost-vs-po...
- paraboul 13y agoThere are no such thing as "UDP connections". UDP is datagram based : you send a packet to an address, either it reaches its destination either it not (regardless of the order). A packet can be dropped in case of network congestion, TTL that reaches 0, etc. That is : One can easily rewrite a TCP-like on top of UDP.
- pjc50 13y agoAlmost every real use of UDP is "pseudo connection based", and lots of NAT systems operate on that basis: that a packet going (host,port) -> (host2,port2) implies both that there might be packets in the reverse direction and further packets between that pair of ports. This isn't baked into the protocol, but in practice everyone relies on it.
- jws 13y agoNAT mapppings will time out.
- BatFastard 13y agoWhile a UDP datagram not a "connection" in the TCP sense. You often send a packet via UDP from a port on one machine, to a different port on the same or different machine. If one of those ports changes, then it effectively "breaks" the connection. Since you are no longer getting data on the port you expect to receive it from. So while a pair of UDP ports are not a "connection", it is often a useful abstraction to treat them as one.
- gz5 13y agoSome RTC implementations (VoIP and video) are using both in order to get through firewalls - try UDP first, resort to TCP (including even HTTP over TCP) if UDP can't connect. The TCP fallback only consistently works well (in terms of QoE) in cases in which the TCP legs are short.
- einhverfr 13y agoThough to some extent it sometimes feels like an RFC 3093[1] solution, as if one is running TCP on HTTP. [1] http://www.ietf.org/rfc/rfc3093.txt http://www.ietf.org/rfc/rfc3093.txt (funny, an April 1 RFC applicable to a serious post on April 1)
- patrickmcmanus 13y agoThe article emphasizes that TCP congestion control can make TCP underperform with respect to the actual available bandwidth. So True and So Frustrating! But it doesn't seem to talk about the virtues of congestion control - the primary one being it prevents the collapse of the Internet. Reacting to packet loss by just trying harder results in nothing but a network filled with doomed packets. This is not theoretical - before Van Jacobson that was the Internet. I'm not defending the status quo nor TCP - TCP has deep problems figuring out what is really loss. and it speeds up too slowly but still doesn't manage to slow down when it should resulting in induced delay. But the mere fact that is worried about congestion (not just unreliability - one result of congestion) and sharing the channel isn't its problem. have a look at QUIC, ledbat, Minion and the recent IETF TAPS BoF for work going on improving transport options in these areas. Its just the wrong takeaway to read that article and say "UDP is better because it doesn't have congestion control" - you need to consider CC in any reliable UDP based transport you roll too. rfc 5405 provides some advice (http://tools.ietf.org/html/rfc5405 http://tools.ietf.org/html/rfc5405) - its a little dated but still useful.
- twic 13y agoExcellent point, awesome RFC! It's worth reproducing this point from the abstract, which makes the same point you do: > Because congestion control is critical to the stable operation of the Internet, applications and upper-layer protocols that choose to use UDP as an Internet transport must employ mechanisms to prevent congestion collapse and to establish some degree of fairness with concurrent traffic. It's a shame that the internet requires this kind of good behaviour from individual hosts, because it leaves it vulnerable to buggy or malicious nodes. But as long as it does, it's essential that users of UDP know about this.
- zurn 13y agoOn the other hand it's very fortunate that the internet architecture is what it is, because the "dumb network, end-to-end protocols" design principle has enabled deployment of new and improved protocols and applications without touching the network routers, which would have been very difficult to coordinate. Might be difficult to believe now but this was a radical idea at the time. (Or at least enabled it for ~25 years until NAT put a damper on it - let's all keep our thumbs up for IPv6).
- ay 13y agoA couple of links that may be of interest on this topic: http://www.ietf.org/proceedings/87/slides/slides-87-tsvarea-1.pdf http://www.ietf.org/proceedings/87/slides/slides-87-tsvarea-... http://www.ietf.org/mail-archive/web/tcmtf/current/maillist.html http://www.ietf.org/mail-archive/web/tcmtf/current/maillist....
- acomjean 13y agoI used to use UDP. Its nice for some things. Things I liked: When you read a UDP message you got the whole message or nothing (its not a stream). Fast. Multicast! Things that challenged us: 64K message size limit (varied slightly from linux/solaris/hpux) you can't tell if the other side is listening or got the message. multicast requires use of certain ip ranges which people seem to forget frequently. I had the UNIX Network Programming by W. Richard Stevens tome I borrowed from my boss. Good for networking.
- JoeAltmaier 13y agoYeah you have to recapitulate some TCP semantics to use UDP effectively. Packet numbering (so you can catch out-of-order packets). Dice-and-reassemble. Congestion control. But the good news is, you CAN do those things yourself. And avoid the TCP lockups, slowdowns and endless retransmissions.
- patrickmcmanus 13y agothe wrs book wears very well. he was a good man. as an aside, its generally a mistake to use IP-Size > PMTU (commonly ~1500).. the result is IP fragmentation and IP stacks commonly have very limited resources devoted to IP reassembly (otherwise its a DoS attack) - so its very easy for your fragmented UDP message to get "lost" even when the network transport does not have an error. When it seems to work ok its basically because your app is the only one doing it.. if it were common practice you would all fight for the same reassembly buffer space and the OS would have to start dropping things. Your application also becomes trivially dosable at low bandwidths by an attacker intentionally using up the reassembly buffer.
- brassybadger 13y agoA very good recap on how networking in Quake3 works without reliable packet delivery: http://trac.bookofhook.com/bookofhook/trac.cgi/wiki/Quake3Networking http://trac.bookofhook.com/bookofhook/trac.cgi/wiki/Quake3Ne... Simple, yet robust.
- dllthomas 13y agoOne side note - what the protocol is on the wire and how you're using it are not quite the same thing. With regard to head-of-line, in principle you can look at later TCP packets before a dropped packet comes in. Of course, correcting framing may be expensive or impossible, and (more importantly, for most applications) if the OS is handling TCP for you you'd have to find some way to cajole it into passing those packets to you.
- twic 13y agoYears ago, i read about some application protocol (for a game, i think) that sat on top of UDP. It packed small messages into datagrams for transmission, and handled acking and redelivery. One of the cute things it did was improve reliability for critical messages by preemptively sending them twice, in successive datagrams. That allowed the protocol to avoid an ack-miss-retransmit cycle for those packets, as long as it was only a single datagram that was dropped.
- sillysaurus3 13y agoIt's the Quake 3 network protocol. Carmack invented it, and that protocol was the first one to solve the UDP ack problem in a way that didn't impact gameplay. It was a huge deal and was probably one of the reasons the Quake 3 engine generated over a billion dollars in revenue.
- lerno 13y agoNot Q3. Q3 has a more elaborate scheme, keeping more updates in the resend buffer.
- sillysaurus3 13y agoYou rock! Thanks for correcting me and for pointing out this wonderful read in your other comment: http://www.gamasutra.com/view/feature/131781/the_internet_sucks_or_what_i_.php?print=1 http://www.gamasutra.com/view/feature/131781/the_internet_su...
- barbs 13y agoA neat write-up of the algorithm used can be found here: http://fabiensanglard.net/quake3/The%20Quake3%20Networking%20Mode.html http://fabiensanglard.net/quake3/The%20Quake3%20Networking%2...
- lerno 13y agoX-Wing vs Tie Fighter, a classic read on networking over the internet: http://www.gamasutra.com/view/feature/131781/the_internet_sucks_or_what_i_.php http://www.gamasutra.com/view/feature/131781/the_internet_su...
- rwaliany 13y agoBe weary of buffer overruns when using a language like C :).
- dllthomas 13y agoWhen you're weary of buffer overruns and can't be wary of buffer overruns, you stop using C.
- stcredzero 13y agoSomeone should write a series of articles about this issue from the POV of Javascript/browser clients. There, UDP isn't available at all, and we also have to deal with quite a bit of pain in terms of differences between platform implementations.
- peterwwillis 13y agoWeb browsers were created to browse text files using hyperlinks. If you want to do advanced networking with a browser, use a plugin like Java.
- stcredzero 13y agoWeb browsers were created to browse text files using hyperlinks. If you want to do advanced networking with a browser, use a plugin like Java. Yup, it's April Fools on the Internet, and everyone's a comic genius!
- scott_karana 13y agoHow is that a joke? It's unpopular, but entirely factual.
- stcredzero 13y agoIt works technically, but the main reason for using Javascript is that distribution and discovery is fantastic and requires no install. Java plugins have a bad reputation in terms of security and require an additional install. Using a Java plugin would take away any distribution and discovery advantage of the browser platform.
- scott_karana 13y agoYeah, I completely agree, but it's the only option for UDP. :-)
- jamesaguilar 13y ago
- callesgg 13y agoBack when i did this kind of stuff i used both of them udp for nonesential stuff like player position. Where a lost packet did not matter, there would still be a new player position packet a half a second later. Then for chat and score and stuff like that tcp was used.
- zurn 13y agoFor a realtime game, player position would be kind of essential, no? "move here unreliably" "shoot at x reliably" Mixing reliable and unreliable updates for game logic would seem to result in a lot of complexity as things can be out of sync in a variety of ways now.
- forgottenpass 13y agoFor a realtime game, player position would be kind of essential, no? Yes, but UDP is choosing unreliable transport, not throwing away data entirely. No matter the transport, once you start dropping a large enough number of packets it will degrade (or break) the experience. The use case being discussed is when getting the next packet fast is more important than the overhead of TCP. You can build retries or various other forms of reliability into your UDP protocol when outdated information is still useful. Mixing reliable and unreliable updates for game logic would seem to result in a lot of complexity Yes and no. It's complex because game state is complex to begin with, but less daunting than you may think because you can compartmentalize to different components of the game. You're not likely to using multiple transports for the same information. See the parent's chat vs player position example.
- Borogove 13y agoObsolete player position data is worse than no player position data.
- fulafel 13y agoThe Quake 3 writeup cites exactly this problem as the reason Carmack abandoned the reliable+unreliable combination: "His next iteration involved using UDP with both reliable and unreliable data, pretty much what many would consider a standard networking architecture. However standard mixed reliabled/unreliable implementations tend to generate very hard to find bugs, e.g. sequencing errors where guaranteed messages referenced entities altered through unreliable messages." (from http://fabiensanglard.net/quake3/The%20Quake3%20Networking%20Mode.html http://fabiensanglard.net/quake3/The%20Quake3%20Networking%2... )
- cinskiy 13y agoThis article is boss. TCP vs. UDP holywar is not going to end any time soon, it's nice that some people can actually explain the differences in simple manner.
- theswan 13y agoIn re the sections on "hiding the lag" - Bungie (the guys behind the original Halo games) have a pretty cool presentation on how they hid lag in Halo Reach. http://downloads.bungie.net/presentations/David_Aldridge_Programming_Gameplay_Networking_Halo_final_pub_without_video.pptx http://downloads.bungie.net/presentations/David_Aldridge_Pro...
- snake_plissken 13y agoI don't get the bottom line part: how can you send something via "HTTP"? What method of transport layer is being used?
- gabipurcaru 13y agonot an expert, but as far as I know, HTTP is a layer over TCP
- snake_plissken 13y agoYup that's what I thought as well. After looking into it some more, I think HTTP/S implies you are using one of the transport layers; conversely, you can directly access the transport layer without using one of the traditional application layers. But that doesn't completely mesh with the parts about modifying TCP or UDP.
- hawleyal 13y agoThe HTTP service sends by responding to a client request. For psuedo-streaming, usually just client polling.
- cpncrunch 13y agoThe reality is that with broadband connections these days there is generally little or no packet loss unless your ISP is having problems. In the case where you are getting packet loss it usually tends to be around 50%, so UDP is going to have trouble as well. Also, these days with fast retransmit TCP isn't as bad as it used to be when there are network problems. I've been using TCP for over 20 years for multi-player games, VOIP and web conferencing, and it works incredibly well. Even if you do all the legwork to get UDP working reliably (which TCP give you for free), you still have the problem of firewalls.
- JoeAltmaier 13y agoPacket loss is usually due to upstream router congestion - you can send packets to your router much faster than it can forward them over a much-slower link to your ISP. SO loss depends upon your burst size/interval and your router buffer size. Wireless still has interesting(?) packet-loss behaviors. It can lose single packets, bursts of packets or even too-large packets (MTU exceeded). It depends on the AP and your local radio noise characteristics.
- jokoon 13y agoenet is a networking library which uses UDP but generate redundancy and other things so it has the advantages of udp with some advantages of TCP.
- mcv 13y agoI've heard before that TCP is a really awful fit for mobile. So shouldn't there be a general replacement for TCP on mobile? Everybody inventing their own custom fix does not sound like the most efficient way to solve this problem.
- deathanatos 13y ago> So shouldn't there be a general replacement for TCP on mobile? But what would this magical protocol do differently that would make it work better? In my opinion, most of TCP's semantics arise not out of the network, but rather the data itself. I can't have packets getting lost in the middle of an SSH session: it just doesn't make sense. My keystrokes are a stream of data that must be in order, and must be delivered: thus TCP. I commute daily, and use 4G on my commute. Another poster has problems very similar to mine[1]: > As an aside, these days it's worth noting that a 3G, WLAN or 4G link layer will try very hard not to drop any packets. So when you're doing UDP, watch out for those 40000 millisecond old packets that will burst to your lap once the rain cloud moves away from the cell tower You can see this just by pinging 8.8.8.8 in the background. The loss of good signal will result in many packets getting dropped, but then when signal resumes, ping will receive the packets that it had presumed lost, and you'll get things like: 64 bytes from 8.8.8.8: icmp_seq=65 ttl=46 time=37324.3 ms 64 bytes from 8.8.8.8: icmp_seq=66 ttl=46 time=36324.3 ms 64 bytes from 8.8.8.8: icmp_seq=67 ttl=46 time=35324.3 ms 64 bytes from 8.8.8.8: icmp_seq=68 ttl=46 time=34324.3 ms ... and so on ... From which you can tell that they were buffered somewhere, and then when the signal cleared up, basically transmitted quickly and cleanly. But it makes you wonder if TCP connections then receive a flood of duplicate packets (the original + the resends); I've not watched wireshark closely enough to see. For all the ads make you think speed is the determining factor in who has the better 4G, I'd say simple packet loss or connectivity loss makes much more of a difference to me day-to-day. "Coverage", you might call it, except typically according to the phone there is a signal, it is 4G, but the strength is just so poor as to be unusable. I use an app to get this info (the bars in the corner just aren't fine-grained enough), and it reports things like "Net. type: NSDPA * 7.2 Mbps", and that'd be okay, except: "Net strength: -99 dBm * 7 ASU" — too low for connectivity; in my experience, I require >-80 dBm for actual data to transmit successfully. (Ping packets to round trip, etc.) [1]: https://news.ycombinator.com/item?id=7507969
- RedneckBob 13y agofrom the_skys_kid: I'll defend UDP. UDP is the honey badger of the internet protocol suite. UDP is all about the transaction. UDP is standing on a cliff yelling, "Come at me, bro", whether you're there or not. UDP is a man's protocol doing real shit like bootstrapping your ass and slapping an IP on you. Get up, motha fucka! UDP will talk shit to one of you or all of you. UDP ain't scared. UDP brought the fear. UDP understands that you may be slow sometimes. So UDP will wait for your sorry ass. UDP grew up without a father, too. UDP sends a message and couldn't give a fuck if you got it or not. UDP got a message from you saying that you got his messages and guess what? UDP didn't even open it! Not one fuck given. Don't try to shake UDP's hand! You crazy? And, when UDP dies because you weren't available, UDP doesn't shed a tear. UDP is hardcore. He's going out even if he knows you ain't there. UDP is a goddam one-man slaughter house. Why? Because UDP doesn't give a fuck.
- wildbunny 13y agoTCP can get you quite far, even in a fast paced multiplayer game, as long as the game is suited. For example, the multiplayer version of asteroids in this article uses TCP sockets: http://www.wildbunny.co.uk/blog/2012/11/20/how-to-make-a-multi-player-game-part-2/ http://www.wildbunny.co.uk/blog/2012/11/20/how-to-make-a-mul... Cheers, Paul.
- RedneckBob 13y agogsfgf says: "Hi, I'd like to hear a TCP joke." "Hello, would you like to hear a TCP joke?" "Yes, I'd like to hear a TCP joke." "OK, I'll tell you a TCP joke." "Ok, I will hear a TCP joke." "Are you ready to hear a TCP joke?" "Yes, I am ready to hear a TCP joke." "Ok, I am about to send the TCP joke. It will last 10 seconds, it has two characters, it does not have a setting, it ends with a punchline." "Ok, I am ready to get your TCP joke that will last 10 seconds, has two characters, does not have an explicit setting, and ends with a punchline." "I'm sorry, your connection has timed out. Hello, would you like to hear a TCP joke?"
- shawabawa3 13y agoI'd reply with a UDP joke, but you might not get it
- ahassan 13y agoThe problem with using UDP is that many ISPs block UDP packets because it doesn't have features like congestion control. So using it for an internet game is out of the question, because there's a chance that a bunch of your users won't be able to play online.
- antihero 13y agoOne thing I've always wondered - has anyone written something that works over UDP but offers fairly basic TCP-like reliability functionality? So you'd have the speed advantage of UDP, however you could test the validity of your data, for instance.
- JoeAltmaier 13y agoWe have one, called STRAW. Our founder asked me to open-source it; but in the crush of work it never happened. Now he's gone; don't know if it would still be allowed by new mgmt. Advantage of STRAW over other reliable-udp protocols: multiple connections with a single port/NAT pinhole. Multiple channels per path.
- puppetmaster3 13y agoLately, you can just use things like http://www.pubnub.com/ http://www.pubnub.com/ now a days and do both and not have to worry. Socket servers are so 80's.
- scurvy 13y agoYou might as well paint "DDoS the hell out of me" on your forehead if you rely on UDP. Use TCP and block all UDP at your edge. Backhaul your own UDP traffic through GRE tunnels to offsite VPS's. If you have cool transit providers, you can have them place upstream filters to block UDP. That might be a pipedream though, I don't know of any large providers who will do that for you these days.
- scurvy 13y agoWhy the downvote? Is there anything technically wrong in my answer? Kids these days.
- VinzO 13y agoCould anyone point me to books and ressources for beginners on the topic of multiplayer game development?
- ggambetta 13y agoI've written a series of articles plus a live demo, with very little previous knowledge required, here: http://gabrielgambetta.com/fpm1.html http://gabrielgambetta.com/fpm1.html
- VinzO 13y agoThanks a lot. It looks very interesting. I will go through it when I have some time. Do you know any good book that covers this subject?
- ggambetta 13y agoNot really... besides these articles, the other two frequently cited sources are Valve (https://developer.valvesoftware.com/wiki/Source_Multiplayer_Networking https://developer.valvesoftware.com/wiki/Source_Multiplayer_...) and Gaffer (http://gafferongames.com/game-physics/networked-physics/ http://gafferongames.com/game-physics/networked-physics/). We all say essentially the same, at different levels of complexity. I have a live demo :) But you should read all three to get a complete picture.
- VinzO 13y agofunny, I just checked your profile and noticed you are also in Switzerland :-)
- ggambetta 13y agoSmall world :)