21 ms·
Is it just me or is networking really hard?
- mahouse 11y agoReading this one could think you would be a fool to use TCP, yet some games (such as World of Warcraft) use TCP exclusively with no problems... What gives?
- icebraining 11y agohttps://1024monkeys.wordpress.com/2014/04/01/game-servers-udp-vs-tcp/ https://1024monkeys.wordpress.com/2014/04/01/game-servers-ud...
- shultays 11y agoWorld of Warcraft can work with big latencies. He is talking about time critical games
- zapu 11y agoA lot of games use TCP and it's perfectly fine. But if you are racing the network, fighting over every millisecond of latency, you should just use UDP.
- Svenstaro 11y agoWell, World of Warcraft does feel like moving in jello as compared to well optimized UDP-based games like Counter-Strike for instance. However, it does not matter for World of Warcraft very much.
- Strom 11y agoSome games require less latency than others. WoW is a good example of a game that doesn't really have any precision combat. You don't have to aim at enemies there, you just have to lock them for your auto-target.
- exDM69 11y agoWorld of Warcraft is a pretty slow-paced game on "the inside", it just looks like a fast real time game but it's almost turn based when you analyze the gameplay. When you cast a spell, you'll instantly see the animation that your character is casting the spell, but the actual effect comes only when the animation is complete. This animation is used to hide the network latency (other players might see the animation played back slightly faster to compensate for the lag). Further, once you cast the spell, it must succeed and you can't really do anything before it has completed. In other words, reliability is essential and latency can be hidden. Contrast this with a fast paced, (soft) real time multiplayer game. When a player jumps, the jump has to start immediately. If you want to jump and shoot, the shots have to be fired right away, you can't wait for the jump to complete before shots are fired. With TCP, you'd be stuck on the ground until the "jump" packet is re-transmitted (two or more network round trips, tens to hundreds of milliseconds, very noticeable) and no shots would be fired until the character is off the ground. In a real time game, old packets are next to useless. Re-transmitting is wasting bandwidth and causing lag by blocking on information that is no longer useful. The networking model (like OP describes) is a constant stream of packets containing redundant information, minimum latency is essential and loss of packets is tolerated. You should be aware that TCP vs. UDP becomes apparent only when network conditions are bad. You could choose either and have satisfactory results 90% of the time, but once packets get lost, TCP does the wrong thing when it comes to fast paced real time games. If you do take the time to read Gaffer's original article, you should see that it is different from TCP in many ways. Yes it does similar things (reliability, flow control, etc) but in a completely different manner, tuned for a completely different use case.
- stefs 11y agoi forgot where i read this, but WoW is a bit of a special case because several issues were just circumvented through game design. example 1: there's no collision detection between players, i.e. you can block neither PCs nor NPCs. players blocking each other wouldn't have been possible with TCP at all. example 2: you don't aim and fire and hit/miss shots due to timing and precision (twitch) - you select a target and execute an action which is sent to the server which will roll a die to see the outcome. it was fairly usual for the early game to break down when a lot of players congregated in a small area (hogger or tarren mill, anyone?), where an instant spell took several seconds to execute and PCs were teleporting around (after running into a wall for several seconds) constantly because their positioning updates took too long. but because of the style of the gameplay this still worked well enough in most situations. what i want to say is that wow worked around TCPs performance issues by adjusting the game design. i can't fathom the reason for this - it might have to do with their focus on low hardware and connectivity requirements (for countries with unreliable connections). but even then i don't see why TCP would have been the better choice.
- nickpsecurity 11y agoCheck out this presentation on UDT protocol: http://udt.sourceforge.net/doc/udt-2009.ppt http://udt.sourceforge.net/doc/udt-2009.ppt It details the problems of TCP, names most alternatives, and describes UDT work. I used to use UDT to eliminate TCP's problems. It's just bad design and the fact that changing it is usually a kernel modification. Hence, all these application-level alternatives with UDT being most general-purpose and one of best. Be interesting to see if someone can adapt it to games. For now, probably best to use game-specific model as in article and comments. UDT for other stuff where you control how the app works on both ends. TCP where you control one side.
- ggambetta 11y agoI've written a series of articles about this topic: http://gabrielgambetta.com/fast_paced_multiplayer.html http://gabrielgambetta.com/fast_paced_multiplayer.html It's usually well received, perhaps because it builds the ideas from the ground up and does not include the words "What the fuck is wrong with you?".
- Luc 11y agoThanks very much for this, these are great articles. It seems one would need some way of synchronizing clocks (or measuring the lag between server and client). Do you know if there is a standard way of accomplishing this?
- exDM69 11y ago> Do you know if there is a standard way of accomplishing this? Synchronizing clocks is one of the difficult problems in distributed computing. Games tend to avoid doing proper synchronization of clocks by working in discrete time steps (frames). Because games run at 15-60 frames per second (a bit more for simulators), you can get away with synchronizing the clocks to +/- a few frames, ie. an accuracy of tens of milliseconds is good enough to provide a perception of real time. In other words: game time is measured with an unsigned integer that tells how many time steps have elapsed since the game started.
- techdragon 11y agoSorry but that needs a rather large asterisk applied to it. In the current era of 120Hz LCD displays and a Billion dollar market for video games. It's becoming very evident that the "best", not in a AAA title sense, but in the 'so popular it becomes a sport with professional players" sense, games are those that have put a LOT more effort into the implications of their network synchronisation algorithms. Some game engines have extremely long tail lives as popular mods take advantage of reliable network behaviour to create gaming experiences that don't infuriate passionate players keeping them alive many years after the game that wrote the engine moved on.
- staunch 11y ago> We have the Tribes Networking Model, The Unreal Networking Model, Valve Latency Compensation article. Which are all derivatives of the Quake model.
- Mithaldu 11y agoCan you point out where the Quake model is described in sufficient detail that one can reimplement it for one's own game? Or is it open source?
- staunch 11y agoAll the Quake engines are open source and there are good articles. It's pretty fascinating stuff to learn about. http://fabiensanglard.net/quake3/network.php http://fabiensanglard.net/quake3/network.php
- poizan42 11y agoWell Quake is open source, so that might be a good place to start: https://github.com/id-Software/Quake https://github.com/id-Software/Quake I don't know if it's described sufficiently detailed anywhere besides the source though.
- tinco 11y agoPerhaps if you're going for simplicity or something then that might be an ok idea, but he mentions those three because they are more modern or important additions to the Quake model. He is of course very aware of their Quake heritage, at the end of this article he credits John Carmack (in an indirect way). I recommend reading the rest of Gaffer's articles before doing anything, it will give you a basis that's needed to understand why Quake does what it does.
- floodyberry 11y agoHow is Tribes networking derivative of Quake, other than using UDP?
- floodyberry- 11y agoHow is Tribes networking derivative of Quakes, other than using UDP?
- deleted 11y ago[deleted]
- voidz 11y agoThat must have been awful.
- vog 11y agoWow, so many downvotes. Why is it considered non-constructive to point out a misleading title? The title is the first impression of an article, and most likely the authors are not aware of that ambiguity.
- kaoD 11y agoI didn't downvote, but calling that a misleading title is... misleading. Bias by profession is probably at play here, and it works the other way around: if I found a title titled "networking is hard" dealing with professional/social networking, I'd be confused at first too. I wouldn't call it misleading though, which is probably what triggered the downvotes.
- nickpsecurity 11y agoFunny thing is, I wondered whether it was about social or technical networking when I looked at it. I don't read articles from the source so it wasn't immediately obvious. Didn't bother me, though, as English is just like that: words get overloaded with different meanings. Know this could affect searches, I might have said "or is networking in games just that hard?" Either way, first sentence of the quote tells reader what it's about so no time wasted really.
- Mithaldu 11y agoSimply put: The mistake of presuming was on OP's side, but OP instead decided to blame the author.
- babuskov 11y agoI'm thinking to add online multiplayer to my game: http://store.steampowered.com/app/363670 http://store.steampowered.com/app/363670 ...so this was very interesting read. I investigated a little bit, and found that there are more protocols than just pure UDP, and now I can't decide which one to try. I narrowed it down to these three: * PCC http://modong.github.io/pcc-page/ http://modong.github.io/pcc-page/ * NORM http://www.nrl.navy.mil/itd/ncs/products/norm http://www.nrl.navy.mil/itd/ncs/products/norm * ENet over UDP http://enet.bespin.org/ http://enet.bespin.org/ What do you think?
- Mithaldu 11y agoSince your players don't seem to be affecting each other a lot, you might even be able to go with simple TCP. UDP is needed when latency matters a LOT, which is mainly when people fight one another.
- TazeTSchnitzel 11y agoI believe World of Warcraft and some other MMORPGs use TCP, since latency isn't too big of a deal. Note that they do some things to compensate for this, however, like making players not collide with each other.
- deleted 11y ago[deleted]
- corysama 11y agoI've worked with enet and I'm a big fan. It is definitely well suited for games. I'm not familiar with the other two. A non-obvious feature of enet is that you can create two channels on a single connection then send permanent info (item 3 has been collected by player 2) reliably on one channel and send short term data (player 2's position is xyz) unreliably on the other channel. The point is that the packets with a channel are ordered but different channels have separate timelines and retry queues. That means even if a reliable packet needs a lot of retries and takes long time getting through, it won't stall the packets on a different channel.
- Arnt 11y agoNetworking has local time, like the relativity theory. "Those two packets are at the same time at point a, but not point b." "A sends b and C sends d, but since time is local you cannot" etc. So IMO it makes sense that networking is really hard. I like it though. Networking problems are clear when you think hard about them, unlike so many of today's programming problems, where the key is to know some half-documented aspect of a large framework/library/whatnot.
- hmans 11y agoSomeone needs a bit of anger management therapy.
- OliverJones 11y agoI sometimes ask the question "how does the internet work?" in interviews. The vague and wrong answers I hear are astonishing. I'm looking for a basic understanding of the layering of TCP, or RDP, or ICMP (or anything actually) on top of IP. I don't get that level of insight much. The people who do have that insight usually can go deeper -- slow-start, BGP, etc. I wonder if we have a modern version of C.P. Snow's mid-twentieth-century two cultures lament. https://en.wikipedia.org/wiki/The_Two_Cultures https://en.wikipedia.org/wiki/The_Two_Cultures. That's the complaint that educated humanities folks don't know the basics of science. Are we miseducating our so-called "full-stack" engineers by not offering them a basic understanding of telecom? Or are they not listening?
- moron4hire 11y agoOr is it just not very important? Obviously, from your line of questioning, you think it is. But for the majority of software developers who only work on CRUD web apps, is it at all necessary to know this sort of stuff?
- userbinator 11y agoInterviews for what type of work? I'm assuming it's networking-related?
- mariojv 11y agoDo you have a good recommendation for introductory to intermediate level guides for understanding the aspects of networking you find important? I learn a little bit about intra-DC networking on the job, but I've never needed to use anything else yet.
- Panoramix 11y agoYou have to stop at some point. Or do you go in depth into the workings of >10Gbps transceivers? the quantum mechanics of optical fibers? how to they manufacture top-notch photodiodes anyway? You can literally spend a lifetime in these depths.
- BatFastard 11y ago
- tormeh 11y ago>You have whole industries (gaming, video streaming, VoIP) that will laugh at you if you send time critical data over TCP and rightly so Skype uses TCP, no? Laugh, indeed.
- douche 11y agoSkype does many things, not all of them the greatest. For instance, it breaks all my HttpListener-based .NET code when it is running.
- tokenizerrr 11y agoFor some reason skype listens on port 80 and 443. You can turn this off in the settings.
- Too 11y agoThis might finally explain why the video quality is so bad then. If you tried any commercial video call software it's quite hard to look back to skype.
- tormeh 11y agoGoogle Hangouts isn't much better. I would assume it's more about the constraints of free services than anything else. If you're talking about systems like Cisco Telepresence, then those are obviously not comparable because of (lots of) dedicated hardware, better connections etc.
- hebdo 11y agoWell, technically speaking any multiplayer game is a distributed database, except that more inconsistency is tolerated ("lag"). But all of the classic results, such as CAP or FLP, apply here as well. And given how hard even the simple distributed protocols are (eg. Paxos) it should not be a surprise that writing a good networking stack for a multiplayer game is also hard.
- toolslive 11y agoI think the topics should be split: you have 'networking' which is about connecting computers typically using TCP, UDP, RDMA, ... and 'distributed systems' which is about how to use connected computers to achieve something. And yes, both are difficult, but for other reasons: - networking is hard because the APIs are old, platform specific and probably don't behave very intuitively. So trying to build messaging systems on top of sockets is not simple, (but you will only find out after you've moved beyond the prototype stage). - distributed systems is hard because the odds of successfully exchanging messages aren't in your favour.
- zobzu 11y agonetcode is the most difficult part of fast multiplier games. then again one could just read quake3 source - its actually, 16 years later, still one of the best for that.
- drxzcl 11y agoI know this attitude all too well. I've replied to a few questions on gamedev.stackexchange with pointers to the usual articles on game state synchronization and lag compensation, and I get lukewarm reactions of "too complicated" type. Meanwhile mr rpc-over-tcp get tens of up votes and accepted answer. Their loss I suppose.
- larrys 11y agoOften, "to complicated" simply means "no time and no patience to RTFM".
- zamalek 11y ago> Their loss I suppose. Our loss, really: less good quality games for everyone. I threw in the towel at GameDev for reasons very similar to yours ("premature optimization" also comes up far too frequently) - from, what can only be, people who have never even worked on specific components of games that they are claiming knowledge on. The StackOverflow system somewhat fails on GameDev, because there aren't enough knowledgeable people on the site to issue corrective votes (likely because these people are scarce in general). Incidentally the bad advice is bad on many levels: even if you somehow ignore UDP it's still bad advice to do "RPC over TCP". TCP works best (fastest, least latency) when the send buffers are full at both ends - which RPC, by definition, does not do. This means that these people simply couldn't have researched the fundamentals of the protocol that they assume to be "good enough," how could they therefore be aware of how it is better or worse than alternatives?
- zurn 11y agoThe single problem with TCP this rightly points out is head-of-line blocking. But overcoming that still leaves you with the problem of coping with out of order or missing events at app level, with clients and servers seeing events in differing sequences (fex packet sequence of move-duck-fire, now shuffle the move around...). I witnessed one game engine project implement custom networking on UDP and then disable unreliable & out of order messages because of game logic headaches. And the payoff is quite small, since packet loss is rare in healthy networks and TCP handles the rare loss pretty efficiently based on ack clocking, not timeouts (fast retransmit, basically same as his idea of "redundantly sending un-acked data").
- jschwartzi 11y agoHow many networks are actually healthy, though? Networked gaming is less about a hardwired LAN and more about reliably synchronizing state using your crap wifi in a oversaturated band when Comcast hasn't repaired the infrastructure I your neighborhood in years. In other words, a healthy network is the edge case.
- zurn 11y agoSure there are people with rotten last mile conditions, but healthy networks are the normal case. You get subpar download speeds with http and choppy skype/facetime/ssh if your network drops packets frequently. It's bad for business from the ISP POV. Using UDP will not work around the conditions you describe, game will still be laggy and choppy.
- Dylan16807 11y agoDownloads will work just fine if you're dropping .1% of packets. Skype is very resilient to dropped packets. Console use isn't quite so fast that a half-second hiccup is a big problem. But a naive game implementation over TCP will stutter badly and obviously every time.
- zurn 11y ago
- ninjakeyboard 11y agoYou know what I found that is actually quite amazing: Video encoders for streams of linear/live video feed for broadcast go over UDP to the next device in the chain. An encoder will often output a stream of video over UDP to devices that will then split the video into multiple renditions.
- jokoon 11y agoTCP is suitable for "flat" static content, which you only retrieve at punctual intervals, and is relatively large. A good example is a file which won't change for the next 5 min, and you know that you want that exact file BUT you are ready to wait until it loads perfectly. UDP is for real time, continuous data, when you can't wait for your hardware to check for packet consistency. A good example is paper mail versus a telephone call. The paper mail will land, but you don't care if your phone line cuts for 1 second. You just wait for the other guy to check if he can hear you UDP is really much better when you want to have performance, especially when it comes to responsiveness and latency, but can afford to discard packets if they're not reliable. I think it boils down to people not realizing how hard it is to transmit data over long distances, and that the protocol don't consider that data transmission is 100% good, because it never is. You can have a lot of interference in your network, but TCP will always manage to land those packets.
- known 11y agohttp://www.kegel.com/c10k.html http://www.kegel.com/c10k.html
- vans 11y agoWhat's the point of this insulting article full of rage ? nothing to do at this ranking on HN.
- Const-me 11y agoNot every game is critical to network latency. Not every game is tolerant to packets loss. If those 4-8 players in the game are playing 3-dimension virtual reality poker, TCP or even HTTP will work just fine. I have read the original thread, and the author says somewhere in the comment, “it is likely to be similar to that you would see in something like Diablo II. Players exist in a world, attack enemies and interact with NPCs (including trading).” Which makes the article irrelevant to the original question. Games like Diablo 2 are not real-time; they can usually use TCP just fine.
- mmaunder 11y agoI guess that's one way to package up a fairly fundamental piece of networking knowledge. Wrap the fucker in expletives to get attention.
- ilurk 11y agoI know the article is language agnostic but I don't think anyone would develop their networking modules (in the context of videogames or finance) in anything but C++ (or Erlang/Elixir?). So could the local experts give their opinion on C++ networking libraries? I know there are a few [1] but how do they compare? I've only used Qt networking and POSIX sockets (although for C), but the application didn't have much of latency or throughput requirements. [1] http://stackoverflow.com/questions/118945/best-c-c-network-library http://stackoverflow.com/questions/118945/best-c-c-network-l...
- t413 11y agoBoost ASIO for me. The ability to single threaded high-speed servers and then to extend to multithreaded with a things like thread pools and atomic stores is invaluable.
- Const-me 11y agoYou shouldn’t just compare C++ libraries. First, define your requirements, both functional (what should your networking do) and technical (latency, throughput, available bandwidth, operating systems of both clients and servers). Second, pick a protocols stack. Only then, look for those C++ libraries. If on step #2 you’ll choose e.g. ProtoBuff + UDP, you’ll have totally different set of available libraries than if you choose SOAP + HTTPS. Besides, very few of those libraries work well on multiple platforms, e.g. libuv does, while libevent does not. In addition, C++ is not the only choice. I once developed networking module for a game in C#. Have a few friends who are doing the same in finance industry.
- chetanahuja 11y ago"You can’t even imagine a way that reliability could be implemented on top of UDP that beats TCP? What total bullshit" This. This "TCP already does it best in every situation" is a common trope usually spouted by people who know nothing about what TCP does and why. The easiest way to get over this misconception is to ask yourself, can you think of a protocol that not only works well over all sorts of networks, all the way from two boxes connected by a 10Gig switch (with 10us ping latency) vs two hosts talking to each other via a satellite link (with perhaps, 2 second ping latency) but is the best option across all those situations? The truth is that TCP is the common denominator that's available in pretty much every device you would want to use. But in almost every individual use case, you could come up with a far better protocol if you took the time and effort to treat the problem seriously.
- nubb 11y agoUDP is used when it's designed by network engineers. That's why we have VOIP and video streaming - all are served UDP. dont get me started on the lack of multicast out there. imagine game servers running multicast. one day.
- chetanahuja 11y agoWell... those are examples of designing custom protocols on top of UDP. These network engineers, who I assume are specially trained in some secret govt. facility for network commando training, actually have to understand the problem at hand and design something specifically for the problem. Calling these protocols just "UDP" is like saying something like "I use wheels to commute from home to office everyday"
- efaref 11y agoI would hardly call protocols like [SIP](https://www.ietf.org/rfc/rfc3261.txt https://www.ietf.org/rfc/rfc3261.txt) and [RTP](https://www.ietf.org/rfc/rfc3550.txt https://www.ietf.org/rfc/rfc3550.txt) "custom". There are many more standard protocols out there than just TCP and UDP. There's even a "better TCP" in the form of [SCTP](https://tools.ietf.org/rfc/rfc4960.txt https://tools.ietf.org/rfc/rfc4960.txt). The fact is that TCP was the MVP for reliable streams of data on the 1970s internet, and is woefully inadequate for modern usage, but we're stuck with it (including its oft-disabled parts like Nagle's Algorithm) because it's what's available everywhere. Game network protocols are pretty easy in comparison with some of the protocols I've encountered in my day job, especially once you've learned the lessons of people like the author of that article.
- nubb 11y agoAs a network engineer, this was great. No body never bothers to learn networking. The article is annoying tho, basically says use UDP idiots. I mean, why do we think VOIP uses UDP? Same concept. If only more devs would implement multicast...
- mahyarm 11y agoIsn't the problem with multicast is most networks filtering it out or not supporting it? Thus it only stays useful for the LAN network scope? So I cannot just multicast to 5000 random IPs on the internet.
- yusyusyus 11y agoNot supporting it because why though? IP multicast routing requires end-to-end state for each (source , group). Further, the boxes on the market for doing multicast (not a little GPCPU router) have limitations concerning how they do multicast within the box that makes scaling multicast a real pain (basically, internally, they will send forwarding engines traffic they don't need and choke up there because it runs out of internal ways to express the separate streams). Some of this can be shortcut by tunneling techniques, but it is just masking the situation and adding another layer of network complexity (in an area where humans still try to manage/troubleshoot mostly by hand).
- Animats 11y agoIt's not that networking is really hard, it's that development of synchronized games that run over unreliable links is really hard. If you want a simple solution for a simple game, here's one. Use a fixed-format UDP packet to pass current location and orientation of the player and whatever else changes rapidly. This should be stateless; if you lose a packet, the next one has a full state update. This is enough for a simple FPS game. Anything else goes over one or more TCP connections. If you have to load assets (level maps, textures, etc.) just use HTTP over TCP, which means you get to use standard servers and client software for that stuff. If your game is too complex for that model, you're probably going to have to do some serious thinking about distributed synchronization.
- jstimpfle 11y ago> This should be stateless; if you lose a packet, the next one has a full state update. This is enough for a simple FPS game. Also enough considering packet reordering? And corruption?
- Animats 11y agoIf you get a bad packet, the problem will disappear when the next one comes in.
- callesgg 11y agothe next packet should come within a few milliseconds so yes. but I see your point adding a simple counter might be somewhat useful ignore any packet that has a Lower counter value. coruption there is generaly enogh checksums in earlier layers so that it is neglisheble.
- Schwolop 11y agoThe best part of this article is the two comments at the end. The first is by the person who posted the Reddit question that prompted the entire rant, describing how asking them question made them rethink everything, and how they ultimately settled on a UDP design just like the one promoted in this article. The second is by the article's author: "Bravo and well done. I’m quite impressed".