4 ms·

Oh hi! I'm a network engineer! In gaming! I happen to routinely use 'the network architecture of a telecom system' via erlang so I may be qualified to answer

Oh hi! I'm a network engineer! In gaming! I happen to routinely use 'the network architecture of a telecom system' via erlang so I may be qualified to answer your post. RFC 6951 style SCTP-over-UDP is not super widely implemented, so you end up having to implement it yourself. When you're doing so, you pretty quickly realize that it's significantly more heavyweight than anything you actually need for a game (e.g. https://tools.ietf.org/html/rfc4960#section-5.1.6 https://tools.ietf.org/html/rfc4960#section-5.1.6), while at the same time missing several key facilities, such as knowing instantly, with every packet received, which of the previous (e.g.) 32 packets have also been received, and so forth. RTP has the same problem. There's fields included with every packet (SSRC, CSRC, others) which are irrelevant to gaming. There's a bunch of libraries, most of which are total junk. There are end to end compatibility problems owing to different interpretations of the spec. This turns out to be a gigantic problem, because gaming network engineers use all sorts of ridiculous tricks to try to bum every last byte out of their protocols. For an entertaining afternoon check out http://gafferongames.com/2015/03/14/the-networked-physics-data-compression-challenge/ http://gafferongames.com/2015/03/14/the-networked-physics-da.... A much more relevant criticism might be, well why don't you use, e.g. protobuf rather than invent your own framing system? Because the case of protobuf -- where you want, essentially, to transmit a purpose designed struct directly over the wire with no bullshit -- is very much like the case of a twitch game network protocol and comes pretty close to hand-bummed byte sizes; close enough that the benefits of using something a bit more extensible and introspectable might be worth it. Anyway, in short, you don't know what you're talking about and Glenn is a national treasure.

derefr 10y ago
Thanks for that, it was informative! I mean, I certainly don't know what I'm talking about with respect to the specific protocols; SCTP and RTP were ones off the top of my head that seemed to fit the use-case. (I have used both—from Erlang, coincidentally!—as part of real-time collaboration software before. But the "real-time" there isn't nearly the same "real-time" as twitch games have, so I never really needed to dig deep into their semantics.) I think I got off-track of the original thing that incensed me, though: it's not that there's any perfect existing network protocol that does everything the way games need it. Rather, my thought was based on this premise: the OSI protocol-layering model is awesome. It allows you to have a stack of very thin protocol layers (e.g. UDP itself) that just add one single thing to the semantics of the layer below them, with as little overhead as it's possible to get away with in doing that. Thin layers, because they just Do One Thing Well, are easy to engineer and QA and port, so you often find (good!) libraries for them available for every platform. Creating your own custom protocol that re-implements the feature a thin protocol layer is intended to provide, often doesn't reduce overhead at all, and in the process often results in worse and less-tested code than you'd get by writing the rest of your custom code on top of the thin layer. So: maybe you can't find a perfect "gaming protocol." But I can't believe that the most sensible thing to do is to implement one directly on top of UDP, rather than using A and B and C orthogonal protocol layers on top of UDP to handle A and B and C features, and then only coding the stuff that's left over. In other words: why write your own multiplexing code instead of relying on existing session-layer logic? And why, indeed, write your own network-encoding code instead of relying on existing presentation-layer logic (like, of course, Protobuf)? And on and on for eight to ten other concerns it's possible to "pull out" of any given protocol.
- felixgallo 10y ago
  Well, I'd caution you that just because you don't understand why something is sensible, it doesn't necessarily mean that it's foolish. In the case of protocols, there really don't exist "A and B and C orthogonal protocol layers on top of UDP to handle A and B and C features." What you have instead is several different, custom ideas that all need to work together at a low level. For example, you need to be able to identify a particular packet so that you can keep track of whether you have it or not; so you have sequence numbers. But you need to keep the packet small, so you don't use a 128 bit float, you use a 16 bit int and keep track of under/overflow yourself. You need to understand if any very recent packets that you've sent need to be reencoded and sent again, so you have an ack field that tells the recipient the high water mark number of the packet you most recently saw from them. You need to know some, but not forever, of the history, so you keep a, say, 32 bit bitfield which tells the recipient the state of the last 32 packets you've sent. You need multiplexed messages to pack as much info in a single transmission as possible, so you have length-encoded payload attachments probably. And all of these lessons were learned in blood. Like, hundreds of games have shipped and died to teach us these things. Many of them taught the bitter lessons of overabstraction and excessive separation of concerns. People shipped competitive fighting games with TCP. People shipped client-authoritative games where real money was involved. I mean, the battleground is littered with people who fucked up. Glenn is one of those guys who has seen everything -- including some of the AAAest of the AAA games that you've ever even heard of -- and he's sharing some wisdom with you. Instead of first trying to question what he is doing, and attacking it for being foolish, why don't you take a chair, sit down, and try to figure out how he got where he did? You will learn a great deal and you will become a radically better engineer.
  - renox 10y ago
    a 128 bit float --> a 128 bit integer Probably a typo: using float to send sequence number would be a very bad idea..
vvanders 10y ago
Agreed with everything above, one of the things I love about games is the really unique problems they tend to present. FWIW if you're looking at protobuf then take a peek at flatbuffers[1]. We found that protobuf's allocation was hurting perf and flatbuffers fit the same role with much better performance. [1] https://google.github.io/flatbuffers/ https://google.github.io/flatbuffers/
the_angry_angel 10y ago
> why don't you use, e.g. protobuf protobuf sounds attractive, but (unless I've missed something in protobuf's implementation), but out of the box you have 2 issues: 1. Protobuf doesn't/can't deal with fragmentation - so you'd have to ensure your protobufs are small enough to fit inside a UDP packet 2. Without a wrapper you couldn't put multiple small protobufs into a single UDP packet? Both things are fixable by wrapping protobuf, but I suppose if you're trying to get something as small as possible, can you actually just do better by avoiding protobuf in the first place (I guess the answer is yes?)
- niftich 10y ago
  I understood it as using protobuf's interface definitions as as an integration point, not necessarily protobuf's exact wire format and everything below it that comes out-of-the-box (like TCP and below) in the default client.
  - felixgallo 10y ago
    I meant everything up to and including protobuf's wire format, but nothing beyond that, which as you note wouldn't be super great.
- mikemarcin 10y ago
  Protobuf and most similar libraries don't support delta compression which is essential for game state updates.
  - felixgallo 10y ago
    protobuf as a structure serialization/deserialization library shouldn't have the responsibility for delta calculations in the first place. Once the application decides what the delta is, it can use protobuf or anything else to encode and transmit that.
    - mikemarcin 10y ago
      You want to build a full update, then for each client diff between the last ack'd full update and send only that delta. The application specific logic builds the full update. The rest can be done automatically by a good library. Building a delta compressed protobuf message for a transform (using protobuf.net) looks something like: http://codepad.org/3vjxogxZ http://codepad.org/3vjxogxZ
jwatte 10y ago
To be fair, protobuf adds field-and-type framing for each field, which a known-in-sync client/server don't need; that adds up to 10-30% over a full packet!

derefr 10y ago

Thanks for that, it was informative! I mean, I certainly don't know what I'm talking about with respect to the specific protocols; SCTP and RTP were ones off the top of my head that seemed to fit the use-case. (I have used both—from Erlang, coincidentally!—as part of real-time collaboration software before. But the "real-time" there isn't nearly the same "real-time" as twitch games have, so I never really needed to dig deep into their semantics.) I think I got off-track of the original thing that incensed me, though: it's not that there's any perfect existing network protocol that does everything the way games need it. Rather, my thought was based on this premise: the OSI protocol-layering model is awesome. It allows you to have a stack of very thin protocol layers (e.g. UDP itself) that just add one single thing to the semantics of the layer below them, with as little overhead as it's possible to get away with in doing that. Thin layers, because they just Do One Thing Well, are easy to engineer and QA and port, so you often find (good!) libraries for them available for every platform. Creating your own custom protocol that re-implements the feature a thin protocol layer is intended to provide, often doesn't reduce overhead at all, and in the process often results in worse and less-tested code than you'd get by writing the rest of your custom code on top of the thin layer. So: maybe you can't find a perfect "gaming protocol." But I can't believe that the most sensible thing to do is to implement one directly on top of UDP, rather than using A and B and C orthogonal protocol layers on top of UDP to handle A and B and C features, and then only coding the stuff that's left over. In other words: why write your own multiplexing code instead of relying on existing session-layer logic? And why, indeed, write your own network-encoding code instead of relying on existing presentation-layer logic (like, of course, Protobuf)? And on and on for eight to ten other concerns it's possible to "pull out" of any given protocol.

felixgallo 10y ago
Well, I'd caution you that just because you don't understand why something is sensible, it doesn't necessarily mean that it's foolish. In the case of protocols, there really don't exist "A and B and C orthogonal protocol layers on top of UDP to handle A and B and C features." What you have instead is several different, custom ideas that all need to work together at a low level. For example, you need to be able to identify a particular packet so that you can keep track of whether you have it or not; so you have sequence numbers. But you need to keep the packet small, so you don't use a 128 bit float, you use a 16 bit int and keep track of under/overflow yourself. You need to understand if any very recent packets that you've sent need to be reencoded and sent again, so you have an ack field that tells the recipient the high water mark number of the packet you most recently saw from them. You need to know some, but not forever, of the history, so you keep a, say, 32 bit bitfield which tells the recipient the state of the last 32 packets you've sent. You need multiplexed messages to pack as much info in a single transmission as possible, so you have length-encoded payload attachments probably. And all of these lessons were learned in blood. Like, hundreds of games have shipped and died to teach us these things. Many of them taught the bitter lessons of overabstraction and excessive separation of concerns. People shipped competitive fighting games with TCP. People shipped client-authoritative games where real money was involved. I mean, the battleground is littered with people who fucked up. Glenn is one of those guys who has seen everything -- including some of the AAAest of the AAA games that you've ever even heard of -- and he's sharing some wisdom with you. Instead of first trying to question what he is doing, and attacking it for being foolish, why don't you take a chair, sit down, and try to figure out how he got where he did? You will learn a great deal and you will become a radically better engineer.
- renox 10y ago
  a 128 bit float --> a 128 bit integer Probably a typo: using float to send sequence number would be a very bad idea..

vvanders 10y ago

Agreed with everything above, one of the things I love about games is the really unique problems they tend to present. FWIW if you're looking at protobuf then take a peek at flatbuffers[1]. We found that protobuf's allocation was hurting perf and flatbuffers fit the same role with much better performance. [1] https://google.github.io/flatbuffers/ https://google.github.io/flatbuffers/

the_angry_angel 10y ago

> why don't you use, e.g. protobuf protobuf sounds attractive, but (unless I've missed something in protobuf's implementation), but out of the box you have 2 issues: 1. Protobuf doesn't/can't deal with fragmentation - so you'd have to ensure your protobufs are small enough to fit inside a UDP packet 2. Without a wrapper you couldn't put multiple small protobufs into a single UDP packet? Both things are fixable by wrapping protobuf, but I suppose if you're trying to get something as small as possible, can you actually just do better by avoiding protobuf in the first place (I guess the answer is yes?)

niftich 10y ago
I understood it as using protobuf's interface definitions as as an integration point, not necessarily protobuf's exact wire format and everything below it that comes out-of-the-box (like TCP and below) in the default client.
- felixgallo 10y ago
  I meant everything up to and including protobuf's wire format, but nothing beyond that, which as you note wouldn't be super great.
mikemarcin 10y ago
Protobuf and most similar libraries don't support delta compression which is essential for game state updates.
- felixgallo 10y ago
  protobuf as a structure serialization/deserialization library shouldn't have the responsibility for delta calculations in the first place. Once the application decides what the delta is, it can use protobuf or anything else to encode and transmit that.
  - mikemarcin 10y ago
    You want to build a full update, then for each client diff between the last ack'd full update and send only that delta. The application specific logic builds the full update. The rest can be done automatically by a good library. Building a delta compressed protobuf message for a transform (using protobuf.net) looks something like: http://codepad.org/3vjxogxZ http://codepad.org/3vjxogxZ

jwatte 10y ago

To be fair, protobuf adds field-and-type framing for each field, which a known-in-sync client/server don't need; that adds up to 10-30% over a full packet!