6 ms·

One elegant solution to this problem is to keep 100% of the game state on the server and to stream x264 to clients. I think if you engineered games explicitly

One elegant solution to this problem is to keep 100% of the game state on the server and to stream x264 to clients. I think if you engineered games explicitly for this use case, you could create a much more economical path than what products like Stadia offer today.

orev 4y ago
This seems so obvious to me that I’m surprised Stadia doesn’t already do that (not having done any real reading on how it works; just making assumptions based on the marketing I’ve seen). I just assumed it was a video stream with a touch control overlay.
milgrim 4y ago
How is that elegant? In return for easy synchronization the server has to perform ALL computations. Sounds more like brute force to me.
- addingnumbers 4y ago
  It's very elegant depending on your goals. It's the ultimate DRM for games, you can't crack a game's copy protection if you can't see it's binaries. You can't data-mine the locations of valuables if you don't have the map data. You can't leak unreleased assets ahead of their marketing debut if you don't have a copy of the assets. You can't expose all the Easter eggs by decompiling if you don't have the code. With subscription and premium currency models those abilities can all be interpreted as lost revenue. The markets for people buying $400 consoles vs buying a $20 HDMI stick and a $15/mo subscription are very different. After the colossal (and to me, surprising) rise of mobile gaming I think the latter might be where the real money will be 10 years from now. They'll address the bottlenecks on the data center end. I'm pretty sure you can list a dozen problems that make it prohibitively expensive right now and for every one of them some Microsoft or NVIDIA engineer can tell you how they are working on engineering away that problem in a couple years.
  - milgrim 4y ago
    Of course, it solves a lot of problems. But you have to pay for it by increasing the load on the servers by orders of magnitude. This makes it a brute force approach in my opinion. This is of course depending on your meaning of „elegant“, but for me this would imply to solve these issues without increasing the server load so much. Let the client decide as much as possible, but check the decisions randomly and in case of suspicious player stats. And DRM for multiplayer games should be no problem anyways? Verify the accounts or your users, but that applies to stadia like services and also the „conventional“ ones. Solving the data mining issue is another topic and yes, giving the server more authority for things that the player should be able to see might be the only way to deal with this. But maybe the server could hand out client specific decryption keys when they are needed? That would be elegant, and not just keeping all the content server-side. Game streaming services will find their place, but they address mainly the entry hurdle and not the issue of game state synchronization.
- atq2119 4y ago
  Depending on how high the encode/decode/transmit overhead is, it might be a more efficient use of resources. Most game consoles and gaming PCs are idle most of the time. Centralizing the rendering in a data center is going to yield better resource utilization rates. Mind you, they're still not going to be great rates, because the data center needs to be geographically close and so utilization will vary a lot in daily and weekly patterns. Then again, maybe you can fill all those GPUs with ML training work when the gaming utilization is low...
  - milgrim 4y ago
    Latency has a big impact on user experience in most multiplayer games and players with a low latency have an advantage in many games. I am not sure how satisfied people will be with low cost low powered systems made for game streaming in the long run when other users have a much better experience and even advantages in gameplay.
viktorcode 4y ago
You don't have to stream encoded video from the server. It is much faster and simpler to stream game state and have the game clients render it. Esentailly, that's how all the games with "authoritative server" work.
lewispollard 4y ago
Then you have a server rendering essentially 4 or 8 or 32 or whatever individual games, capturing and encoding them to a streamable format, streaming via a socket or webrtc or whatever to the client, who then sees the action, they input a control, the control has to get back to the server to be processed, render the next frame for every client, send them all back, and then the client sees the result of their action. Doesn't seem elegant to me, it seems like a way to have wildly uncontrollable latency, and have one player's poor connection disrupt everyone else's experience. I have a Steam Link hardware device to stream games from my PC to my TV over ethernet LAN, and even that can have issues with latency and encoding that make it a worse experience than just playing on the PC.
kaetemi 4y ago
Stream OpenGL rendering calls from the server to the client.
- bob1029 4y ago
  Not all clients have the same capabilities with regard to 3d hardware. Virtually any modern device can decode x264 without difficulty.
- Const-me 4y ago
  That works fine for simple 3D scenes. One application, that’s how MS remote desktop worked until recently — Windows streamed Direct3D rendering calls to the client. However, I’m not sure this gonna work with modern triple-A games. The current-gen low level GPU APIs were designed to allow developers of game engines to saturate bandwidth of PCI-express with these rendering calls. That’s too many gigabytes/second for networking, I’m afraid.
TinkersW 4y ago
That defeats most of what this article is talking about--hiding the perceived latency. In addition to the other issues like higher server cpu & bandwidth requirements, and poor video quality.
dylan604 4y ago
But it's not just one video render. It is one video render per user. People have to build pretty decent rigs to get their single view of the game to render at speed. Multiplying that times the number of players seems wildly expensive for a single machine to do to be a non-starter after 3 seconds of thought.
Const-me 4y ago
The article is about hiding latency between other people’s input, and your display. While it happens, your own input only has couple frames of latency. The solution you proposed introduces latency between your own input, and your own display. Unless the server is in your house or at least within 100km from it, that latency gonna make the game unplayable.
- toast0 4y ago
  Unless the server is in my house, I get at least 20ms round trip (thanks bonded VDSL2), so I'm a frame behind already. Add input sampling, codec delays, and video output delays and I'm at least two frames, probably three. That's going to be not great.
dbttdft 4y ago
there's a very small chance that cloud gaming may one day work. even if that small window of opportunity exists, the incompetent game industry will miss it.
GuB-42 4y ago
It doesn't solve the problem, in fact, it limits your options. And I may be wrong but x264 doesn't seem to have the lowest latency codecs. There are essentially two ways of dealing with latency : input lag (wait until we know everything before showing player actions) and prediction/rollback (respond immediately trying to guess what we don't know yet and fix it later, what is shown in the article). Games often do a mix of both, like a bit of input lag for stability and prediction to deal with latency spikes. With video streaming, you limit your prediction options.
datalopers 4y ago
Most online games are 100% authoritative on the server already. Streaming rendered video is absurdly inefficient and does not help with the problems.
- AustinDev 4y ago
  Yeah, I've worked on AAA games with great networking stacks and the network traffic is orders of magnitude lower than streaming something like 4k@60fps to the client over x264.
- alduin32 4y ago
  It is inefficient, but it makes cheating much harder (but unfortunately still possible, it will never be possible to prevent it completely).
  - setr 4y ago
    Keeping the server 100% authoritative but maintaining all rendering on the client with no interpolation gets you all of the same benefits and drawbacks as shipping video, and ease of implementation, but with dramatically lower network costs. It’s not really done however except for turn-based games, because you still have horrendous input latency. It’s also entirely the same defense against cheating, except I suppose a user could do things like edit assets locally but I don’t think anyone cares about that concern
rossnordby 4y ago
Another related option that sidesteps a big chunk of perceptible latency is to send clients a trivially reprojectable scene. In other words, geometry or geometry proxies that even a mobile device could draw locally with up to date camera state at extremely low cost. The client would have very little responsibility and effectively no options for cheating. View independent shading response and light simulations can be shared between clients. Even much of view dependent response could be approximated in shared representations like spherical gaussians. The scene can also be aggressively prefiltered; prefiltering would also be shared across all clients. This would be a massive change in rendering architecture, there's no practical way to retrofit it onto any existing games, and it would still be incredibly expensive for servers compared to 'just let the client do it', and it can't address game logic latency without giving the client more awareness of game logic, but... seems potentially neat!

orev 4y ago

This seems so obvious to me that I’m surprised Stadia doesn’t already do that (not having done any real reading on how it works; just making assumptions based on the marketing I’ve seen). I just assumed it was a video stream with a touch control overlay.

milgrim 4y ago

How is that elegant? In return for easy synchronization the server has to perform ALL computations. Sounds more like brute force to me.

addingnumbers 4y ago
It's very elegant depending on your goals. It's the ultimate DRM for games, you can't crack a game's copy protection if you can't see it's binaries. You can't data-mine the locations of valuables if you don't have the map data. You can't leak unreleased assets ahead of their marketing debut if you don't have a copy of the assets. You can't expose all the Easter eggs by decompiling if you don't have the code. With subscription and premium currency models those abilities can all be interpreted as lost revenue. The markets for people buying $400 consoles vs buying a $20 HDMI stick and a $15/mo subscription are very different. After the colossal (and to me, surprising) rise of mobile gaming I think the latter might be where the real money will be 10 years from now. They'll address the bottlenecks on the data center end. I'm pretty sure you can list a dozen problems that make it prohibitively expensive right now and for every one of them some Microsoft or NVIDIA engineer can tell you how they are working on engineering away that problem in a couple years.
- milgrim 4y ago
  Of course, it solves a lot of problems. But you have to pay for it by increasing the load on the servers by orders of magnitude. This makes it a brute force approach in my opinion. This is of course depending on your meaning of „elegant“, but for me this would imply to solve these issues without increasing the server load so much. Let the client decide as much as possible, but check the decisions randomly and in case of suspicious player stats. And DRM for multiplayer games should be no problem anyways? Verify the accounts or your users, but that applies to stadia like services and also the „conventional“ ones. Solving the data mining issue is another topic and yes, giving the server more authority for things that the player should be able to see might be the only way to deal with this. But maybe the server could hand out client specific decryption keys when they are needed? That would be elegant, and not just keeping all the content server-side. Game streaming services will find their place, but they address mainly the entry hurdle and not the issue of game state synchronization.
atq2119 4y ago
Depending on how high the encode/decode/transmit overhead is, it might be a more efficient use of resources. Most game consoles and gaming PCs are idle most of the time. Centralizing the rendering in a data center is going to yield better resource utilization rates. Mind you, they're still not going to be great rates, because the data center needs to be geographically close and so utilization will vary a lot in daily and weekly patterns. Then again, maybe you can fill all those GPUs with ML training work when the gaming utilization is low...
- milgrim 4y ago
  Latency has a big impact on user experience in most multiplayer games and players with a low latency have an advantage in many games. I am not sure how satisfied people will be with low cost low powered systems made for game streaming in the long run when other users have a much better experience and even advantages in gameplay.

viktorcode 4y ago

You don't have to stream encoded video from the server. It is much faster and simpler to stream game state and have the game clients render it. Esentailly, that's how all the games with "authoritative server" work.

lewispollard 4y ago

Then you have a server rendering essentially 4 or 8 or 32 or whatever individual games, capturing and encoding them to a streamable format, streaming via a socket or webrtc or whatever to the client, who then sees the action, they input a control, the control has to get back to the server to be processed, render the next frame for every client, send them all back, and then the client sees the result of their action. Doesn't seem elegant to me, it seems like a way to have wildly uncontrollable latency, and have one player's poor connection disrupt everyone else's experience. I have a Steam Link hardware device to stream games from my PC to my TV over ethernet LAN, and even that can have issues with latency and encoding that make it a worse experience than just playing on the PC.

kaetemi 4y ago

Stream OpenGL rendering calls from the server to the client.

bob1029 4y ago
Not all clients have the same capabilities with regard to 3d hardware. Virtually any modern device can decode x264 without difficulty.
Const-me 4y ago
That works fine for simple 3D scenes. One application, that’s how MS remote desktop worked until recently — Windows streamed Direct3D rendering calls to the client. However, I’m not sure this gonna work with modern triple-A games. The current-gen low level GPU APIs were designed to allow developers of game engines to saturate bandwidth of PCI-express with these rendering calls. That’s too many gigabytes/second for networking, I’m afraid.

TinkersW 4y ago

That defeats most of what this article is talking about--hiding the perceived latency. In addition to the other issues like higher server cpu & bandwidth requirements, and poor video quality.

dylan604 4y ago

But it's not just one video render. It is one video render per user. People have to build pretty decent rigs to get their single view of the game to render at speed. Multiplying that times the number of players seems wildly expensive for a single machine to do to be a non-starter after 3 seconds of thought.

Const-me 4y ago

The article is about hiding latency between other people’s input, and your display. While it happens, your own input only has couple frames of latency. The solution you proposed introduces latency between your own input, and your own display. Unless the server is in your house or at least within 100km from it, that latency gonna make the game unplayable.

toast0 4y ago
Unless the server is in my house, I get at least 20ms round trip (thanks bonded VDSL2), so I'm a frame behind already. Add input sampling, codec delays, and video output delays and I'm at least two frames, probably three. That's going to be not great.

dbttdft 4y ago

there's a very small chance that cloud gaming may one day work. even if that small window of opportunity exists, the incompetent game industry will miss it.

GuB-42 4y ago

It doesn't solve the problem, in fact, it limits your options. And I may be wrong but x264 doesn't seem to have the lowest latency codecs. There are essentially two ways of dealing with latency : input lag (wait until we know everything before showing player actions) and prediction/rollback (respond immediately trying to guess what we don't know yet and fix it later, what is shown in the article). Games often do a mix of both, like a bit of input lag for stability and prediction to deal with latency spikes. With video streaming, you limit your prediction options.

datalopers 4y ago

Most online games are 100% authoritative on the server already. Streaming rendered video is absurdly inefficient and does not help with the problems.

AustinDev 4y ago
Yeah, I've worked on AAA games with great networking stacks and the network traffic is orders of magnitude lower than streaming something like 4k@60fps to the client over x264.
alduin32 4y ago
It is inefficient, but it makes cheating much harder (but unfortunately still possible, it will never be possible to prevent it completely).
- setr 4y ago
  Keeping the server 100% authoritative but maintaining all rendering on the client with no interpolation gets you all of the same benefits and drawbacks as shipping video, and ease of implementation, but with dramatically lower network costs. It’s not really done however except for turn-based games, because you still have horrendous input latency. It’s also entirely the same defense against cheating, except I suppose a user could do things like edit assets locally but I don’t think anyone cares about that concern

rossnordby 4y ago

Another related option that sidesteps a big chunk of perceptible latency is to send clients a trivially reprojectable scene. In other words, geometry or geometry proxies that even a mobile device could draw locally with up to date camera state at extremely low cost. The client would have very little responsibility and effectively no options for cheating. View independent shading response and light simulations can be shared between clients. Even much of view dependent response could be approximated in shared representations like spherical gaussians. The scene can also be aggressively prefiltered; prefiltering would also be shared across all clients. This would be a massive change in rendering architecture, there's no practical way to retrofit it onto any existing games, and it would still be incredibly expensive for servers compared to 'just let the client do it', and it can't address game logic latency without giving the client more awareness of game logic, but... seems potentially neat!