5 ms·

Hi everyone, I'm making a game engine that uses rollback netcode for its multiplayer architecture. As far as I can tell, no physics engine supports incremental

Hi everyone, I'm making a game engine that uses rollback netcode for its multiplayer architecture. As far as I can tell, no physics engine supports incremental rollback thus far. This means the entire physics engine state has to be snapshotted every frame, which basically means it's infeasible to have large worlds with rollback netcode. I've made a physics engine which only snapshots the changes, and so now I think you can have large worlds, as long as most of the world is static. I think that's true in most cases, like when you're walking around a big spaceship for example, all the walls, tables, control panels etc don't really move. I wrote up a bit of a post to describe some of the cool things I discovered while making my own physics engine.

Jyaif 6mo ago
How big is the state that you want to rollback?
stephc_int13 6mo ago
Have you considered the opportunity of using delta compression on snapshots? Like the internal state of the physics simulation, most of the gamestate itself don't change between frames. Using delta compression on the whole structure is doable.
- gafferongames 6mo ago
  The thing the author is trying to solve for here is reducing the amount of CPU used on the client when it rolls back the simulation and re-simulates to keep server authority. He does this by only rolling back and re-simulating only a subset of the world, greatly reducing the amount of CPU required. It's cool that he's approaching this from the point of view of adding support for it in the physics engine itself, vs. making it something that the game has to do themselves. Delta compression is an unrelated technique which reduce the amount of bandwidth sent from server to client, by sending only the differences between the snapshot at baseline frame n and the current snapshot frame m on the server. Just want to clear this up for anybody trying to follow along. Bringing in delta compression is an unrelated thing (but somewhat similar conceptually). It might confuse people to talk about these things at the same time, if they're really just trying to understand what the author is doing in the article. cheers - Glenn
  - BSTRhino 6mo ago
    Yes thank you, you understand perfectly and thank you for explaining. Also, I LOVE your series of blog posts, thank you for making them! Just to add to the general discussion for everyone following along - rollback netcode only sends inputs around, not state, so it doesn't really have much to send. I think I'm doing about 1.5 KB per second. When you point your mouse it sends that data in 10 bytes. There's not a lot to delta compress.
    - gafferongames 6mo ago
      One cool trick you could try (although you are probably doing it already) is to include all inputs for some long period (like 1-2 seconds!) in every input packet the client sends to the server. This way if one input packet gets lost, the very next one getting through will have all the inputs for the last 1-2 seconds, and this greatly improves how well your game will play under packet loss. When you do this, you can even do an encoding from left -> right for all inputs, and actually, sort of delta encode inputs within the packet! Inputs don't change that much, so you can even get smart with the encoding and optimize it down to basically nothing.
      - BSTRhino 6mo ago
        Ah yes, I have heard of this method but the idea of sorting/delta encoding is new to me and now I’m reconsidering! Maybe some people might find it interesting - I’m relaying packets from peer-to-peer using Cloudflare Realtime, which is like a one-to-many broadcast system for WebRTC. Each peer sends their input packet to Cloudflare, then Cloudflare forwards that on to the 10 other players (for example). It’s cool because (a) 10x less upload bandwidth from the peer (b) people IP addresses are not revealed to their peers and (c) Cloudflare is in 400 datacenters around the world so it adds minimal latency. Cloudflare Realtime is a really cool system that maybe more web game developers should look into! Unfortunately, I’m paying for all the bandwidth that goes through Cloudflare Realtime and so I have perhaps over optimised on minimising bandwidth by sending only one input per packet. The other part of my equation is I’m getting my server to broadcast authoritative batches of inputs every 100ms or so via TCP, so if a packet gets lost, every peer will eventually receive the input but it might be a bit slow, and it will cause a big rollback that might be noticeable. Reading your comment makes me think it might not be as expensive as I thought, and maybe I can play around with how long of an input period I resend for. Perhaps there is a better balance to strike between cost and reliability. So thanks for bringing this up!
- tikotus 6mo ago
  I was also curious about this, and I don't think the other replies understood what was being suggested. If I understood correctly, the aim of the engine is to lower the in-memory size of the history of game states, by only snapshotting the delta. I'm also curious what would happen if, instead, you'd just run any deterministic snapshottable physics engine, and delta compressed the history on the fly. I think this is how, for example, Braid works. Might be that it doesn't work, that running the delta check on two big enough snapshots would be too slow, and that's what this engine fixes. But would love to hear if it was considered.
gafferongames 6mo ago
I think this is a cool idea. Well done!
- BSTRhino 6mo ago
  Thank you so much!
ornitorrincos 6mo ago
How does it compare with deterministic physics engines, given that their appeal for multiplayer is that they can perform rollback?
- BSTRhino 6mo ago
  This physics engine is a deterministic physics engine. It has to be to make sure that when it rolls back and resimulates forward, it gets the same answer on all machines. The determinism is partly possible because WebAssembly is deterministic (except for a few known cases https://github.com/WebAssembly/design/blob/main/Nondeterminism.md https://github.com/WebAssembly/design/blob/main/Nondetermini...), and partly because I’m making sure to use my own trigonometric functions, and the entire game simulation is executed single threaded with a known order of execution. If you meant to ask how does it compare to non deterministic physics engines, I’m sure they might be faster on the physics but would be slower on the rollback, and I think on most reasonably-sized games the slow rollback would dominate and so they would be slower overall. But, you wouldn’t make a rollback netcode game with that size of world anyway, at least maybe not until now, so it’s a bit of a false comparison. They’re good at their different use cases.
  - wordpad 5mo ago
    Planetary Annihilation did it and wrote and gave talks about it.

Jyaif 6mo ago

How big is the state that you want to rollback?

stephc_int13 6mo ago

Have you considered the opportunity of using delta compression on snapshots? Like the internal state of the physics simulation, most of the gamestate itself don't change between frames. Using delta compression on the whole structure is doable.

gafferongames 6mo ago
The thing the author is trying to solve for here is reducing the amount of CPU used on the client when it rolls back the simulation and re-simulates to keep server authority. He does this by only rolling back and re-simulating only a subset of the world, greatly reducing the amount of CPU required. It's cool that he's approaching this from the point of view of adding support for it in the physics engine itself, vs. making it something that the game has to do themselves. Delta compression is an unrelated technique which reduce the amount of bandwidth sent from server to client, by sending only the differences between the snapshot at baseline frame n and the current snapshot frame m on the server. Just want to clear this up for anybody trying to follow along. Bringing in delta compression is an unrelated thing (but somewhat similar conceptually). It might confuse people to talk about these things at the same time, if they're really just trying to understand what the author is doing in the article. cheers - Glenn
- BSTRhino 6mo ago
  Yes thank you, you understand perfectly and thank you for explaining. Also, I LOVE your series of blog posts, thank you for making them! Just to add to the general discussion for everyone following along - rollback netcode only sends inputs around, not state, so it doesn't really have much to send. I think I'm doing about 1.5 KB per second. When you point your mouse it sends that data in 10 bytes. There's not a lot to delta compress.
  - gafferongames 6mo ago
    One cool trick you could try (although you are probably doing it already) is to include all inputs for some long period (like 1-2 seconds!) in every input packet the client sends to the server. This way if one input packet gets lost, the very next one getting through will have all the inputs for the last 1-2 seconds, and this greatly improves how well your game will play under packet loss. When you do this, you can even do an encoding from left -> right for all inputs, and actually, sort of delta encode inputs within the packet! Inputs don't change that much, so you can even get smart with the encoding and optimize it down to basically nothing.
    - BSTRhino 6mo ago
      Ah yes, I have heard of this method but the idea of sorting/delta encoding is new to me and now I’m reconsidering! Maybe some people might find it interesting - I’m relaying packets from peer-to-peer using Cloudflare Realtime, which is like a one-to-many broadcast system for WebRTC. Each peer sends their input packet to Cloudflare, then Cloudflare forwards that on to the 10 other players (for example). It’s cool because (a) 10x less upload bandwidth from the peer (b) people IP addresses are not revealed to their peers and (c) Cloudflare is in 400 datacenters around the world so it adds minimal latency. Cloudflare Realtime is a really cool system that maybe more web game developers should look into! Unfortunately, I’m paying for all the bandwidth that goes through Cloudflare Realtime and so I have perhaps over optimised on minimising bandwidth by sending only one input per packet. The other part of my equation is I’m getting my server to broadcast authoritative batches of inputs every 100ms or so via TCP, so if a packet gets lost, every peer will eventually receive the input but it might be a bit slow, and it will cause a big rollback that might be noticeable. Reading your comment makes me think it might not be as expensive as I thought, and maybe I can play around with how long of an input period I resend for. Perhaps there is a better balance to strike between cost and reliability. So thanks for bringing this up!
tikotus 6mo ago
I was also curious about this, and I don't think the other replies understood what was being suggested. If I understood correctly, the aim of the engine is to lower the in-memory size of the history of game states, by only snapshotting the delta. I'm also curious what would happen if, instead, you'd just run any deterministic snapshottable physics engine, and delta compressed the history on the fly. I think this is how, for example, Braid works. Might be that it doesn't work, that running the delta check on two big enough snapshots would be too slow, and that's what this engine fixes. But would love to hear if it was considered.

gafferongames 6mo ago

I think this is a cool idea. Well done!

BSTRhino 6mo ago
Thank you so much!

ornitorrincos 6mo ago

How does it compare with deterministic physics engines, given that their appeal for multiplayer is that they can perform rollback?

BSTRhino 6mo ago
This physics engine is a deterministic physics engine. It has to be to make sure that when it rolls back and resimulates forward, it gets the same answer on all machines. The determinism is partly possible because WebAssembly is deterministic (except for a few known cases https://github.com/WebAssembly/design/blob/main/Nondeterminism.md https://github.com/WebAssembly/design/blob/main/Nondetermini...), and partly because I’m making sure to use my own trigonometric functions, and the entire game simulation is executed single threaded with a known order of execution. If you meant to ask how does it compare to non deterministic physics engines, I’m sure they might be faster on the physics but would be slower on the rollback, and I think on most reasonably-sized games the slow rollback would dominate and so they would be slower overall. But, you wouldn’t make a rollback netcode game with that size of world anyway, at least maybe not until now, so it’s a bit of a false comparison. They’re good at their different use cases.
- wordpad 5mo ago
  Planetary Annihilation did it and wrote and gave talks about it.