4 ms·

> Do game engines make heavy use of threads though? Boy do they, yes.

by sho_hn 9y ago

iainmerrick 9y ago
Can you go into more detail? (Or point me at some up-to-date books or blog posts!) I'm specifically curious about CPU-intensive stuff that is sharded out to multiple threads. That's your classic multithreaded programming, the sort of thing you'd do in scientific computing, but I always got the impression games people were skeptical about it, due to unpredictable performance and the high risk of bugs.
- wlesieutre 9y ago
  Not an expert, but the version of it that I've heard is your main game loop handles all interaction with the game state to avoid issues. You can offload rendering into another thread pretty safely, and anything that alters game state would queue up its state changes for batch processing. That way if you're running a multithreaded physics simulation you can get two separate bullet collision detections on one object, and instead of trying to delete the object twice you put both "kill this thing" actions on a to-do list. When it comes time to handle that in the main loop, you sweep through it for conflicts before executing any of the changes. UE4 in particular I know handles all game logic in a single thread, and you can't touch UObjects from outside of that. But here's an example (without much technical detail) of someone implementing multithreaded pathfinding for UE4: https://forums.unrealengine.com/community/work-in-progress/11929-rama-s-multi-threaded-dynamic-pathing-system-full-physics-support https://forums.unrealengine.com/community/work-in-progress/1...
  - iainmerrick 9y ago
    UE4 in particular I know handles all game logic in a single thread, and you can't touch UObjects from outside of that. Yeah, that's the kind of thing I was thinking of when I said games programmers seemed skeptical of threads. Some of that coarse-grained parallelism could possibly be done in JS with Web Workers, but those have their own problems. (See the recent discussion here about the "tasklets" proposal: https://news.ycombinator.com/item?id=15511519 https://news.ycombinator.com/item?id=15511519)
- corysama 9y ago
  Motivation: The Xbox360 pretty much forced gamedevs into heavy threading if they wanted to get anything done. It had 3 PowerPC cores with 2 hardware threads each. The cores had huge memory latency and no out-of-order execution. IBM's attitude about OOE was "Statically compile for a fixed target" and "Run 2 threads per core and that that'll cut the effective stall cycles per thread in half". The PS3 had only 1 of those PC cores, but it also had 6 unique cores that were practically high-power DSPs. If you manually pipelined data movement and vectorized execution, you could get amazing results. If you ignored those cores, the PS3 was crippled. The XBone and PS4 have friendlier cores, but they are still surprisingly low-power and there are 8 of them. So, you still need to thread and vectorize or you'll be dragging. Even on the PC, Sutter's "The Free Lunch is Over" is over 12 years old. Outside of games, the browsers force single-threading and cloud servers profit by selling multicore machines pretending to be many single-core machines. But, in games have to run on non-virtual hardware. Execution: You can google around for "game engine job system", but unfortunately, game engine blogs have really fallen off a while back as most of that crew has moved to twitter. So, the best material out there is in the form of GDC presentations such as "Parallelizing the Naughty Dog engine using fibers", " Destiny's Multithreaded Rendering Architecture", "Multithreading the Entire Destiny Engine", "Killzone Shadow Fall: Threading the Entity Update on PS4". Slides and videos are available around the web.
  - iainmerrick 9y ago
    Cool, thanks! I never got to program one but I remember being fascinated by the weird architectures of the PS2 and PS3. It occurs to me that there are some similarly weird architectures in mobile right now -- it's not uncommon to see Android flagship phones with 8(!) cores, which is just ludicrous. And there are lots of asymmetric "big.LITTLE" designs with a mix of high-speed and low-power cores. Maybe I'm just reading the wrong blogs, but I've barely seen any discussion on how to optimize code for those crazy Android multicore CPUs, even though it seems like there's potentially a lot of upside. I guess Android is so fragmented and fast-moving that it's a tougher challenge than optimizing for two or three specific games consoles; also Android app prices are low so there probably isn't as much motivation. Also, Apple is miles and miles ahead of everybody else in mobile performance, and they've consistently gone with just 2-3 cores. Their mobile CPUs and GPUs are very smartly designed, really well-balanced.
- maccard 9y ago
  Here’s a good talk on a system used for the PS4 - https://www.gdcvault.com/play/1022186/Parallelizing-the-Naughty-Dog-Engine https://www.gdcvault.com/play/1022186/Parallelizing-the-Naug... And another one on multithreading a cross platform game - https://m.youtube.com/watch?v=v2Q_zHG3vqg https://m.youtube.com/watch?v=v2Q_zHG3vqg There’s also this talk - http://developer2.download.nvidia.com/assets/gameworks/downloads/regular/GDC17/Game_Physics_on_the_GPU_with_PhysX3_GDC17_FINAL.pdf?vfd5F0nkud3wgQLs-TtorS6kljzqV2aiiP7aOPlqLTK7PN1KtA4PBcXgFl9YMILGzupfbKehL1T7BImeXAKK54vZWb5qto90CbnQu_7wIyLPcryA_6L0uTuHRLHKgl8V7XQPu1V51EBpaw3hUQNn2pEtIv0cuuMg4VjpJKOiyuBX4TC0C9bNNEVd1tRORLp96g7WzrL9o8ywv4A http://developer2.download.nvidia.com/assets/gameworks/downl... which talks about collision detection and collision resolution on the GPU that actually explains the architecture of the cpu engine quite well and shows what parts are serial and what parts are well parallelised.
- looki 9y ago
  A bit older but this presentation about Killzone 2's usage of the PS3's Cell architecture was pretty neat https://www.guerrilla-games.com/read/the-playstation-3s-spus-in-the-real-world-a-killzone-2-case-study https://www.guerrilla-games.com/read/the-playstation-3s-spus...
- robbies 9y ago
  Modern game engines parcel out work via a job system, where functional tasks are dispatched to the cores in a system. This is opposed to the idea of 'one core/thread will own a task for the lifetime of the application, while checking in with a master core/thread'. Actually, most games have a hybrid model. Usually one thread/core is dedicated to rendering, another thread/core is dedicated to high priority tasks/jobs, and then the remaining resources are used by whatever jobs are left. http://fabiensanglard.net/doom3_bfg/threading.php http://fabiensanglard.net/doom3_bfg/threading.php
  - iainmerrick 9y ago
    That threading style potentially fits OK with the JS "Web Worker" / "isolate" model, as long as you can pass messages around between threads/isolates very efficiently (probably not the case in current JS implementations).
    - TheHegemon 9y ago
      There are Transferable objects (https://developer.mozilla.org/en-US/docs/Web/API/Transferable https://developer.mozilla.org/en-US/docs/Web/API/Transferabl...) so you at least don't have to serialize the data between workers.
    - robbies 9y ago
      Oh interesting. TBH, I have zero knowledge on how to do any of this on the web side. All my experience is from working with traditional game engines on the native side.
      - mhink 9y ago
        The worker pooling system you describe is eminently possible in the browser these days. Web Workers [1] are really just threads with a JS execution context and a facility for messaging back to the thread which created them. (Or, if you set them up with a MessageChannel [2], they can do full-duplex messaging with any thread that gets the other end of the pipe) Of course, you're still dealing with the event loop in most cases, which is probably a stumbling block when it comes to really low-level stuff. That said, there are even facilities for shared memory and atomics operations [3] these days, which helps. I've messed around with it a little bit on a side project- as a JS developer, it's really weird and fun to say "screw the event loop!" and just enter an endless synchronous loop. :D [1] https://developer.mozilla.org/en-US/docs/Web/API/Worker https://developer.mozilla.org/en-US/docs/Web/API/Worker [2] https://developer.mozilla.org/en-US/docs/Web/API/MessageChannel/MessageChannel https://developer.mozilla.org/en-US/docs/Web/API/MessageChan... [3] https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Atomics https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...

> Do game engines make heavy use of threads though? Boy do they, yes.

by sho_hn 9y ago

> Do game engines make heavy use of threads though? Boy do they, yes.

Can you go into more detail? (Or point me at some up-to-date books or blog posts!) I'm specifically curious about CPU-intensive stuff that is sharded out to multiple threads. That's your classic multithreaded programming, the sort of thing you'd do in scientific computing, but I always got the impression games people were skeptical about it, due to unpredictable performance and the high risk of bugs.

wlesieutre 9y ago
Not an expert, but the version of it that I've heard is your main game loop handles all interaction with the game state to avoid issues. You can offload rendering into another thread pretty safely, and anything that alters game state would queue up its state changes for batch processing. That way if you're running a multithreaded physics simulation you can get two separate bullet collision detections on one object, and instead of trying to delete the object twice you put both "kill this thing" actions on a to-do list. When it comes time to handle that in the main loop, you sweep through it for conflicts before executing any of the changes. UE4 in particular I know handles all game logic in a single thread, and you can't touch UObjects from outside of that. But here's an example (without much technical detail) of someone implementing multithreaded pathfinding for UE4: https://forums.unrealengine.com/community/work-in-progress/11929-rama-s-multi-threaded-dynamic-pathing-system-full-physics-support https://forums.unrealengine.com/community/work-in-progress/1...
- iainmerrick 9y ago
  UE4 in particular I know handles all game logic in a single thread, and you can't touch UObjects from outside of that. Yeah, that's the kind of thing I was thinking of when I said games programmers seemed skeptical of threads. Some of that coarse-grained parallelism could possibly be done in JS with Web Workers, but those have their own problems. (See the recent discussion here about the "tasklets" proposal: https://news.ycombinator.com/item?id=15511519 https://news.ycombinator.com/item?id=15511519)
corysama 9y ago
Motivation: The Xbox360 pretty much forced gamedevs into heavy threading if they wanted to get anything done. It had 3 PowerPC cores with 2 hardware threads each. The cores had huge memory latency and no out-of-order execution. IBM's attitude about OOE was "Statically compile for a fixed target" and "Run 2 threads per core and that that'll cut the effective stall cycles per thread in half". The PS3 had only 1 of those PC cores, but it also had 6 unique cores that were practically high-power DSPs. If you manually pipelined data movement and vectorized execution, you could get amazing results. If you ignored those cores, the PS3 was crippled. The XBone and PS4 have friendlier cores, but they are still surprisingly low-power and there are 8 of them. So, you still need to thread and vectorize or you'll be dragging. Even on the PC, Sutter's "The Free Lunch is Over" is over 12 years old. Outside of games, the browsers force single-threading and cloud servers profit by selling multicore machines pretending to be many single-core machines. But, in games have to run on non-virtual hardware. Execution: You can google around for "game engine job system", but unfortunately, game engine blogs have really fallen off a while back as most of that crew has moved to twitter. So, the best material out there is in the form of GDC presentations such as "Parallelizing the Naughty Dog engine using fibers", " Destiny's Multithreaded Rendering Architecture", "Multithreading the Entire Destiny Engine", "Killzone Shadow Fall: Threading the Entity Update on PS4". Slides and videos are available around the web.
- iainmerrick 9y ago
  Cool, thanks! I never got to program one but I remember being fascinated by the weird architectures of the PS2 and PS3. It occurs to me that there are some similarly weird architectures in mobile right now -- it's not uncommon to see Android flagship phones with 8(!) cores, which is just ludicrous. And there are lots of asymmetric "big.LITTLE" designs with a mix of high-speed and low-power cores. Maybe I'm just reading the wrong blogs, but I've barely seen any discussion on how to optimize code for those crazy Android multicore CPUs, even though it seems like there's potentially a lot of upside. I guess Android is so fragmented and fast-moving that it's a tougher challenge than optimizing for two or three specific games consoles; also Android app prices are low so there probably isn't as much motivation. Also, Apple is miles and miles ahead of everybody else in mobile performance, and they've consistently gone with just 2-3 cores. Their mobile CPUs and GPUs are very smartly designed, really well-balanced.
maccard 9y ago
Here’s a good talk on a system used for the PS4 - https://www.gdcvault.com/play/1022186/Parallelizing-the-Naughty-Dog-Engine https://www.gdcvault.com/play/1022186/Parallelizing-the-Naug... And another one on multithreading a cross platform game - https://m.youtube.com/watch?v=v2Q_zHG3vqg https://m.youtube.com/watch?v=v2Q_zHG3vqg There’s also this talk - http://developer2.download.nvidia.com/assets/gameworks/downloads/regular/GDC17/Game_Physics_on_the_GPU_with_PhysX3_GDC17_FINAL.pdf?vfd5F0nkud3wgQLs-TtorS6kljzqV2aiiP7aOPlqLTK7PN1KtA4PBcXgFl9YMILGzupfbKehL1T7BImeXAKK54vZWb5qto90CbnQu_7wIyLPcryA_6L0uTuHRLHKgl8V7XQPu1V51EBpaw3hUQNn2pEtIv0cuuMg4VjpJKOiyuBX4TC0C9bNNEVd1tRORLp96g7WzrL9o8ywv4A http://developer2.download.nvidia.com/assets/gameworks/downl... which talks about collision detection and collision resolution on the GPU that actually explains the architecture of the cpu engine quite well and shows what parts are serial and what parts are well parallelised.
looki 9y ago
A bit older but this presentation about Killzone 2's usage of the PS3's Cell architecture was pretty neat https://www.guerrilla-games.com/read/the-playstation-3s-spus-in-the-real-world-a-killzone-2-case-study https://www.guerrilla-games.com/read/the-playstation-3s-spus...
robbies 9y ago
Modern game engines parcel out work via a job system, where functional tasks are dispatched to the cores in a system. This is opposed to the idea of 'one core/thread will own a task for the lifetime of the application, while checking in with a master core/thread'. Actually, most games have a hybrid model. Usually one thread/core is dedicated to rendering, another thread/core is dedicated to high priority tasks/jobs, and then the remaining resources are used by whatever jobs are left. http://fabiensanglard.net/doom3_bfg/threading.php http://fabiensanglard.net/doom3_bfg/threading.php
- iainmerrick 9y ago
  That threading style potentially fits OK with the JS "Web Worker" / "isolate" model, as long as you can pass messages around between threads/isolates very efficiently (probably not the case in current JS implementations).
  - TheHegemon 9y ago
    There are Transferable objects (https://developer.mozilla.org/en-US/docs/Web/API/Transferable https://developer.mozilla.org/en-US/docs/Web/API/Transferabl...) so you at least don't have to serialize the data between workers.
  - robbies 9y ago
    Oh interesting. TBH, I have zero knowledge on how to do any of this on the web side. All my experience is from working with traditional game engines on the native side.
    - mhink 9y ago
      The worker pooling system you describe is eminently possible in the browser these days. Web Workers [1] are really just threads with a JS execution context and a facility for messaging back to the thread which created them. (Or, if you set them up with a MessageChannel [2], they can do full-duplex messaging with any thread that gets the other end of the pipe) Of course, you're still dealing with the event loop in most cases, which is probably a stumbling block when it comes to really low-level stuff. That said, there are even facilities for shared memory and atomics operations [3] these days, which helps. I've messed around with it a little bit on a side project- as a JS developer, it's really weird and fun to say "screw the event loop!" and just enter an endless synchronous loop. :D [1] https://developer.mozilla.org/en-US/docs/Web/API/Worker https://developer.mozilla.org/en-US/docs/Web/API/Worker [2] https://developer.mozilla.org/en-US/docs/Web/API/MessageChannel/MessageChannel https://developer.mozilla.org/en-US/docs/Web/API/MessageChan... [3] https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Atomics https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...