6 ms·

I think the async/await patterns solve one problem really well: UI latency. My UI background is web testing and C++ game UIs in C++. There are like 3 patterns

by sqeaky 2y ago

I think the async/await patterns solve one problem really well: UI latency. My UI background is web testing and C++ game UIs in C++. There are like 3 patterns in multithreaded games. Thread per responsibility (old/bad), Barriers blocking all threads and give each game system all the threads, or something smart. Few games do something smart, and the thread per responsibility is not ideal and we will ignore it for now. In games often pausing everything and letting the physics system have all the threads for a few milliseconds is "fast enough". Then the graphics systems will try to use all the threads, and so on so that eventually everything will get all the threads even thought everything else rarely needs it. Sometimes two things are both close to single threaded and have no data contention so they might both be given threads, but this is almost always a manually decided things by experts. This means that once UI, the buttons, the text, cursors, status bars, etc, gets its turn there won't be any race conditions (good), but if it needs to request something from disk that pause will happen on a thread in the UI system (bad and analogous to web sites making web API calls) so UI latency can be a real problem. If any IO small or some resource system has preloaded it then there isn't a detectable slowdown, but there are still plenty of silly periods of waiting. There is also a lot of time when some single threaded part of the game isn't using N-1 hardware threads and all that IO could have been asynchronous. But often game UIs are a frame behind the rest of the game simulation and there is often detectable latency in the UI like the mouse feeling like it drags or similar. Allowing IO to run in the back while active events are processed can reduce latency and this is the default in web browsers. IO latency in web pages is worse than in games and other computation seems smaller than games, so the the event loop is close to ideal. A function is waiting? throw it on the stack and grab something else to do! This means that all the work that can be done while waiting on IO is done and when does well makes a UI snappy. If that were available sensibly in games it could allow a game designed appropriately to span IO across multiple frames and be snappy without stutters. With games using the strategy I described above latency in the game simulation or IO can cause the UI to feel sluggish and vice versa. In games caching UI details and trying to pump the frame rate is "good enough". If the UI is a frame behind but we have 200 frames per second, that isn't really a problem. But when it chugs and the mouse stops responding because the player built the whole game world out of dynamite and set it off the game will not process the mouse until that 30 minutes of physics work is done. There are better scheduling schemes for games. I am a big fan of doing "something smart" but that usually means scheduling heterogenous work with nuanced dependencies and I have written libraries just for that because it isn't actually that hard. But if you don't have the raw compute demands of a game scheduling IO along side your UI computation is often "fast enough" and is any easy enough mental model for JS devs to grok and allow them freedom to speed things up with their own solutions like caching schemes and reworking their UIs.

jayd16 2y ago
Isn't async/await "scheduling heterogenous work with nuanced dependencies"? Or is that what you were implying? Although my real guess is ECS but that's more like the "everyone gets every thread for a time."
- sqeaky 2y ago
  TLDR; I hadn't meant it that way, but in web pages it really is enough. Web pages generally don't have computation time to worry about, mostly just IO. This simplifies scheduling because whatever is coordinating the event loop in the browser (or other UI) can just background any amount of independent IO tasks. If there is computation screwing with share mutable state something with internal knowledge needs to be involved and that isn't current event loops, but in the long run it could be. Sorry for the novel. I meant those nuanced dependencies as a way of managing shared mutable state and complex computations that really do take serious CPU time. Let's make a simple example from a complex game design. This example is ridiculous but conveys the real nature of the problems with CPU and IO. Consider these pieces of work that might exist in a hypothetical game where NPCs and a player are moving around a simulated environment with some physics calculations and that the physics simulation is the source of truth for locations of items in the game. Here are parts of a game: Physics broad phase: Trivially parallelizable, depends on previous frame. Produces "islands" of physics objects. Imagine two piles of stuff miles apart, they can't interact except with items in the same pile, each island is just a pile of math to do. Perhaps in this game this might take 20 ms of CPU time. Across the minimum target machine with 4 cores that is 5ms apiece. Physics narrow phase: Each physics island is single threaded but threadsafe from each other, depends on broad phase to produce islands. Each island takes unknown and unequal time, likely between 0 and 2 ms of just math. Graphics scene/render: Might have a scene graph culling that is parallelizable, and converts game state into a series of commands independent of any specific GPU API. Depends on all physics completing because that is what it is drawing. Likely 1 or 2 ms per island. Graphics draw calls: Single threaded, sends render results to GPU using directx/opengl/vulkan/metal. This converts the independent render commands to API specific commands. Likely less than 1 ms of actual CPU work, but larger wait on GPU because it is IO. NPC AI: NPCs are independent but light weight so threading makes no sense if there are fewer than hundreds. Depends on physics to know what NPCs are responding to. Wants to add forces to the physics sim next frame. For this game lets say there are many, I don't know maybe this is dynasty warriors or something, so lets say a 1~3 ms. User input: Single threaded, will to add forces to the physics sim next frame based on user commands. Can't run at the same time as NPCs because both want to mutate the physics state. Less than 1 ms. We are ignoring: Sound, Network, Environment, Disk IO, OS events (window resize, etc), UI (not buttons or text positioning), and a few other things. A first attempt at a real game would likely be coded to give all the threads to each piece of work one at a time in some hand picked order, or at least until this was demonstrate to be slow: Physics Broad -> Physics Narrow -> Graphics render -> Graphic GPU -> NPC AI -> User input -> Wait/Next frame But that is likely slow, and I picked our hypothetical math to be slow and marginal. Sending stuff to the GPU is a high latency activity it might take 5 ms to respond, and if this is a 60 FPS game then that is like 1/3 of our time. If we simply add our hypothetical times that is frequently more than 16ms making the game slower than 60fps. Even an ideal frame with just a little physics is right at 15 to 16 ms So a practical game studio might do other work while waiting on the GPU to respond: Physics Broad -> Physics Narrow -> Graphics render -> At the same time: { Graphics GPU calls (Uses one thread) NPC AI (Uses all but one thread) -> User input } -> Wait/Next frame Most of the time something like this is "fast enough". In this example that 5 ms of CPU time wait on the GPU is now running alongside all that NPC AI so we only need to add the larger of the two. If this takes only a few days of engineer time and keeps the game under the 16ms on most machines then maybe the team makes a business decision to raise the minimum specs just a bit (from 4 to 6 cores would reduce physics time by another ms) and now can they ship this game. There are still needless waits and from a purely GigaFLOPS perspective perhaps much weaker computers could do the work but there is so much waiting that it isn't practical. But this compromise gets all target machines to just about 60 FPS. Alternatively, if the game is smart enough to make new threads of work for each physics islands (actually super complex and not a great idea in real game, but this all hypothetical but there are similar wins to be had in real games) and manage dependencies carefully based on the the simulation state then something more detailed might be possible: 1. Physics broadphase, create known amount of physics islands. 2. Start a paused GPU thread waiting on known amount of physics islands to be done rendering. This will start step 5 as soon as the last step 4c completes. 3. Add the player's input work to the appropriate group of NPCs 4. Each Physics island gets a thread that does the following: a. Physics narrow phase for this island, b. Partial render for just this island, c. Sets a threadsafe flag this island is done, d. NPC AI is processed for NPCs near this physics island, e. If this is the island with the player process their input. 5. The GPU thread waits for all physics islands threads to get to step 3c then starts sending commands to the GPU! and 3d gets to keep running. 6. When all threads from step 4 and 5 complete pause all game threads to hit the desired frame rate (save battery life for mobile gamers!) or advance to next thread if past frame budget or framerate is uncapped. This moves all the waits to the end of each thread's frame runtime. This means a bunch of nice things. That last thread can likely do some turbo boosting, a feature of most CPUs where they clock up one CPU if it is the only one working. If the NPCs ever took longer than the GPU they still might complete earlier because they get started earlier. If there are more islands than hardware threads this likely results in better utilization because there are no early pauses. This would likely a take a ton of engineering time. This might move the frame time down a few more ms and maybe let them lower the minimum requirements perhaps even letting the game run on an older console if market conditions support that. Conceptually, it might be a thing that could be done with async/await, but I don't think that is how most game engines are designed. I also think this makes dependencies implicit and scattered through the code, but likely that could be avoided with careful design. I am a big fan of libraries that let you provide work units, or functors, and say which depend on each other. They all get to read/write to the global state, but with the dependencies there won't be race conditions. Such libraries locate the threading logic in one place. Then if there is some particularly contentious state that many things need to touch it can be wrapped in a mutex. I suppose this might just be the iterative vs recursive discussion applied to threading strategies. It just happens that most event loops are single threaded, no reason they need to be in the long run. In the long run I could see making that fastest scenario happen in either paradigm even though the code would look completely different.

by sqeaky 2y ago

jayd16 2y ago

Isn't async/await "scheduling heterogenous work with nuanced dependencies"? Or is that what you were implying? Although my real guess is ECS but that's more like the "everyone gets every thread for a time."

sqeaky 2y ago
TLDR; I hadn't meant it that way, but in web pages it really is enough. Web pages generally don't have computation time to worry about, mostly just IO. This simplifies scheduling because whatever is coordinating the event loop in the browser (or other UI) can just background any amount of independent IO tasks. If there is computation screwing with share mutable state something with internal knowledge needs to be involved and that isn't current event loops, but in the long run it could be. Sorry for the novel. I meant those nuanced dependencies as a way of managing shared mutable state and complex computations that really do take serious CPU time. Let's make a simple example from a complex game design. This example is ridiculous but conveys the real nature of the problems with CPU and IO. Consider these pieces of work that might exist in a hypothetical game where NPCs and a player are moving around a simulated environment with some physics calculations and that the physics simulation is the source of truth for locations of items in the game. Here are parts of a game: Physics broad phase: Trivially parallelizable, depends on previous frame. Produces "islands" of physics objects. Imagine two piles of stuff miles apart, they can't interact except with items in the same pile, each island is just a pile of math to do. Perhaps in this game this might take 20 ms of CPU time. Across the minimum target machine with 4 cores that is 5ms apiece. Physics narrow phase: Each physics island is single threaded but threadsafe from each other, depends on broad phase to produce islands. Each island takes unknown and unequal time, likely between 0 and 2 ms of just math. Graphics scene/render: Might have a scene graph culling that is parallelizable, and converts game state into a series of commands independent of any specific GPU API. Depends on all physics completing because that is what it is drawing. Likely 1 or 2 ms per island. Graphics draw calls: Single threaded, sends render results to GPU using directx/opengl/vulkan/metal. This converts the independent render commands to API specific commands. Likely less than 1 ms of actual CPU work, but larger wait on GPU because it is IO. NPC AI: NPCs are independent but light weight so threading makes no sense if there are fewer than hundreds. Depends on physics to know what NPCs are responding to. Wants to add forces to the physics sim next frame. For this game lets say there are many, I don't know maybe this is dynasty warriors or something, so lets say a 1~3 ms. User input: Single threaded, will to add forces to the physics sim next frame based on user commands. Can't run at the same time as NPCs because both want to mutate the physics state. Less than 1 ms. We are ignoring: Sound, Network, Environment, Disk IO, OS events (window resize, etc), UI (not buttons or text positioning), and a few other things. A first attempt at a real game would likely be coded to give all the threads to each piece of work one at a time in some hand picked order, or at least until this was demonstrate to be slow: Physics Broad -> Physics Narrow -> Graphics render -> Graphic GPU -> NPC AI -> User input -> Wait/Next frame But that is likely slow, and I picked our hypothetical math to be slow and marginal. Sending stuff to the GPU is a high latency activity it might take 5 ms to respond, and if this is a 60 FPS game then that is like 1/3 of our time. If we simply add our hypothetical times that is frequently more than 16ms making the game slower than 60fps. Even an ideal frame with just a little physics is right at 15 to 16 ms So a practical game studio might do other work while waiting on the GPU to respond: Physics Broad -> Physics Narrow -> Graphics render -> At the same time: { Graphics GPU calls (Uses one thread) NPC AI (Uses all but one thread) -> User input } -> Wait/Next frame Most of the time something like this is "fast enough". In this example that 5 ms of CPU time wait on the GPU is now running alongside all that NPC AI so we only need to add the larger of the two. If this takes only a few days of engineer time and keeps the game under the 16ms on most machines then maybe the team makes a business decision to raise the minimum specs just a bit (from 4 to 6 cores would reduce physics time by another ms) and now can they ship this game. There are still needless waits and from a purely GigaFLOPS perspective perhaps much weaker computers could do the work but there is so much waiting that it isn't practical. But this compromise gets all target machines to just about 60 FPS. Alternatively, if the game is smart enough to make new threads of work for each physics islands (actually super complex and not a great idea in real game, but this all hypothetical but there are similar wins to be had in real games) and manage dependencies carefully based on the the simulation state then something more detailed might be possible: 1. Physics broadphase, create known amount of physics islands. 2. Start a paused GPU thread waiting on known amount of physics islands to be done rendering. This will start step 5 as soon as the last step 4c completes. 3. Add the player's input work to the appropriate group of NPCs 4. Each Physics island gets a thread that does the following: a. Physics narrow phase for this island, b. Partial render for just this island, c. Sets a threadsafe flag this island is done, d. NPC AI is processed for NPCs near this physics island, e. If this is the island with the player process their input. 5. The GPU thread waits for all physics islands threads to get to step 3c then starts sending commands to the GPU! and 3d gets to keep running. 6. When all threads from step 4 and 5 complete pause all game threads to hit the desired frame rate (save battery life for mobile gamers!) or advance to next thread if past frame budget or framerate is uncapped. This moves all the waits to the end of each thread's frame runtime. This means a bunch of nice things. That last thread can likely do some turbo boosting, a feature of most CPUs where they clock up one CPU if it is the only one working. If the NPCs ever took longer than the GPU they still might complete earlier because they get started earlier. If there are more islands than hardware threads this likely results in better utilization because there are no early pauses. This would likely a take a ton of engineering time. This might move the frame time down a few more ms and maybe let them lower the minimum requirements perhaps even letting the game run on an older console if market conditions support that. Conceptually, it might be a thing that could be done with async/await, but I don't think that is how most game engines are designed. I also think this makes dependencies implicit and scattered through the code, but likely that could be avoided with careful design. I am a big fan of libraries that let you provide work units, or functors, and say which depend on each other. They all get to read/write to the global state, but with the dependencies there won't be race conditions. Such libraries locate the threading logic in one place. Then if there is some particularly contentious state that many things need to touch it can be wrapped in a mutex. I suppose this might just be the iterative vs recursive discussion applied to threading strategies. It just happens that most event loops are single threaded, no reason they need to be in the long run. In the long run I could see making that fastest scenario happen in either paradigm even though the code would look completely different.