3 ms·

> There are many hard problems in video game performance, but "games just can't make good use of multiple cores" is not one of them. That's not quite what I s

> There are many hard problems in video game performance, but "games just can't make good use of multiple cores" is not one of them. That's not quite what I said. I'm happy you're using specs to work on an open source game. Specs and bevy and all the ECS work being done is super exciting and fun. I <3 Rust. In the meantime there are no major games that effectively scale to, let's say, 64 cores. I don't know of anything shipped that can saturate a 12-core/24-thread Ryzen. ECS alone will not get us to that level of scaling. Yes it's trivial to to throw audio, networking, and a few other subsystems onto separate threads. Modern games definitely leverage 4 cores. Although several of those cores will be severely under utilized. Modern ECS designs are rapidly evolving and rapidly improving our ability to better leverage multiple cores. But we're not yet to a point where games can easily and efficiently saturate 10+ cores. Personally I'd love to see a game like Eve Online that can effectively simulate a universe with tens of thousands of players either spread across the universe or all in one places during one giant battle. > It is certainly far far easier to see real performance wins with multicore than by using multiple servers, which introduce very heavy coordination costs. This is extremely true.

Jweb_Guru 5y ago
If you're just using ECS to parallelize disjoint subsystems then no, it won't get you there. But if you're using it (as Veloren is, and as I hope more people do) in a more deliberate way, to further parallelize within a system, you can indeed scale quite well to large numbers of cores. I've done some theoretical bottleneck calculations and we will still have tons of work to do with 64 cores available, if the game is written properly. We can already get decent utilization out of 32 threads at busy times, and the server was not really close to peak load in terms of player count (and we are very far from done optimizing): https://media.discordapp.net/attachments/539518074106413056/825375910969737287/unknown.png https://media.discordapp.net/attachments/539518074106413056/.... That's just for our server. Our clients can make use of cores in even more ways, although they have less work to do and generally have fewer cores available, and you can see Veloren taking advantage of 16 client threads with similar utilization here: https://twitter.com/sahajsarup/status/1431837669391142916 https://twitter.com/sahajsarup/status/1431837669391142916. The most important thing I want to note is that in both cases, you are not seeing tremendous imbalance between the cores most of the time. While there are definitely single-threaded bottlenecks in games, you have to be working pretty hard before they start bottlenecking the workload! Instead, we are just suffering from a combination of general inefficiency and lack of work to do. So no, I'm going to push back against this notion that multicore scaling for games is some sort of crazy intractable problem. It's not. Like any other kind of parallel scaling, it's trivial in some places, more challenging in others, and depends a lot on your workload (including you actually having enough work to saturate the cores in the first place!). But there's nothing special about games here.
- forrestthewoods 5y ago
  The original question was: > Is there a specific reason why Minecraft hasn’t been “adapted” from the original game to a robust design that could scale to larger worlds and player populations before your project ? The answer is because it's a lot of hard work. I am happy that a bunch of smart and talented people are working really hard to optimize Veloren. Good for you. I hope you help push the state of the art.
  - Jweb_Guru 5y ago
    We have plenty of contributors who have never even programmed before, and certainly have no experience with parallel programming, and the bulk of our physics parallelization (easily the trickiest part to make concurrent) has been done by people who aren't professional programmers. And I don't think I can recall a single change (proposed or implemented) by any of these contributors that resulted in a significant reduction in parallelization opportunities, nor any that found the difficulty of contributing to have been increased by the fact that much of the game is parallelized--so this is not primarily a result of benevolent guardians gatekeeping unfriendly features, or anything like that. I can also count on one hand the number of bugs we've had due to race conditions caused by parallelizing things that were previously single-threaded. I think that modern tooling, languages, and libraries have gotten good enough that correct parallel programming is no longer nearly as hard as it's reputed to be. Nor is scaling well on multicore the "point" of the game or even an explicit goal (though handling lots of players is)--taking advantage of multiple CPU cores is just one of many ways to improve performance, which we try to tackle on multiple fronts (including increased utilization of the GPU, explicit SIMD, smarter algorithms, allocation reduction, structure compression for improved cache locality and network utilization, etc. etc.). We haven't made any special effort to parallelize at the expense of single-threaded optimizations, and generally only do parallelization within a system, move things to the background, etc. where it is revealed as a bottleneck. And we've mostly done so by utilizing existing Rust libraries like crossbeam, specs, rayon, and wgpu, not rolling our own stuff. So again, there is nothing at all special about Veloren's design or focus here that makes it more amenable to parallelization than any other game would be, despite it being in a genre that is supposedly difficult to make scale. And that's the thing I'm specifically trying to push back on--the idea that multicore scaling for games is only possible if you have some dedicated cabal of programming wizards who want to push the state of the art. We live in the age of libraries, and wizards are only needed deep in the guts of the implementations of those libraries (just as they always have been, and probably will be until the end of time). A game programmer does not need to understand how a lockfree work stealing queue is implemented (or even what it is!) in order to use "parallel for" to beat the snot out of a carefully optimized single-threaded version of the same task, and it's usually far easier to do the former than the latter. I certainly understand why for a game like Minecraft, with a lot of legacy mechanics and mods that were never designed to be threadsafe, or a game engine like Unity, Unreal, or Roblox, that similarly have lots of plugins and customer code they would like to keep working, it would be very challenging to parallelize after the fact. And naturally, there are limits to what you can do on a single system, and your game design options become far more restricted once you're talking about 10k rather than 1k concurrent players. But for a brand new game without any legacy baggage, there's really no reason why it should scale poorly on multicore systems.

Jweb_Guru 5y ago

If you're just using ECS to parallelize disjoint subsystems then no, it won't get you there. But if you're using it (as Veloren is, and as I hope more people do) in a more deliberate way, to further parallelize within a system, you can indeed scale quite well to large numbers of cores. I've done some theoretical bottleneck calculations and we will still have tons of work to do with 64 cores available, if the game is written properly. We can already get decent utilization out of 32 threads at busy times, and the server was not really close to peak load in terms of player count (and we are very far from done optimizing): https://media.discordapp.net/attachments/539518074106413056/825375910969737287/unknown.png https://media.discordapp.net/attachments/539518074106413056/.... That's just for our server. Our clients can make use of cores in even more ways, although they have less work to do and generally have fewer cores available, and you can see Veloren taking advantage of 16 client threads with similar utilization here: https://twitter.com/sahajsarup/status/1431837669391142916 https://twitter.com/sahajsarup/status/1431837669391142916. The most important thing I want to note is that in both cases, you are not seeing tremendous imbalance between the cores most of the time. While there are definitely single-threaded bottlenecks in games, you have to be working pretty hard before they start bottlenecking the workload! Instead, we are just suffering from a combination of general inefficiency and lack of work to do. So no, I'm going to push back against this notion that multicore scaling for games is some sort of crazy intractable problem. It's not. Like any other kind of parallel scaling, it's trivial in some places, more challenging in others, and depends a lot on your workload (including you actually having enough work to saturate the cores in the first place!). But there's nothing special about games here.

forrestthewoods 5y ago
The original question was: > Is there a specific reason why Minecraft hasn’t been “adapted” from the original game to a robust design that could scale to larger worlds and player populations before your project ? The answer is because it's a lot of hard work. I am happy that a bunch of smart and talented people are working really hard to optimize Veloren. Good for you. I hope you help push the state of the art.
- Jweb_Guru 5y ago
  We have plenty of contributors who have never even programmed before, and certainly have no experience with parallel programming, and the bulk of our physics parallelization (easily the trickiest part to make concurrent) has been done by people who aren't professional programmers. And I don't think I can recall a single change (proposed or implemented) by any of these contributors that resulted in a significant reduction in parallelization opportunities, nor any that found the difficulty of contributing to have been increased by the fact that much of the game is parallelized--so this is not primarily a result of benevolent guardians gatekeeping unfriendly features, or anything like that. I can also count on one hand the number of bugs we've had due to race conditions caused by parallelizing things that were previously single-threaded. I think that modern tooling, languages, and libraries have gotten good enough that correct parallel programming is no longer nearly as hard as it's reputed to be. Nor is scaling well on multicore the "point" of the game or even an explicit goal (though handling lots of players is)--taking advantage of multiple CPU cores is just one of many ways to improve performance, which we try to tackle on multiple fronts (including increased utilization of the GPU, explicit SIMD, smarter algorithms, allocation reduction, structure compression for improved cache locality and network utilization, etc. etc.). We haven't made any special effort to parallelize at the expense of single-threaded optimizations, and generally only do parallelization within a system, move things to the background, etc. where it is revealed as a bottleneck. And we've mostly done so by utilizing existing Rust libraries like crossbeam, specs, rayon, and wgpu, not rolling our own stuff. So again, there is nothing at all special about Veloren's design or focus here that makes it more amenable to parallelization than any other game would be, despite it being in a genre that is supposedly difficult to make scale. And that's the thing I'm specifically trying to push back on--the idea that multicore scaling for games is only possible if you have some dedicated cabal of programming wizards who want to push the state of the art. We live in the age of libraries, and wizards are only needed deep in the guts of the implementations of those libraries (just as they always have been, and probably will be until the end of time). A game programmer does not need to understand how a lockfree work stealing queue is implemented (or even what it is!) in order to use "parallel for" to beat the snot out of a carefully optimized single-threaded version of the same task, and it's usually far easier to do the former than the latter. I certainly understand why for a game like Minecraft, with a lot of legacy mechanics and mods that were never designed to be threadsafe, or a game engine like Unity, Unreal, or Roblox, that similarly have lots of plugins and customer code they would like to keep working, it would be very challenging to parallelize after the fact. And naturally, there are limits to what you can do on a single system, and your game design options become far more restricted once you're talking about 10k rather than 1k concurrent players. But for a brand new game without any legacy baggage, there's really no reason why it should scale poorly on multicore systems.