6 ms·

Is there a specific reason why Minecraft hasn’t been “adapted” from the original game to a robust design that could scale to larger worlds and player population

by Fiahil 5y ago

Is there a specific reason why Minecraft hasn’t been “adapted” from the original game to a robust design that could scale to larger worlds and player populations before your project ?

forrestthewoods 5y ago
Pretty much no video game engine scales with cores. There are too many dependent systems. There's no game engine that I'm aware of that effectively uses more than a couple of cores for gameplay simulation. Distributed physics is a well understood problem with no "solution". Just different trade-offs. This solution is cool, but it's not novel or anything. Minecraft is particularly challenging because the entire world is highly mutable. There are no general solutions to any of this. Just a bunch of custom one-offs. SpatialOS is trying. My opinion on it is extremely, extremely negative. And I'll leave it at that. Unity and Unreal both have primarily single-threaded gameplay. Writing multi-threaded gameplay code is extraordinarily difficult. Unity has been working on their DOTS/ECS system for years, but it does not appear to be close to ready for the mainstream. So I think the short answer is "it's really really hard. Like, radically harder than you are imagining. Even if you think it's hard. The value of that work is likely not worth the cost. If most players WANT to play on small servers with close friends then the mountain range of work to effectively support a 1000 player server is not worth it".
- 8note 5y ago
  Couldn't Minecraft chunks run fairly independently, with some message passing when something needs to move between chunks?
  - kevingadd 5y ago
    Yes*, but players build complex machinery fairly often that can span chunk boundaries, so that requires synchronous coordination between the chunks for those machines to work right.
    - devoutsalsa 5y ago
      When will players learn to parallelize their simulated DOGE coin miners?!
  - Uyuxo 5y ago
    Did you read the article
  - Kiro 5y ago
    Imagine two players fighting on a seam between chunks.
- Jweb_Guru 5y ago
  Video games can scale just fine with many cores, including MMOs and games like Minecraft. Veloren, for example, uses multiple cores everywhere for just about everything--most of the gameplay loop, physics, chunk meshing, world generation, realtime simulation, rendering (can be parallelized a lot further now that we've switched to wgpu), networking, and background tasks like snapshotting persistence--and it is certainly not hard to find places where parallelization yields real speedup (quite the opposite--every time we further parallelize physics it yields significant performance improvements!). There are many hard problems in video game performance, but "games just can't make good use of multiple cores" is not one of them. If anything, I often wish we could rely on many more cores being available than actually are! It is certainly far far easier to see real performance wins with multicore than by using multiple servers, which introduce very heavy coordination costs.
  - forrestthewoods 5y ago
    > There are many hard problems in video game performance, but "games just can't make good use of multiple cores" is not one of them. That's not quite what I said. I'm happy you're using specs to work on an open source game. Specs and bevy and all the ECS work being done is super exciting and fun. I <3 Rust. In the meantime there are no major games that effectively scale to, let's say, 64 cores. I don't know of anything shipped that can saturate a 12-core/24-thread Ryzen. ECS alone will not get us to that level of scaling. Yes it's trivial to to throw audio, networking, and a few other subsystems onto separate threads. Modern games definitely leverage 4 cores. Although several of those cores will be severely under utilized. Modern ECS designs are rapidly evolving and rapidly improving our ability to better leverage multiple cores. But we're not yet to a point where games can easily and efficiently saturate 10+ cores. Personally I'd love to see a game like Eve Online that can effectively simulate a universe with tens of thousands of players either spread across the universe or all in one places during one giant battle. > It is certainly far far easier to see real performance wins with multicore than by using multiple servers, which introduce very heavy coordination costs. This is extremely true.
    - Jweb_Guru 5y ago
      If you're just using ECS to parallelize disjoint subsystems then no, it won't get you there. But if you're using it (as Veloren is, and as I hope more people do) in a more deliberate way, to further parallelize within a system, you can indeed scale quite well to large numbers of cores. I've done some theoretical bottleneck calculations and we will still have tons of work to do with 64 cores available, if the game is written properly. We can already get decent utilization out of 32 threads at busy times, and the server was not really close to peak load in terms of player count (and we are very far from done optimizing): https://media.discordapp.net/attachments/539518074106413056/825375910969737287/unknown.png https://media.discordapp.net/attachments/539518074106413056/.... That's just for our server. Our clients can make use of cores in even more ways, although they have less work to do and generally have fewer cores available, and you can see Veloren taking advantage of 16 client threads with similar utilization here: https://twitter.com/sahajsarup/status/1431837669391142916 https://twitter.com/sahajsarup/status/1431837669391142916. The most important thing I want to note is that in both cases, you are not seeing tremendous imbalance between the cores most of the time. While there are definitely single-threaded bottlenecks in games, you have to be working pretty hard before they start bottlenecking the workload! Instead, we are just suffering from a combination of general inefficiency and lack of work to do. So no, I'm going to push back against this notion that multicore scaling for games is some sort of crazy intractable problem. It's not. Like any other kind of parallel scaling, it's trivial in some places, more challenging in others, and depends a lot on your workload (including you actually having enough work to saturate the cores in the first place!). But there's nothing special about games here.
- Animats 5y ago
  There are no general solutions to any of this. Just a bunch of custom one-offs. SpatialOS is trying. My opinion on it is extremely, extremely negative. And I'll leave it at that. I'd like to hear more about that. nagle at animats.com, if you don't want to say much in public. This ties into the "metaverse" business. Lots of metaverse talk about big seamless worlds full of user-created content, but not much is running. So far, nothing with user created content really scales. There are lots of little shared worlds, like Breakroom/Sinespace, Facebook Horizon, IMVU, etc. There are big-space voxel worlds, such as Dual Universe and Roblox. There are general purpose region-oriented big "seamless" worlds such as Second Life, which have trouble at the seams and can't handle crowds in one place, the same problem these Minecraft improvers hit in round 1. Spatial OS was going to fix all this. Their system is basically objects which can be accessed remotely and which migrate to where the most accesses are coming from. It cost over $100 million to develop, they had to do the hosting, and the first four games all went broke due to the high cost of hosting. So, since they had too much venture capital, they set up an in-house game studio and created Scavengers, which is reportedly a so-so shooter. Roblox has plans to solve this, somehow, by sheer money power. When you have a few billion dollars to spare, that might work. If we're headed for the "metaverse", this has to be cracked. Somehow. (I've been writing a multi-threaded Rust/Vulkan client for Second Life / Open Simulator so I'm painfully aware of these problems.)
  - evgen 5y ago
    Forgive the late follow-up, but my first startup in the valley did this back in the mid-90s (Electric Communities, everal EC founders and key hires are mentioned in the Snow Crash acknowledgements) and it is amusing to see people continue to make the same mistakes and step on the same landmines every few years. EC created an entire language (E) to try to hide the complexity and remain secure, had a PERT chart where an actual color on the chart (purple I think) was used to signify that this task would be a good PhD thesis for someone, and really only managed to get a feel for just how large this elephant was before eventually pivoting in a direction that had the possibility to lead to profit in the pre-dotCom era. At best, I think they managed to stumble in the right direction for close to a decade by collecting a large group of really smart people and somehow keeping them happy and engaged for several years and through the ability of the president of the company to get Hollywood people to keep throwing money at them year after year. If you want an interesting read for the evening you should find a copy of Chip Morningstar's 'Cyberspace Protocols Requirements' paper; it holds up incredibly well for something written almost 25 years ago when most people were still on dial-up and we thought 64MB of RAM and a Pentium Pro was too much to require on the client side...
- imtringued 5y ago
  I agree it's hard. It's only worth doing if your business depends on it.
Qi_ 5y ago
The development priorities of Minecraft are a little strange. They have had a lot of opportunities to increase extensibility and performance (see modding API, cubic chunks, etc) but were too slow about it or focused on other things. AFAIK the long-term vision is to move the game entirely to the C++ clients which are much more performant, but a lot of the core playerbase is still on Java for modding and platform compatibility.
- sbierwagen 5y ago
  That would be fine, but there's substantial UI, graphical and gameplay (!) differences between the C++ and Java clients. Crafting recipes are different, redstone has very different connection rules... If the plan is to gently ease Java users to Bedrock, then I would have thought the first step would be to actually port the game, rather than rewrite it with many small pointless changes.
  - rcxdude 5y ago
    Yeah, it's explicitly not a goal to make the two versions the same (despite being incredibly similar), and bedrock is (un)affectionately known as 'bugrock' in the community due to its tendency to glitch out even more so than the java edition, often in extremely frustrating ways like randomly applying fatal fall damage. AFAICT there's not a huge performance win from bedrock anyway (especially compared to the various performance mods for java).
- zimpenfish 5y ago
  > The development priorities of Minecraft are a little strange [...] focused on other things Cynical to say that Microsoft are focused more on milking as much out of the brand and merchandising as possible rather than actually improving the core gameplay in meaningful ways. "Can we have more performance?" "I can do you a glow squid?" "What about shaders and whatnot?" "Axolotl?"

by Fiahil 5y ago

Is there a specific reason why Minecraft hasn’t been “adapted” from the original game to a robust design that could scale to larger worlds and player populations before your project ?

forrestthewoods 5y ago

Pretty much no video game engine scales with cores. There are too many dependent systems. There's no game engine that I'm aware of that effectively uses more than a couple of cores for gameplay simulation. Distributed physics is a well understood problem with no "solution". Just different trade-offs. This solution is cool, but it's not novel or anything. Minecraft is particularly challenging because the entire world is highly mutable. There are no general solutions to any of this. Just a bunch of custom one-offs. SpatialOS is trying. My opinion on it is extremely, extremely negative. And I'll leave it at that. Unity and Unreal both have primarily single-threaded gameplay. Writing multi-threaded gameplay code is extraordinarily difficult. Unity has been working on their DOTS/ECS system for years, but it does not appear to be close to ready for the mainstream. So I think the short answer is "it's really really hard. Like, radically harder than you are imagining. Even if you think it's hard. The value of that work is likely not worth the cost. If most players WANT to play on small servers with close friends then the mountain range of work to effectively support a 1000 player server is not worth it".

8note 5y ago
Couldn't Minecraft chunks run fairly independently, with some message passing when something needs to move between chunks?
- kevingadd 5y ago
  Yes*, but players build complex machinery fairly often that can span chunk boundaries, so that requires synchronous coordination between the chunks for those machines to work right.
  - devoutsalsa 5y ago
    When will players learn to parallelize their simulated DOGE coin miners?!
- Uyuxo 5y ago
  Did you read the article
- Kiro 5y ago
  Imagine two players fighting on a seam between chunks.
Jweb_Guru 5y ago
Video games can scale just fine with many cores, including MMOs and games like Minecraft. Veloren, for example, uses multiple cores everywhere for just about everything--most of the gameplay loop, physics, chunk meshing, world generation, realtime simulation, rendering (can be parallelized a lot further now that we've switched to wgpu), networking, and background tasks like snapshotting persistence--and it is certainly not hard to find places where parallelization yields real speedup (quite the opposite--every time we further parallelize physics it yields significant performance improvements!). There are many hard problems in video game performance, but "games just can't make good use of multiple cores" is not one of them. If anything, I often wish we could rely on many more cores being available than actually are! It is certainly far far easier to see real performance wins with multicore than by using multiple servers, which introduce very heavy coordination costs.
- forrestthewoods 5y ago
  > There are many hard problems in video game performance, but "games just can't make good use of multiple cores" is not one of them. That's not quite what I said. I'm happy you're using specs to work on an open source game. Specs and bevy and all the ECS work being done is super exciting and fun. I <3 Rust. In the meantime there are no major games that effectively scale to, let's say, 64 cores. I don't know of anything shipped that can saturate a 12-core/24-thread Ryzen. ECS alone will not get us to that level of scaling. Yes it's trivial to to throw audio, networking, and a few other subsystems onto separate threads. Modern games definitely leverage 4 cores. Although several of those cores will be severely under utilized. Modern ECS designs are rapidly evolving and rapidly improving our ability to better leverage multiple cores. But we're not yet to a point where games can easily and efficiently saturate 10+ cores. Personally I'd love to see a game like Eve Online that can effectively simulate a universe with tens of thousands of players either spread across the universe or all in one places during one giant battle. > It is certainly far far easier to see real performance wins with multicore than by using multiple servers, which introduce very heavy coordination costs. This is extremely true.
  - Jweb_Guru 5y ago
    If you're just using ECS to parallelize disjoint subsystems then no, it won't get you there. But if you're using it (as Veloren is, and as I hope more people do) in a more deliberate way, to further parallelize within a system, you can indeed scale quite well to large numbers of cores. I've done some theoretical bottleneck calculations and we will still have tons of work to do with 64 cores available, if the game is written properly. We can already get decent utilization out of 32 threads at busy times, and the server was not really close to peak load in terms of player count (and we are very far from done optimizing): https://media.discordapp.net/attachments/539518074106413056/825375910969737287/unknown.png https://media.discordapp.net/attachments/539518074106413056/.... That's just for our server. Our clients can make use of cores in even more ways, although they have less work to do and generally have fewer cores available, and you can see Veloren taking advantage of 16 client threads with similar utilization here: https://twitter.com/sahajsarup/status/1431837669391142916 https://twitter.com/sahajsarup/status/1431837669391142916. The most important thing I want to note is that in both cases, you are not seeing tremendous imbalance between the cores most of the time. While there are definitely single-threaded bottlenecks in games, you have to be working pretty hard before they start bottlenecking the workload! Instead, we are just suffering from a combination of general inefficiency and lack of work to do. So no, I'm going to push back against this notion that multicore scaling for games is some sort of crazy intractable problem. It's not. Like any other kind of parallel scaling, it's trivial in some places, more challenging in others, and depends a lot on your workload (including you actually having enough work to saturate the cores in the first place!). But there's nothing special about games here.
Animats 5y ago
There are no general solutions to any of this. Just a bunch of custom one-offs. SpatialOS is trying. My opinion on it is extremely, extremely negative. And I'll leave it at that. I'd like to hear more about that. nagle at animats.com, if you don't want to say much in public. This ties into the "metaverse" business. Lots of metaverse talk about big seamless worlds full of user-created content, but not much is running. So far, nothing with user created content really scales. There are lots of little shared worlds, like Breakroom/Sinespace, Facebook Horizon, IMVU, etc. There are big-space voxel worlds, such as Dual Universe and Roblox. There are general purpose region-oriented big "seamless" worlds such as Second Life, which have trouble at the seams and can't handle crowds in one place, the same problem these Minecraft improvers hit in round 1. Spatial OS was going to fix all this. Their system is basically objects which can be accessed remotely and which migrate to where the most accesses are coming from. It cost over $100 million to develop, they had to do the hosting, and the first four games all went broke due to the high cost of hosting. So, since they had too much venture capital, they set up an in-house game studio and created Scavengers, which is reportedly a so-so shooter. Roblox has plans to solve this, somehow, by sheer money power. When you have a few billion dollars to spare, that might work. If we're headed for the "metaverse", this has to be cracked. Somehow. (I've been writing a multi-threaded Rust/Vulkan client for Second Life / Open Simulator so I'm painfully aware of these problems.)
- evgen 5y ago
  Forgive the late follow-up, but my first startup in the valley did this back in the mid-90s (Electric Communities, everal EC founders and key hires are mentioned in the Snow Crash acknowledgements) and it is amusing to see people continue to make the same mistakes and step on the same landmines every few years. EC created an entire language (E) to try to hide the complexity and remain secure, had a PERT chart where an actual color on the chart (purple I think) was used to signify that this task would be a good PhD thesis for someone, and really only managed to get a feel for just how large this elephant was before eventually pivoting in a direction that had the possibility to lead to profit in the pre-dotCom era. At best, I think they managed to stumble in the right direction for close to a decade by collecting a large group of really smart people and somehow keeping them happy and engaged for several years and through the ability of the president of the company to get Hollywood people to keep throwing money at them year after year. If you want an interesting read for the evening you should find a copy of Chip Morningstar's 'Cyberspace Protocols Requirements' paper; it holds up incredibly well for something written almost 25 years ago when most people were still on dial-up and we thought 64MB of RAM and a Pentium Pro was too much to require on the client side...
imtringued 5y ago
I agree it's hard. It's only worth doing if your business depends on it.

Qi_ 5y ago

The development priorities of Minecraft are a little strange. They have had a lot of opportunities to increase extensibility and performance (see modding API, cubic chunks, etc) but were too slow about it or focused on other things. AFAIK the long-term vision is to move the game entirely to the C++ clients which are much more performant, but a lot of the core playerbase is still on Java for modding and platform compatibility.

sbierwagen 5y ago
That would be fine, but there's substantial UI, graphical and gameplay (!) differences between the C++ and Java clients. Crafting recipes are different, redstone has very different connection rules... If the plan is to gently ease Java users to Bedrock, then I would have thought the first step would be to actually port the game, rather than rewrite it with many small pointless changes.
- rcxdude 5y ago
  Yeah, it's explicitly not a goal to make the two versions the same (despite being incredibly similar), and bedrock is (un)affectionately known as 'bugrock' in the community due to its tendency to glitch out even more so than the java edition, often in extremely frustrating ways like randomly applying fatal fall damage. AFAICT there's not a huge performance win from bedrock anyway (especially compared to the various performance mods for java).
zimpenfish 5y ago
> The development priorities of Minecraft are a little strange [...] focused on other things Cynical to say that Microsoft are focused more on milking as much out of the brand and merchandising as possible rather than actually improving the core gameplay in meaningful ways. "Can we have more performance?" "I can do you a glow squid?" "What about shaders and whatnot?" "Axolotl?"