16 ms·

Testing a 1,000 player Minecraft server with Folia

222 points by creatonez 3y ago

AgentK20 3y ago
Always fun to continue to see innovation within the Minecraft Java space. At Hypixel we ran (and I believe they still do run) a custom fork of Spigot from 2014-ish, with features from each sequential update to Minecraft being added to our fork via our Spigot fork. This let us diverge greatly and customize the Minecraft protocol to our own needs, saving hugely on internal bandwidth and letting us optimize crap out of the L7 "BungeeCord" reverse-proxy we used.
- nabogh 3y ago
  That sounds cool as hell. I was always impressed when I jumped on hypixel just how much custom server work must've gone into it.
- lyu07282 3y ago
  > Always fun to continue to see innovation within the Minecraft Java space. Just don't expect that innovation to come out of Mojang/Minecraft
  - LeeroyWasHere 3y ago
    Let's all remember that Notch didn't want any multiplayer and purposefully did everything he could to make it painful. Minecraft would be dead without the community devs and artists.
    - lyu07282 3y ago
      That can't be true, minecraft already had multiplayer in 2009. He didn't make it intentionally painful and never said anything like that. I remember there being huge issues with multiplayer but that was mostly because of bad architecture, not intentional.
      - creatonez 3y ago
        Correct, I don't think he hesitated even a little bit on adding multiplayer support. The very first announcement of Minecraft's existence mentioned two planned game modes (fortress and team survival) that would require multiplayer and that all gamemodes would support it https://forums.tigsource.com/index.php?topic=6273.0 https://forums.tigsource.com/index.php?topic=6273.0
        lyu07282 3y ago
        yep, just to add to that I remember there was a big deal to rearchitect the entire engine to use a client-server architecture even when running singleplayer. This also enabled the "Open to LAN" feature in which you can start a server from your singleplayer world with a single click. https://web.archive.org/web/20120713082903/http://www.mojang.com/2012/07/minecraft-1-3-info-snapshot-w27/ https://web.archive.org/web/20120713082903/http://www.mojang...
    - hu3 3y ago
      > Let's all remember that Notch didn't want any multiplayer and purposefully did everything he could to make it painful. That's a big claim. Got a source?
- mooman219 3y ago
  Oh god that's a throwback. I remember hanging out in the bukket IRC before all the legal debacles. I worked on MCSG and Wynncraft. It honestly was a blast slamming together patches on spigot(craftbukkit(nms))) onion; I remember writing a patch to run entities off main-thread among other things. I hope you're doing well and I hope Hytale is doing alright! I feel like everyone I intereacted with back then ended up doing well in tech in some form.
tehsauce 3y ago
Very excited to see how far minecraft servers can be pushed! If anyone is curious why you’d want to have 1000 players in minecraft, this video documents a pretty amazing example scenario: https://youtu.be/zv-TS_mEHE4 https://youtu.be/zv-TS_mEHE4
- 7373737373 3y ago
  According to the making of video of this, they are using Multipaper: https://github.com/MultiPaper/MultiPaper https://github.com/MultiPaper/MultiPaper
- 93po 3y ago
  I wonder if we’re going to look back on how YouTubers talk in the same way we look back at stereotypical forced accents on corporate news channels. It feels so unnatural
rasz 3y ago
>Players were spread into 49 different teams across the map. Each team consisted of around 20 players. How is that different from sharding into mini 20 player servers? I was hoping for 1000 players all in visual range :) Eve Online record is >6000 players all in visual range, but they cheat by lowering server tick from 1Hz down to 0.1Hz https://wiki.eveuniversity.org/Time_dilation https://wiki.eveuniversity.org/Time_dilation
- creatonez 3y ago
  As I understand, Folia can't help you if all your players are bunched together, but unlike having separate servers connected by Velocity or Bungee, it is a seamless experience -- everyone is in the same world and can visit anyone else without having to use commands, go through portals, or even wait for a loading screen. The "bubbles" that make up the world merge and split seamlessly, with special attention paid to avoid breaking vanilla game mechanics. [Speculation, I don't have the data and wasn't involved in the test] The test placed each 'team' 10240 blocks apart, which is close enough for a ~30 minute walk to get to another team (~4 minutes if you take the nether), so I assume a lot of bubble merging/splitting happened. It's also far enough away that, if the regionizing logic allows for it, you're split off onto another thread relatively early into your journey to visit another 'team' spawnpoint. This is a relatively realistic scenario for an SMP, where towns are usually 2,000 to 20,000 blocks apart.
  - skrebbel 3y ago
    Tbf “a ~30 minute walk to get to another team (~4 minutes if you take the nether)” doesnt seem very seamless to me. That’s quite a commute!
    - creatonez 3y ago
      Hah. On 2b2t, an "anarchy" server where any player can blow up any build and type whatever garbage into chat they want, players often opt to travel out tens of millions of blocks in an effort to make their builds extremely difficult to find. Can take many hours of nether travel to get to where you want to go. And inevitably, people still find them, usually through advanced techniques -- hacks that tell you whether a chunk is newly generated or part of an existing 'chunk trail' can be used to hunt down players that have taken great effort to hide their location. Anyways, a distance of 10240 is just how this particular test was set up. The regionizing is still useful in a much cozier world. As I understand the only way for it to be guaranteed that everyone is inside the same bubble (and therefore the ticking is all on a single thread) is for everyone to be in the same 768 block radius, or for there to be a line of players spaced this distance apart. It's rather atypical for a server to organically develop this way, since people really like to explore for hours before settling in the perfect spot. But some heavily planned/curated server are like this.
dwohnitmok 3y ago
Has anyone made a comparison of Minecraft server performance on ZGC vs Shenandoah vs G1GC on a recent JDK version? I'm curious whether the amazing GC pause numbers hyped by the latest versions of ZGC hold up (max pause times of 500 microseconds and averages of 50 microseconds). Also what's going on with the generational Shenandoah GC spikes in that graph? There's some pretty crazy spikes that completely dominate the pause numbers and make it pretty hard to read the graph. It looks like GC pauses kept increasing up until the server crashed?
- creatonez 3y ago
  I can't attest to how valid these benchmarks are, but this explores the topic and is referenced a lot in the community - https://github.com/brucethemoose/Minecraft-Performance-Flags-Benchmarks https://github.com/brucethemoose/Minecraft-Performance-Flags...
cjensen 3y ago
Worth mentioning that this requires mods which break many in-game devices. The official in-game behavior was not able to be scaled. That's an interesting and legitimate strategy to the scaling problem, but good to know the downsides with the upsides when trying to take lessons from this.
- ocelotpotpie 3y ago
  (Disclaimer: I'm on the Paper team, and Paper is the org under which Folia sits.) Paper (and Folia) have differences from "vanilla" (official) Minecraft, which is part of how they improve performance to begin with. So it is true that there are differences from vanilla. Most "in-game devices" - if you mean builds, redstone machines, mob farms, etc - have versions that people have found will work on Paper and Folia. They might require some tweaking but complex redstone and mob farms are definitely very doable. The goal is to have a close-to-vanilla experience while fixing bugs, patching exploits, and improving performance to run more people on a given piece of server hardware. The biggest issue with Folia right now is that it's very new so there aren't a lot of plugins that support it just yet. That's changing every day! And of course it still has some crashing issues because it's still in development. :)
- creatonez 3y ago
  Do you know any redstone incompatibilities that Folia has that Paper doesn't? From what I've heard, the goal is to have very good redstone compatibility, but I haven't yet heard much about actual testing with complicated machines.
Animats 3y ago
Interesting to see how they scale a multiplayer big world. Not many systems try to do this. Improbable, Minecraft, Second Life, and Roblox are the ones I know about. Almost everybody else shards. Any other examples of really big shared seamless multiplayer worlds?
- fla 3y ago
  EVE Online
  - moritonal 3y ago
    I believe they shard each sector and their inability to sub-divide their hosting on a sector level has created their infamous timewarped battles?
    - Ntrails 3y ago
      Pretty much. Once you got to 1500+ fleet fights it would originally just drop lots and lots of player actions on the floor as could not cope. Time Dilation came in to run at 1/10 game/real time ratio, allowing fights to scale better and be a less frustrating experience for participants
    - emil0r 3y ago
      Yes, so in that sense it's sharded. In another sense it's all one server, as you can warp between different systems and meet all the players there in the game. Everything is run on one giant server. Not sure how it is now, but back in 2014-2016 you could inform them of big battles ahead of time for a specific system, at which point they would, in their own words, reinforce the node (moving that particular system to its own allocation of resources). This often was the difference between being able to duke it out in an epic space battle or wait for the system to load while everyone lagged to death.
    - Cthulhu_ 3y ago
      Well, at the smallest level (if I recall correctly) it's a single solar system; they have a few beefy servers on stand-by, one is always in use for Jita, the main and most active trading hub, and others can be spun up and a whole solar system transferred over if it gets busy. And as someone else pointed out, you can organize a large fight in advance so they can transfer it over. But there's the bottleneck, because they can only do the calculations of ship movement & actions on a single node. I'm sure it's been optimized to no end as well. IIRC it's written in Python, but that's not going to be the main performance bottleneck. I have only the smallest of clues about distributed systems, the only way they could scale it up is to somehow make it so they can run a single solar system or cluster of ships on multiple servers, but for that you get the overhead of inter-server communication or you need an asynchronous, eventually-consistent game instead of something realtime.
stevefan1999 3y ago
What about ZGC?
- tehbeard 3y ago
  Somewhat orthogonal to what Folia is doing, and already explored by the community. Folia's main trick is to dynamically split the "main game tick loop", where logic and actions are processed into several threads based on location. Such that processing for say a mob farm in the South of the map does not impede on processing for a hopper based storage system in the North West etc. This doesn't solve the everyone's at one place problem (or "Jita problem", in EvE online parlance) but does solve some issues with populated servers where player density isn't extreme.
  - birracerveza 3y ago
    Is there a solution for the Jita problem that isn't just beefing up the server?
    - lordnacho 3y ago
      From reading the comments here, it would appear the problem is sending out position updates to all the clients. If each client is on a TCP connection, this will end up being a problem as each movement causes everyone else within range to receive a message. If everyone is moving, that's a heck of a lot of messages everyone is getting. Not only that, everyone needs a copy of the message in their TCP pipe, one at a time, sent from the server. So it quickly explodes. I guess you can think of it as O(N) clients each causing O(N) messages to go out, so it explodes quadratically. This isn't likely what I'm about to suggest, but you could move to a broadcast type system where you have some sort of UDP broadcast where you simply tell everyone about all the movements, and a alternative negative-ack channel where clients can ask for missed broadcasts. That way only O(N) messages are sent (the magic happens in the network devices). The issue with that is the internet isn't great for packet loss or multicast routing, so if you have O(N) resend requests you're not much better off. If you had all your clients in one datacenter it might work :)
  - 3y ago
Nican 3y ago
It would be interesting to have blog posts about the optimizations that were made.
- creatonez 3y ago
  If you want to read more, some technical details that were previously only covered by scattered blog posts and github gists have been consolidated into the project's documentation - https://docs.papermc.io/folia/reference/overview https://docs.papermc.io/folia/reference/overview
d--b 3y ago
Honest question: is that a lot?
- ocelotpotpie 3y ago
  Most servers with the latest CPUs can only handle a few hundred players without bogging down, so yeah it's a fair bit.
- zimpenfish 3y ago
  My Beelink Ryzen 7 sometimes chugs with 3 people on it. Java Minecraft is terrible at actually doing stuff in an efficient manner.
lrem 3y ago
I'll just drop this factoid: Mineplex has won the Guinness World Records award on January 28, 2015 for having 34,434 concurrent players, the most on a Minecraft server at the time.
- ocelotpotpie 3y ago
  This is where things start to get a little fuzzy. Mineplex had the most concurrent players but they weren't on a single individual bare metal server or single server jar instance. A lot of large "servers" like Mineplex, Hypixel, etc run a proxy which sits in front of a bunch of other servers. The concept of a "server" can have many meanings in Minecraft so it gets fuzzy quickly. I'm not sure if this 1,000 person test is a record but I'm not aware of any other test running so many concurrent players on a single dedicated box with a single instance of Minecraft running. Folia takes advantage of more CPU threads so everyone is in sitting in the same server instance. Hypixel beat out Mineplex later on, BTW.
  - lrem 3y ago
    Ha, nice, thanks. I've never looked into this deeper than accepting the fact that it's doable. Now, if only I had the spare time to look into how did they shard the servers...
    - ocelotpotpie 3y ago
      We tend to call those "networks", when they have a bunch of different gametypes and servers interconnected. At a smaller scale of a few hundred players it's pretty easily handled by off the shelf software like Velocity for the proxy and a few server instances running Paper. You generally want good performing (not VPS or "cloud" instances) but you can run a bunch of different worlds and have people with cross server talk, the ability to warp between worlds, etc. Most of the larger networks and nearly all of the smaller networks use a variation of this kind of model.
    - AgentK20 3y ago
      ocelotpotpie's sibling reply to this has good explanations for the terminology. If you're curious to actually see the software: * https://www.spigotmc.org/wiki/bungeecord/ https://www.spigotmc.org/wiki/bungeecord/ - Older, less-performant but still used by some teams * https://papermc.io/software/velocity https://papermc.io/software/velocity - Newer, more performant, maintained by the team that makes Paper, one of the leading performance MC server implementations. For context of scale btw, when Hypixel had 200k+ players online at peak we had something like 2,000+ 1U E3-1271v3's each with 32GB RAM, all colo'd in a single DC in Chicago. Egress is 70-80gbps or so 95th percentile, with most months (at high peak) egressing 10PB/mo+ of real-time, uncacheable data.
samsquire 3y ago
This is interesting. I am interested in multithreading and parallelism. So I have a journal entry to explore which is about deliberately desynchronizing and resynchronizing game loops for performance. * For the number of clients can use epoll or liburing. * Can multiplex multiple sockets per thread my epoll-server does this. * Can split out recv and send across threads so you can send while receiving and receive while sending * For the game loops can divide the territory covered to a different game loop. My question becomes how do you synchronize game loops across threads, you could latch on each thread for partial causality between game loops.
- pjc50 3y ago
  > deliberately desynchronizing and resynchronizing game loops for performance. > latch on each thread for partial causality between game loops. This sounds like a recipe for misery. You have to start thinking about ""light cones"" if causality propagates at finite speed. Do all observers in the game universe witness events in the same order? What if they don't? You get a little bit of this with rollback netcode, on the level of a few frames, but in that case there's always an authoritative causality and the client's guess is subordinate to it. It also sounds like fertile ground for exploits, both of the duplicating-object type and the glitch-through-geometry type.
  - samsquire 3y ago
    Thank you for your reply. Your comment made me amused because it made me think of a game world where retrocausality was in effect, it would be absurd. Is the bottleneck for networked games the network of broadcasting updates? Or the CPU usage of updating object states? Sharding or instancing means you have fewer clients to broadcast stream updates to. Different game loops interacting with eachother, that's interesting. How else do you scale a game engine across threads?
    - pjc50 3y ago
      > Is the bottleneck for networked games the network of broadcasting updates? Or the CPU usage of updating object states? Bit of both depending on precisely what's happening. Ultimately the problem is that, for N "agents" (players or NPCs or active blocks or monsters, etc) in a space, if you're not careful you end up with O(N ^ 2) checks of the form "has X collided/interacted with Y". Octree systems can spread this out, but you're still vulnerable to "what if all the players go to the same spot?" Instancing lets you back off the worst-case situation by limiting the maximum number of players in a particular spot.
viraptor 3y ago
I know this isn't the point of Folia's test, but: > During the period when 1,000 players were online, we reached a maximum of ~7.9GB/s heap allocation and our GC was hovering around 2-3GB/s when averaged over a minute. As someone who dealt with soft-realtime telephony stuff, this makes me want to scream in horror. It seems like the platform really hurts the performance here. In an average second with that many players, most (all?) of them will not do any action apart from changing their position. (Just moving around would ideally only use preallocated per-player space only; extra in-game events, npc AI, etc. could still use some allocs) It's great Folia can get more out of the bad situation, but... oofff. (not a criticism of Minecraft by the way, it's better to be popular and inefficient than not exist - I just didn't realise quite how much overhead there is due to the runtime)
- madduci 3y ago
  Has anyone already tried to use GraalVM with it? Cool would be trying to do a native compilation and compare the performance
  - tuetuopay 3y ago
    That's a sample size of one, but I did try it once and it ended up being basically the same. HotSpot took a measurable amount of time to reach GraalVM performance (in the minute if memory serves), but once the hot path got JITed, they were basically on par. Since a minecraft server is (usually) a long-lived process, it does not matter in the end.
  - polski-g 3y ago
    I've tried different JVMs, different garbage collectors, etc, etc. They all make negligible difference.
- jandrese 3y ago
  The Minecraft Java version is notorious for thrashing memory. It allocates and discards objects at an incredible rate, even when the player isn't doing anything. The server has a GUI where you can watch the memory allocation with its distinctive sawtooth pattern.
blindriver 3y ago
On a side note, I've been trying to figure out how to write Java mods for server-side vanilla Minecraft server, but there is basically no documentation at all. Is it safe to say that this isn't possible? Which flavor of minecraft server is the best documented to write mods for, Spigot?
- whatsthatabout 3y ago
  Its paper nowadays :) https://papermc.io https://papermc.io
- ocelotpotpie 3y ago
  There's a bit of a distinction in the Minecraft world between "mods" and "plugins". Even though to the outside perspective they're the same thing, the terms are used for two different communities/ecosystems, basically. "Plugins" are for bukkit-based compatible servers, like Spigot and Paper (and Folia, with some modifications). "Mods" are more for forge, sponge, etc servers. So you can kinda pick which direction you want to go. But the Paper project has a ton of API documentation and a few Github repos to even get you started writing your own plugins. The Discord server has a dev help channel with people happy to answer any questions. The Sponge team has a similar setup as well if that's the route you'd like to go.
  - bredren 3y ago
    Are any of these choices restricted in their existing or likely future compatibility with where Microsoft is taking the game? My understanding is server admins will spend money on mods / plugins. Are there any server types more likely to garner paid license or plugins?
    - ocelotpotpie 3y ago
      Microsoft has struct rules about the stuff you can sell for the server, and servers need to be EULA compliant. If they aren't they can get blacklisted or shutdown. People do buy plugins but the vast majority of the plugins you'd want to use are free open source plugins. A lot of the paid/closed source stuff is kinda crappy or misleading anyway. You can run a great server using free plugins and software. It might not be a money maker, but if your goal is to have people play and have fun and not buy stuff it's super easy and doable. People do still donate to projects to support the authors who make the wheels spin.
bullen 3y ago
https://www.twitch.tv/videos/1849538310 https://www.twitch.tv/videos/1849538310 No comment.
- ocelotpotpie 3y ago
  (Disclaimer: I'm on the Paper team, and Paper is the org under which Folia sits) I didn't watch the stream because I was offline when it happened, but I just want to note that the test was run by someone not on the Paper team who got tubbo to stream it, which is how they got so many players. It can be tricky to get enough players for such a large test, so it made sense for cubxity to pair up with someone to get more players. Twitch can get pretty funky, especially twitch chat, so hopefully there's nothing "bad" in that video, though.
haezee 3y ago
While very impressive, something to note: >The server was prepared with a 100k x 100k block pre-generated world. Our custom plugin distributed new players to the least-occupied region. Minecraft chunk generation is notoriously though, and even a few players generating new chunks will bring the TPS down to an unplayable state very fast.
- skerit 3y ago
  > Minecraft chunk generation is notoriously though, and even a few players generating new chunks will bring the TPS down to an unplayable state very fast. And that's knowing a big part of the chunk generation already happens off the main thread.
- ocelotpotpie 3y ago
  (Disclaimer: I'm on the Paper team, and Paper is the org under which Folia sits.) This is true to an extent, yes. Chunk generation can be real tough. That said, chunk generation has been re-written in Paper, so it's substantially faster than vanilla Minecraft. When we did a large scale real player test on a much earlier build of Folia we had ~327 players at peak and we did not generate chunks because we specifically wanted to see how it ran. We didn't max out at ~327, we just didn't have more joins than that. The project improved a bunch already so that number without chunk generation is very doable to beat if you didn't pre-gen any chunks. Generally it's free performance to pre-gen your world though. Highly recommended for regular players.
- circuit10 3y ago
  "generating new chunks" That's not happeneing, it says "pre-generated world"
  - xboxnolifes 3y ago
    That's the point. The test doesn't include chunk generation.
    - circuit10 3y ago
josefrichter 3y ago
I know nothing about Minecraft servers, but has anyone tried to build one in Elixir/Erlang? Sounds like 1000 players should be a piece of cake.
- zimpenfish 3y ago
  I think the devil is in the details there - getting the terrain generation right, handling of things like redstone, and replicating the various glitches+bugs that people tend to use. Otherwise you've made a Minecraft-alike, not Minecraft.
- nottorp 3y ago
  It may run faster than java but it will have the same problems with close enough players having to be processed together in the same thread.
c7DJTLrn 3y ago
Cool stuff. The possibilities of Minecraft if some optimisation work was done would be amazing. It's demotivating just how rotten the internals of Minecraft are. Bedrock edition has really nice performance but nobody wants to play it.
- tehbeard 3y ago
  Plenty play bedrock. It's just.... a very sanitized and corporate experience. And for Java edition players just different enough in mechanics / ux / feel to annoy them. Very much pushes you to stick to official servers or small friend groups, and is quite aggressive on its monetization.
  - circuit10 3y ago
    Bedrock is also much harder to mod, being written in C++, and mods are a big part of what people like about Minecraft. They have an add-on system but that's just fancy JSON config files so you can only do things like adding a new block, not complete reworks of the internal code to improve performance or anything like that which is totally possible on Java Edition
- aprilnya 3y ago
  Bedrock gives me an uncanny valley feeling… it’s Minecraft, sure, but everything is just slightly off, and it makes my brain really not like it
- Pannoniae 3y ago
  Bedrock rendering performance is really good. Tick performance? Not so much, the game is riddled with horrible issues such as world fallthrough which don't exist on Java (due to floating-point rounding errors), and I think the default tick distance is something like 4 or 6 chunks. Bedrock looks good, but the simulation itself is just broken. Not to mention the uncanny feeling as previous posters said - they've managed to create a similar game but with a sterilised, odd feeling. It's like watching a bad Disney adaptation of a book.
conaclos 3y ago
Just wondering: Is Paper consuming less resources than the vanilla Minecraft server for a few players (1 to 10 players). In other words, is it less costly to operate a Paper server on the cloud?
- doodlesdev 3y ago
  Absolutely. Depending on your hardware using Paper can be the difference of the server not even starting and a playable game. Paper is an absolutely incredible effort.

222 points by creatonez 3y ago

Always fun to continue to see innovation within the Minecraft Java space. At Hypixel we ran (and I believe they still do run) a custom fork of Spigot from 2014-ish, with features from each sequential update to Minecraft being added to our fork via our Spigot fork. This let us diverge greatly and customize the Minecraft protocol to our own needs, saving hugely on internal bandwidth and letting us optimize crap out of the L7 "BungeeCord" reverse-proxy we used.

nabogh 3y ago
That sounds cool as hell. I was always impressed when I jumped on hypixel just how much custom server work must've gone into it.
lyu07282 3y ago
> Always fun to continue to see innovation within the Minecraft Java space. Just don't expect that innovation to come out of Mojang/Minecraft
- LeeroyWasHere 3y ago
  Let's all remember that Notch didn't want any multiplayer and purposefully did everything he could to make it painful. Minecraft would be dead without the community devs and artists.
  - lyu07282 3y ago
    That can't be true, minecraft already had multiplayer in 2009. He didn't make it intentionally painful and never said anything like that. I remember there being huge issues with multiplayer but that was mostly because of bad architecture, not intentional.
    - creatonez 3y ago
      Correct, I don't think he hesitated even a little bit on adding multiplayer support. The very first announcement of Minecraft's existence mentioned two planned game modes (fortress and team survival) that would require multiplayer and that all gamemodes would support it https://forums.tigsource.com/index.php?topic=6273.0 https://forums.tigsource.com/index.php?topic=6273.0
      - lyu07282 3y ago
        yep, just to add to that I remember there was a big deal to rearchitect the entire engine to use a client-server architecture even when running singleplayer. This also enabled the "Open to LAN" feature in which you can start a server from your singleplayer world with a single click. https://web.archive.org/web/20120713082903/http://www.mojang.com/2012/07/minecraft-1-3-info-snapshot-w27/ https://web.archive.org/web/20120713082903/http://www.mojang...
  - hu3 3y ago
    > Let's all remember that Notch didn't want any multiplayer and purposefully did everything he could to make it painful. That's a big claim. Got a source?
mooman219 3y ago
Oh god that's a throwback. I remember hanging out in the bukket IRC before all the legal debacles. I worked on MCSG and Wynncraft. It honestly was a blast slamming together patches on spigot(craftbukkit(nms))) onion; I remember writing a patch to run entities off main-thread among other things. I hope you're doing well and I hope Hytale is doing alright! I feel like everyone I intereacted with back then ended up doing well in tech in some form.

tehsauce 3y ago

Very excited to see how far minecraft servers can be pushed! If anyone is curious why you’d want to have 1000 players in minecraft, this video documents a pretty amazing example scenario: https://youtu.be/zv-TS_mEHE4 https://youtu.be/zv-TS_mEHE4

7373737373 3y ago
According to the making of video of this, they are using Multipaper: https://github.com/MultiPaper/MultiPaper https://github.com/MultiPaper/MultiPaper
93po 3y ago
I wonder if we’re going to look back on how YouTubers talk in the same way we look back at stereotypical forced accents on corporate news channels. It feels so unnatural

rasz 3y ago

>Players were spread into 49 different teams across the map. Each team consisted of around 20 players. How is that different from sharding into mini 20 player servers? I was hoping for 1000 players all in visual range :) Eve Online record is >6000 players all in visual range, but they cheat by lowering server tick from 1Hz down to 0.1Hz https://wiki.eveuniversity.org/Time_dilation https://wiki.eveuniversity.org/Time_dilation

creatonez 3y ago
As I understand, Folia can't help you if all your players are bunched together, but unlike having separate servers connected by Velocity or Bungee, it is a seamless experience -- everyone is in the same world and can visit anyone else without having to use commands, go through portals, or even wait for a loading screen. The "bubbles" that make up the world merge and split seamlessly, with special attention paid to avoid breaking vanilla game mechanics. [Speculation, I don't have the data and wasn't involved in the test] The test placed each 'team' 10240 blocks apart, which is close enough for a ~30 minute walk to get to another team (~4 minutes if you take the nether), so I assume a lot of bubble merging/splitting happened. It's also far enough away that, if the regionizing logic allows for it, you're split off onto another thread relatively early into your journey to visit another 'team' spawnpoint. This is a relatively realistic scenario for an SMP, where towns are usually 2,000 to 20,000 blocks apart.
- skrebbel 3y ago
  Tbf “a ~30 minute walk to get to another team (~4 minutes if you take the nether)” doesnt seem very seamless to me. That’s quite a commute!
  - creatonez 3y ago
    Hah. On 2b2t, an "anarchy" server where any player can blow up any build and type whatever garbage into chat they want, players often opt to travel out tens of millions of blocks in an effort to make their builds extremely difficult to find. Can take many hours of nether travel to get to where you want to go. And inevitably, people still find them, usually through advanced techniques -- hacks that tell you whether a chunk is newly generated or part of an existing 'chunk trail' can be used to hunt down players that have taken great effort to hide their location. Anyways, a distance of 10240 is just how this particular test was set up. The regionizing is still useful in a much cozier world. As I understand the only way for it to be guaranteed that everyone is inside the same bubble (and therefore the ticking is all on a single thread) is for everyone to be in the same 768 block radius, or for there to be a line of players spaced this distance apart. It's rather atypical for a server to organically develop this way, since people really like to explore for hours before settling in the perfect spot. But some heavily planned/curated server are like this.

dwohnitmok 3y ago

Has anyone made a comparison of Minecraft server performance on ZGC vs Shenandoah vs G1GC on a recent JDK version? I'm curious whether the amazing GC pause numbers hyped by the latest versions of ZGC hold up (max pause times of 500 microseconds and averages of 50 microseconds). Also what's going on with the generational Shenandoah GC spikes in that graph? There's some pretty crazy spikes that completely dominate the pause numbers and make it pretty hard to read the graph. It looks like GC pauses kept increasing up until the server crashed?

creatonez 3y ago
I can't attest to how valid these benchmarks are, but this explores the topic and is referenced a lot in the community - https://github.com/brucethemoose/Minecraft-Performance-Flags-Benchmarks https://github.com/brucethemoose/Minecraft-Performance-Flags...

cjensen 3y ago

Worth mentioning that this requires mods which break many in-game devices. The official in-game behavior was not able to be scaled. That's an interesting and legitimate strategy to the scaling problem, but good to know the downsides with the upsides when trying to take lessons from this.

ocelotpotpie 3y ago
(Disclaimer: I'm on the Paper team, and Paper is the org under which Folia sits.) Paper (and Folia) have differences from "vanilla" (official) Minecraft, which is part of how they improve performance to begin with. So it is true that there are differences from vanilla. Most "in-game devices" - if you mean builds, redstone machines, mob farms, etc - have versions that people have found will work on Paper and Folia. They might require some tweaking but complex redstone and mob farms are definitely very doable. The goal is to have a close-to-vanilla experience while fixing bugs, patching exploits, and improving performance to run more people on a given piece of server hardware. The biggest issue with Folia right now is that it's very new so there aren't a lot of plugins that support it just yet. That's changing every day! And of course it still has some crashing issues because it's still in development. :)
creatonez 3y ago
Do you know any redstone incompatibilities that Folia has that Paper doesn't? From what I've heard, the goal is to have very good redstone compatibility, but I haven't yet heard much about actual testing with complicated machines.

Animats 3y ago

Interesting to see how they scale a multiplayer big world. Not many systems try to do this. Improbable, Minecraft, Second Life, and Roblox are the ones I know about. Almost everybody else shards. Any other examples of really big shared seamless multiplayer worlds?

fla 3y ago
EVE Online
- moritonal 3y ago
  I believe they shard each sector and their inability to sub-divide their hosting on a sector level has created their infamous timewarped battles?
  - Ntrails 3y ago
    Pretty much. Once you got to 1500+ fleet fights it would originally just drop lots and lots of player actions on the floor as could not cope. Time Dilation came in to run at 1/10 game/real time ratio, allowing fights to scale better and be a less frustrating experience for participants
  - emil0r 3y ago
    Yes, so in that sense it's sharded. In another sense it's all one server, as you can warp between different systems and meet all the players there in the game. Everything is run on one giant server. Not sure how it is now, but back in 2014-2016 you could inform them of big battles ahead of time for a specific system, at which point they would, in their own words, reinforce the node (moving that particular system to its own allocation of resources). This often was the difference between being able to duke it out in an epic space battle or wait for the system to load while everyone lagged to death.
  - Cthulhu_ 3y ago
    Well, at the smallest level (if I recall correctly) it's a single solar system; they have a few beefy servers on stand-by, one is always in use for Jita, the main and most active trading hub, and others can be spun up and a whole solar system transferred over if it gets busy. And as someone else pointed out, you can organize a large fight in advance so they can transfer it over. But there's the bottleneck, because they can only do the calculations of ship movement & actions on a single node. I'm sure it's been optimized to no end as well. IIRC it's written in Python, but that's not going to be the main performance bottleneck. I have only the smallest of clues about distributed systems, the only way they could scale it up is to somehow make it so they can run a single solar system or cluster of ships on multiple servers, but for that you get the overhead of inter-server communication or you need an asynchronous, eventually-consistent game instead of something realtime.

stevefan1999 3y ago

What about ZGC?

tehbeard 3y ago
Somewhat orthogonal to what Folia is doing, and already explored by the community. Folia's main trick is to dynamically split the "main game tick loop", where logic and actions are processed into several threads based on location. Such that processing for say a mob farm in the South of the map does not impede on processing for a hopper based storage system in the North West etc. This doesn't solve the everyone's at one place problem (or "Jita problem", in EvE online parlance) but does solve some issues with populated servers where player density isn't extreme.
- birracerveza 3y ago
  Is there a solution for the Jita problem that isn't just beefing up the server?
  - lordnacho 3y ago
    From reading the comments here, it would appear the problem is sending out position updates to all the clients. If each client is on a TCP connection, this will end up being a problem as each movement causes everyone else within range to receive a message. If everyone is moving, that's a heck of a lot of messages everyone is getting. Not only that, everyone needs a copy of the message in their TCP pipe, one at a time, sent from the server. So it quickly explodes. I guess you can think of it as O(N) clients each causing O(N) messages to go out, so it explodes quadratically. This isn't likely what I'm about to suggest, but you could move to a broadcast type system where you have some sort of UDP broadcast where you simply tell everyone about all the movements, and a alternative negative-ack channel where clients can ask for missed broadcasts. That way only O(N) messages are sent (the magic happens in the network devices). The issue with that is the internet isn't great for packet loss or multicast routing, so if you have O(N) resend requests you're not much better off. If you had all your clients in one datacenter it might work :)
- 3y ago

Nican 3y ago

It would be interesting to have blog posts about the optimizations that were made.

creatonez 3y ago
If you want to read more, some technical details that were previously only covered by scattered blog posts and github gists have been consolidated into the project's documentation - https://docs.papermc.io/folia/reference/overview https://docs.papermc.io/folia/reference/overview

d--b 3y ago

Honest question: is that a lot?

ocelotpotpie 3y ago
Most servers with the latest CPUs can only handle a few hundred players without bogging down, so yeah it's a fair bit.
zimpenfish 3y ago
My Beelink Ryzen 7 sometimes chugs with 3 people on it. Java Minecraft is terrible at actually doing stuff in an efficient manner.

lrem 3y ago

I'll just drop this factoid: Mineplex has won the Guinness World Records award on January 28, 2015 for having 34,434 concurrent players, the most on a Minecraft server at the time.

ocelotpotpie 3y ago
This is where things start to get a little fuzzy. Mineplex had the most concurrent players but they weren't on a single individual bare metal server or single server jar instance. A lot of large "servers" like Mineplex, Hypixel, etc run a proxy which sits in front of a bunch of other servers. The concept of a "server" can have many meanings in Minecraft so it gets fuzzy quickly. I'm not sure if this 1,000 person test is a record but I'm not aware of any other test running so many concurrent players on a single dedicated box with a single instance of Minecraft running. Folia takes advantage of more CPU threads so everyone is in sitting in the same server instance. Hypixel beat out Mineplex later on, BTW.
- lrem 3y ago
  Ha, nice, thanks. I've never looked into this deeper than accepting the fact that it's doable. Now, if only I had the spare time to look into how did they shard the servers...
  - ocelotpotpie 3y ago
    We tend to call those "networks", when they have a bunch of different gametypes and servers interconnected. At a smaller scale of a few hundred players it's pretty easily handled by off the shelf software like Velocity for the proxy and a few server instances running Paper. You generally want good performing (not VPS or "cloud" instances) but you can run a bunch of different worlds and have people with cross server talk, the ability to warp between worlds, etc. Most of the larger networks and nearly all of the smaller networks use a variation of this kind of model.
  - AgentK20 3y ago
    ocelotpotpie's sibling reply to this has good explanations for the terminology. If you're curious to actually see the software: * https://www.spigotmc.org/wiki/bungeecord/ https://www.spigotmc.org/wiki/bungeecord/ - Older, less-performant but still used by some teams * https://papermc.io/software/velocity https://papermc.io/software/velocity - Newer, more performant, maintained by the team that makes Paper, one of the leading performance MC server implementations. For context of scale btw, when Hypixel had 200k+ players online at peak we had something like 2,000+ 1U E3-1271v3's each with 32GB RAM, all colo'd in a single DC in Chicago. Egress is 70-80gbps or so 95th percentile, with most months (at high peak) egressing 10PB/mo+ of real-time, uncacheable data.

samsquire 3y ago

This is interesting. I am interested in multithreading and parallelism. So I have a journal entry to explore which is about deliberately desynchronizing and resynchronizing game loops for performance. * For the number of clients can use epoll or liburing. * Can multiplex multiple sockets per thread my epoll-server does this. * Can split out recv and send across threads so you can send while receiving and receive while sending * For the game loops can divide the territory covered to a different game loop. My question becomes how do you synchronize game loops across threads, you could latch on each thread for partial causality between game loops.

pjc50 3y ago
> deliberately desynchronizing and resynchronizing game loops for performance. > latch on each thread for partial causality between game loops. This sounds like a recipe for misery. You have to start thinking about ""light cones"" if causality propagates at finite speed. Do all observers in the game universe witness events in the same order? What if they don't? You get a little bit of this with rollback netcode, on the level of a few frames, but in that case there's always an authoritative causality and the client's guess is subordinate to it. It also sounds like fertile ground for exploits, both of the duplicating-object type and the glitch-through-geometry type.
- samsquire 3y ago
  Thank you for your reply. Your comment made me amused because it made me think of a game world where retrocausality was in effect, it would be absurd. Is the bottleneck for networked games the network of broadcasting updates? Or the CPU usage of updating object states? Sharding or instancing means you have fewer clients to broadcast stream updates to. Different game loops interacting with eachother, that's interesting. How else do you scale a game engine across threads?
  - pjc50 3y ago
    > Is the bottleneck for networked games the network of broadcasting updates? Or the CPU usage of updating object states? Bit of both depending on precisely what's happening. Ultimately the problem is that, for N "agents" (players or NPCs or active blocks or monsters, etc) in a space, if you're not careful you end up with O(N ^ 2) checks of the form "has X collided/interacted with Y". Octree systems can spread this out, but you're still vulnerable to "what if all the players go to the same spot?" Instancing lets you back off the worst-case situation by limiting the maximum number of players in a particular spot.

viraptor 3y ago

I know this isn't the point of Folia's test, but: > During the period when 1,000 players were online, we reached a maximum of ~7.9GB/s heap allocation and our GC was hovering around 2-3GB/s when averaged over a minute. As someone who dealt with soft-realtime telephony stuff, this makes me want to scream in horror. It seems like the platform really hurts the performance here. In an average second with that many players, most (all?) of them will not do any action apart from changing their position. (Just moving around would ideally only use preallocated per-player space only; extra in-game events, npc AI, etc. could still use some allocs) It's great Folia can get more out of the bad situation, but... oofff. (not a criticism of Minecraft by the way, it's better to be popular and inefficient than not exist - I just didn't realise quite how much overhead there is due to the runtime)

madduci 3y ago
Has anyone already tried to use GraalVM with it? Cool would be trying to do a native compilation and compare the performance
- tuetuopay 3y ago
  That's a sample size of one, but I did try it once and it ended up being basically the same. HotSpot took a measurable amount of time to reach GraalVM performance (in the minute if memory serves), but once the hot path got JITed, they were basically on par. Since a minecraft server is (usually) a long-lived process, it does not matter in the end.
- polski-g 3y ago
  I've tried different JVMs, different garbage collectors, etc, etc. They all make negligible difference.
jandrese 3y ago
The Minecraft Java version is notorious for thrashing memory. It allocates and discards objects at an incredible rate, even when the player isn't doing anything. The server has a GUI where you can watch the memory allocation with its distinctive sawtooth pattern.

blindriver 3y ago

On a side note, I've been trying to figure out how to write Java mods for server-side vanilla Minecraft server, but there is basically no documentation at all. Is it safe to say that this isn't possible? Which flavor of minecraft server is the best documented to write mods for, Spigot?

whatsthatabout 3y ago
Its paper nowadays :) https://papermc.io https://papermc.io
ocelotpotpie 3y ago
There's a bit of a distinction in the Minecraft world between "mods" and "plugins". Even though to the outside perspective they're the same thing, the terms are used for two different communities/ecosystems, basically. "Plugins" are for bukkit-based compatible servers, like Spigot and Paper (and Folia, with some modifications). "Mods" are more for forge, sponge, etc servers. So you can kinda pick which direction you want to go. But the Paper project has a ton of API documentation and a few Github repos to even get you started writing your own plugins. The Discord server has a dev help channel with people happy to answer any questions. The Sponge team has a similar setup as well if that's the route you'd like to go.
- bredren 3y ago
  Are any of these choices restricted in their existing or likely future compatibility with where Microsoft is taking the game? My understanding is server admins will spend money on mods / plugins. Are there any server types more likely to garner paid license or plugins?
  - ocelotpotpie 3y ago
    Microsoft has struct rules about the stuff you can sell for the server, and servers need to be EULA compliant. If they aren't they can get blacklisted or shutdown. People do buy plugins but the vast majority of the plugins you'd want to use are free open source plugins. A lot of the paid/closed source stuff is kinda crappy or misleading anyway. You can run a great server using free plugins and software. It might not be a money maker, but if your goal is to have people play and have fun and not buy stuff it's super easy and doable. People do still donate to projects to support the authors who make the wheels spin.

bullen 3y ago

https://www.twitch.tv/videos/1849538310 https://www.twitch.tv/videos/1849538310 No comment.

ocelotpotpie 3y ago
(Disclaimer: I'm on the Paper team, and Paper is the org under which Folia sits) I didn't watch the stream because I was offline when it happened, but I just want to note that the test was run by someone not on the Paper team who got tubbo to stream it, which is how they got so many players. It can be tricky to get enough players for such a large test, so it made sense for cubxity to pair up with someone to get more players. Twitch can get pretty funky, especially twitch chat, so hopefully there's nothing "bad" in that video, though.

haezee 3y ago

While very impressive, something to note: >The server was prepared with a 100k x 100k block pre-generated world. Our custom plugin distributed new players to the least-occupied region. Minecraft chunk generation is notoriously though, and even a few players generating new chunks will bring the TPS down to an unplayable state very fast.

skerit 3y ago
> Minecraft chunk generation is notoriously though, and even a few players generating new chunks will bring the TPS down to an unplayable state very fast. And that's knowing a big part of the chunk generation already happens off the main thread.
ocelotpotpie 3y ago
(Disclaimer: I'm on the Paper team, and Paper is the org under which Folia sits.) This is true to an extent, yes. Chunk generation can be real tough. That said, chunk generation has been re-written in Paper, so it's substantially faster than vanilla Minecraft. When we did a large scale real player test on a much earlier build of Folia we had ~327 players at peak and we did not generate chunks because we specifically wanted to see how it ran. We didn't max out at ~327, we just didn't have more joins than that. The project improved a bunch already so that number without chunk generation is very doable to beat if you didn't pre-gen any chunks. Generally it's free performance to pre-gen your world though. Highly recommended for regular players.
circuit10 3y ago
"generating new chunks" That's not happeneing, it says "pre-generated world"
- xboxnolifes 3y ago
  That's the point. The test doesn't include chunk generation.
  - circuit10 3y ago

josefrichter 3y ago

I know nothing about Minecraft servers, but has anyone tried to build one in Elixir/Erlang? Sounds like 1000 players should be a piece of cake.

zimpenfish 3y ago
I think the devil is in the details there - getting the terrain generation right, handling of things like redstone, and replicating the various glitches+bugs that people tend to use. Otherwise you've made a Minecraft-alike, not Minecraft.
nottorp 3y ago
It may run faster than java but it will have the same problems with close enough players having to be processed together in the same thread.

c7DJTLrn 3y ago

Cool stuff. The possibilities of Minecraft if some optimisation work was done would be amazing. It's demotivating just how rotten the internals of Minecraft are. Bedrock edition has really nice performance but nobody wants to play it.

tehbeard 3y ago
Plenty play bedrock. It's just.... a very sanitized and corporate experience. And for Java edition players just different enough in mechanics / ux / feel to annoy them. Very much pushes you to stick to official servers or small friend groups, and is quite aggressive on its monetization.
- circuit10 3y ago
  Bedrock is also much harder to mod, being written in C++, and mods are a big part of what people like about Minecraft. They have an add-on system but that's just fancy JSON config files so you can only do things like adding a new block, not complete reworks of the internal code to improve performance or anything like that which is totally possible on Java Edition
aprilnya 3y ago
Bedrock gives me an uncanny valley feeling… it’s Minecraft, sure, but everything is just slightly off, and it makes my brain really not like it
Pannoniae 3y ago
Bedrock rendering performance is really good. Tick performance? Not so much, the game is riddled with horrible issues such as world fallthrough which don't exist on Java (due to floating-point rounding errors), and I think the default tick distance is something like 4 or 6 chunks. Bedrock looks good, but the simulation itself is just broken. Not to mention the uncanny feeling as previous posters said - they've managed to create a similar game but with a sterilised, odd feeling. It's like watching a bad Disney adaptation of a book.

conaclos 3y ago

Just wondering: Is Paper consuming less resources than the vanilla Minecraft server for a few players (1 to 10 players). In other words, is it less costly to operate a Paper server on the cloud?

doodlesdev 3y ago
Absolutely. Depending on your hardware using Paper can be the difference of the server not even starting and a playable game. Paper is an absolutely incredible effort.