5 ms·

Building a gameserver in Elixir is not great imo, performance is subpar. I work in online games and no one would want to use that language. I guess they have l

by Thaxll 5y ago

Building a gameserver in Elixir is not great imo, performance is subpar. I work in online games and no one would want to use that language. I guess they have low constrains since they only need to support 10000 players at a time and they want a single instance for that. - Erlang runtime is slow - Memory consumption is high - GC is not very good ( vs Java / C# / Go ) - high cost in term of performance because of immutability It's not the kind of things you want for a game. To give you some perspective on popular FPS games, they run between 30hz and 120hz with up to 64 players and usually only use a single CPU core. And they're doing gameplay simulation / state replication / physics sometimes AI etc ... Also it must be pretty painful to not re-use their C++ client code on the server side, they have to re-implement twice? ( this concern is actually higher than the performance one ) Overall I think it's an impressive achievement to be able to do that for a single person in 6 month!

tudelo 5y ago
So, since you seem to have some knowledge here, what would you suggest a single engineer trying to make a somewhat scalable multiplayer game? Either instanced with <64 players or a more larger user count. I did some experimenting with rolling my own networking layer in c++ and it was a lot of work, and tried to do it through unity but it seems to require quite a large investment in time to understand how networking fits in the overall picture of the game. But maybe I'm looking for magic where it does not or can not exist :)
- Thaxll 5y ago
  The way to go is usually to re-use your game client and strip the rendering part.
ravi-delia 5y ago
I don't disagree with you on running games on the BEAM (although slow is a little too far imo), but in this case what they're doing is using Elixir to connect clients with the server. The actual physics simulation and such isn't running on Elixir. edit: I just read through the article, and it doesn't seem to say one way or the other. 99% sure they aren't running any kind of physics engine on the BEAM, but there isn't anything specific
- Thaxll 5y ago
  The article is no clear on what the server is actually doing, for example just replicating state between players through that hub or it's doing simulation. Either way they had to re-implement Racknet which is annoying because from now on everytime they add / change something in the network layer they won't be able to re-use the client code in the server. I'm curious on why they did not go with C++.
  - Zababa 5y ago
    Answered here: https://news.ycombinator.com/item?id=28000050 https://news.ycombinator.com/item?id=28000050 "Despite being way more familiar with C++, I can't imagine going from zero to production in 6 months (or even 2 years) if we'd tried to do this in C++"
  - s3cur3 5y ago
    This post on our dev blog gives some detail on the language decision here: https://developer.x-plane.com/2021/01/have-you-heard-the-good-news-about-elixir/ https://developer.x-plane.com/2021/01/have-you-heard-the-goo...
andrewmcwatters 5y ago
Only on HN can you provide specific industry advice and be downvoted for your insight because your opinion wasn’t agreeable. I upvoted your comment to bring it out of 0 or less because these concerns are valid and critically make or break important when building multiplayer servers. My relevant background here is in building multiplayer server software where a minimum of 1000 concurrent players is a required goal metric.
- Zababa 5y ago
  I don't think the downvoting is linked to opinions, it's linked to facts. Why should "specific industry advice" be preferred over working code? That sounds dogmatic to me. > My relevant background here is in building multiplayer server software where a minimum of 1000 concurrent players is a required goal metric. They are handling 10k concurrent players with low resources usage.
  - andrewmcwatters 5y ago
    That’s really great, but as soon as you understand their specific requirements it’s simple to go from feasible to do 100000 concurrent clients to impossible with compute budget constraints. You can build a “game” with a million “concurrent” clients that do absolutely nothing, to dealing with thousands where you have tick rates that make that impossible.[1] [1]: https://www.planimeter.org/grid-sdk/api/Tick_rate_and_bandwidth https://www.planimeter.org/grid-sdk/api/Tick_rate_and_bandwi...
    - Zababa 5y ago
      I don't really understand your point. Are you saying that the resources used will scale exponentially with the number of players? If that's the case, I can see how this could make things difficult in the long run but caring about it now sounds a bit like premature optimization to me. Maybe the players don't even want to be all on the same world?
      - andrewmcwatters 5y ago
s3cur3 5y ago
YMMV, but: - Memory consumption in production peaks at like 300 MB for us - Scalability (i.e., being able to go wide on, say, a 64-core CPU) was more important to us than raw, synchronous speed - Garbage collection is not something I've ever had to think about with Elixir (which hasn't been my experience with Python and Java) - Because it's a sim and not something like a FPS, we don't have to worry about cheating (what would cheating even mean?), so we don't have to do any physics on the server side - Despite being way more familiar with C++, I can't imagine going from zero to production in 6 months (or even 2 years) if we'd tried to do this in C++
- andrewmcwatters 5y ago
  > what would cheating even mean? This is such a rudimentary question in game development that I don’t know how you can’t answer it. Mainstream server authoritative design has been a thing since 1999, and probably even longer when you look beyond Quakeworld. All it takes is a few malformed packets from a script kiddie and then you realize you have to do distance checking and other anti-cheat functionality. “Cheating” in gamedev doesn’t specifically mean there is a win scenario and you need to make sure people don’t shortcut that and play unfairly. It means in this case that someone can’t connect to your server and transport their plane to another nearby player and wave it around in their flight path to annoy others while they spam chat to grief people for fun and to produce a YouTube video. And those checks are pretty simple.
  - s3cur3 5y ago
    Sorry, to clarify, our thinking on this point was: our players aren't particularly motivated to lie to the server since there isn't a "goal" to a flight simulator. Given the choice between shipping (much, much) faster and completely locking down the experience against the vague, yet-to-be-realized threat of people sending fake packets to the server, we opted for the former.
    - networked 5y ago
      Have you had any reports of multiplayer griefing so far? I wonder if griefing happens in serious flight simulators like X-Plane and what form it takes if it does. (If there are griefers, then they might conceivably find a way to exploit this. But I am asking mostly because I am curious about what the player community is like.)
Zababa 5y ago
They did implement their game server in Elixir, and it performs great for them in reality. From what I understand, they are supporting 10 000 players with a low CPU and memory usage, so if this scales linearly they should be able to support 100k players. You're also ignoring the reliability aspect, which is why they chose Elixir (and the BEAM) in the first place. Performance is also less of a problem since they're on the server, and thus can add more hardware as needed. Could you share a bit more about your experience in the industry? I think you and they have different use cases, which makes you reach different conclusions.
andy_ppp 5y ago
Using the right tool for the right job is important, here they chose Elixir and it’s working well for them. There’s no need to be this disgruntled with your ill informed opinions about the Erlang VM (gc not very good - no it’s just optimised for different things, latency being one). The Elixir core team have recently released nx-elixir which allows much faster maths and allows mutability as the calculations happen outside of the runtime, so you might want to look at this before you make assumptions.
- anonymousDan 5y ago
  To claim GC of the JVM as an advantage Vs elixir is farcical, the lack of shared state/independent heaps is one of the main advantages of Erlang Vs stop-the-world GC with the JVM.
  - h0l0cube 5y ago
    Another recent advance is the BEAM JIT. And I'm pretty sure you can call native code using 'nifs' for physics calculations etc. I'm wondering if the OP is basing their opinion on recent performance metrics, or just speculation, or something they heard?
  - masklinn 5y ago
    TBF it depends on the workload and architecture (likewise, say, Go v JVM GC). If you can split your workload in tons of actors the beam GC works great. If you need very large synchronous actors (even just a few) then it’s excruciating, so is splitting a simple single-threaded sequential workload into a concurrent mess just so the GC does not eat itself. Of course you probably should not be using erlang/elixir then. But let’s not pretend the beam gc doesn’t have issues and pitfalls.
    - anonymousDan 5y ago
      Can you give an example of what you mean by 'synchronous actors'? Blocking on what exactly? The point is surely that elixir/Erlang makes it difficult to write concurrent code with lots of global shared state in the first place no?
jbluepolarbear 5y ago
What would you choose to build this same feature set? Would it be feasible for 1 person in 6 months? What if C++ isn’t an option?
namelosw 5y ago
> GC is not very good ( vs Java / C# / Go ) IMHO it’s the opposite, at least in the context of games: In languages like Java and C#, the GC is global and freezes the world. The time would be in terms of milliseconds, while in Erlang the GC is per-process, in terms of microseconds. So in terms of GC and latency, Erlang is not as good as C++ and Rust, but it should be better than Java and C#. > Also it must be pretty painful to not re-use their C++ client code on the server side, they have to re-implement twice? I haven’t dive into the problem but my intuitive tells me it could be done through NIF. It’s a pretty common practice to use Rust to compliment Elixir in performance hotspots.
yellowapple 5y ago
Re: performance issues, offloading CPU-intensive operations to NIFs (or ports, if I/O bandwidth needs are low enough and you need to run something long-running) is a pretty common strategy for speeding up Erlang (and therefore Elixir) applications. It doesn't seem like the X-Plane folks have felt the need to do this, but it's definitely a possibility should they run into some computational bottleneck. Re: GC, the Erlang VM does it per-process ("process" in the EVM's sense, i.e. a preemptively-scheduled userspace/"green" thread rather than an OS process), so not only are GC operations fast, but they also only affect the process being GC'd. The immutability is also a factor here; since data is copied instead of modified in place, it's arguably a lot easier to know when to discard unused objects (namely: for an object to persist within a process, the recursive function(s) backing some process has to explicitly pass it along to the next invocation, so that's a natural place to discard everything else).

by Thaxll 5y ago

tudelo 5y ago

So, since you seem to have some knowledge here, what would you suggest a single engineer trying to make a somewhat scalable multiplayer game? Either instanced with <64 players or a more larger user count. I did some experimenting with rolling my own networking layer in c++ and it was a lot of work, and tried to do it through unity but it seems to require quite a large investment in time to understand how networking fits in the overall picture of the game. But maybe I'm looking for magic where it does not or can not exist :)

Thaxll 5y ago
The way to go is usually to re-use your game client and strip the rendering part.

ravi-delia 5y ago

I don't disagree with you on running games on the BEAM (although slow is a little too far imo), but in this case what they're doing is using Elixir to connect clients with the server. The actual physics simulation and such isn't running on Elixir. edit: I just read through the article, and it doesn't seem to say one way or the other. 99% sure they aren't running any kind of physics engine on the BEAM, but there isn't anything specific

Thaxll 5y ago
The article is no clear on what the server is actually doing, for example just replicating state between players through that hub or it's doing simulation. Either way they had to re-implement Racknet which is annoying because from now on everytime they add / change something in the network layer they won't be able to re-use the client code in the server. I'm curious on why they did not go with C++.
- Zababa 5y ago
  Answered here: https://news.ycombinator.com/item?id=28000050 https://news.ycombinator.com/item?id=28000050 "Despite being way more familiar with C++, I can't imagine going from zero to production in 6 months (or even 2 years) if we'd tried to do this in C++"
- s3cur3 5y ago
  This post on our dev blog gives some detail on the language decision here: https://developer.x-plane.com/2021/01/have-you-heard-the-good-news-about-elixir/ https://developer.x-plane.com/2021/01/have-you-heard-the-goo...

andrewmcwatters 5y ago

Only on HN can you provide specific industry advice and be downvoted for your insight because your opinion wasn’t agreeable. I upvoted your comment to bring it out of 0 or less because these concerns are valid and critically make or break important when building multiplayer servers. My relevant background here is in building multiplayer server software where a minimum of 1000 concurrent players is a required goal metric.

Zababa 5y ago
I don't think the downvoting is linked to opinions, it's linked to facts. Why should "specific industry advice" be preferred over working code? That sounds dogmatic to me. > My relevant background here is in building multiplayer server software where a minimum of 1000 concurrent players is a required goal metric. They are handling 10k concurrent players with low resources usage.
- andrewmcwatters 5y ago
  That’s really great, but as soon as you understand their specific requirements it’s simple to go from feasible to do 100000 concurrent clients to impossible with compute budget constraints. You can build a “game” with a million “concurrent” clients that do absolutely nothing, to dealing with thousands where you have tick rates that make that impossible.[1] [1]: https://www.planimeter.org/grid-sdk/api/Tick_rate_and_bandwidth https://www.planimeter.org/grid-sdk/api/Tick_rate_and_bandwi...
  - Zababa 5y ago
    I don't really understand your point. Are you saying that the resources used will scale exponentially with the number of players? If that's the case, I can see how this could make things difficult in the long run but caring about it now sounds a bit like premature optimization to me. Maybe the players don't even want to be all on the same world?
    - andrewmcwatters 5y ago

s3cur3 5y ago

YMMV, but: - Memory consumption in production peaks at like 300 MB for us - Scalability (i.e., being able to go wide on, say, a 64-core CPU) was more important to us than raw, synchronous speed - Garbage collection is not something I've ever had to think about with Elixir (which hasn't been my experience with Python and Java) - Because it's a sim and not something like a FPS, we don't have to worry about cheating (what would cheating even mean?), so we don't have to do any physics on the server side - Despite being way more familiar with C++, I can't imagine going from zero to production in 6 months (or even 2 years) if we'd tried to do this in C++

andrewmcwatters 5y ago
> what would cheating even mean? This is such a rudimentary question in game development that I don’t know how you can’t answer it. Mainstream server authoritative design has been a thing since 1999, and probably even longer when you look beyond Quakeworld. All it takes is a few malformed packets from a script kiddie and then you realize you have to do distance checking and other anti-cheat functionality. “Cheating” in gamedev doesn’t specifically mean there is a win scenario and you need to make sure people don’t shortcut that and play unfairly. It means in this case that someone can’t connect to your server and transport their plane to another nearby player and wave it around in their flight path to annoy others while they spam chat to grief people for fun and to produce a YouTube video. And those checks are pretty simple.
- s3cur3 5y ago
  Sorry, to clarify, our thinking on this point was: our players aren't particularly motivated to lie to the server since there isn't a "goal" to a flight simulator. Given the choice between shipping (much, much) faster and completely locking down the experience against the vague, yet-to-be-realized threat of people sending fake packets to the server, we opted for the former.
  - networked 5y ago
    Have you had any reports of multiplayer griefing so far? I wonder if griefing happens in serious flight simulators like X-Plane and what form it takes if it does. (If there are griefers, then they might conceivably find a way to exploit this. But I am asking mostly because I am curious about what the player community is like.)

Zababa 5y ago

They did implement their game server in Elixir, and it performs great for them in reality. From what I understand, they are supporting 10 000 players with a low CPU and memory usage, so if this scales linearly they should be able to support 100k players. You're also ignoring the reliability aspect, which is why they chose Elixir (and the BEAM) in the first place. Performance is also less of a problem since they're on the server, and thus can add more hardware as needed. Could you share a bit more about your experience in the industry? I think you and they have different use cases, which makes you reach different conclusions.

andy_ppp 5y ago

Using the right tool for the right job is important, here they chose Elixir and it’s working well for them. There’s no need to be this disgruntled with your ill informed opinions about the Erlang VM (gc not very good - no it’s just optimised for different things, latency being one). The Elixir core team have recently released nx-elixir which allows much faster maths and allows mutability as the calculations happen outside of the runtime, so you might want to look at this before you make assumptions.

anonymousDan 5y ago
To claim GC of the JVM as an advantage Vs elixir is farcical, the lack of shared state/independent heaps is one of the main advantages of Erlang Vs stop-the-world GC with the JVM.
- h0l0cube 5y ago
  Another recent advance is the BEAM JIT. And I'm pretty sure you can call native code using 'nifs' for physics calculations etc. I'm wondering if the OP is basing their opinion on recent performance metrics, or just speculation, or something they heard?
- masklinn 5y ago
  TBF it depends on the workload and architecture (likewise, say, Go v JVM GC). If you can split your workload in tons of actors the beam GC works great. If you need very large synchronous actors (even just a few) then it’s excruciating, so is splitting a simple single-threaded sequential workload into a concurrent mess just so the GC does not eat itself. Of course you probably should not be using erlang/elixir then. But let’s not pretend the beam gc doesn’t have issues and pitfalls.
  - anonymousDan 5y ago
    Can you give an example of what you mean by 'synchronous actors'? Blocking on what exactly? The point is surely that elixir/Erlang makes it difficult to write concurrent code with lots of global shared state in the first place no?

jbluepolarbear 5y ago

What would you choose to build this same feature set? Would it be feasible for 1 person in 6 months? What if C++ isn’t an option?

namelosw 5y ago

> GC is not very good ( vs Java / C# / Go ) IMHO it’s the opposite, at least in the context of games: In languages like Java and C#, the GC is global and freezes the world. The time would be in terms of milliseconds, while in Erlang the GC is per-process, in terms of microseconds. So in terms of GC and latency, Erlang is not as good as C++ and Rust, but it should be better than Java and C#. > Also it must be pretty painful to not re-use their C++ client code on the server side, they have to re-implement twice? I haven’t dive into the problem but my intuitive tells me it could be done through NIF. It’s a pretty common practice to use Rust to compliment Elixir in performance hotspots.

yellowapple 5y ago

Re: performance issues, offloading CPU-intensive operations to NIFs (or ports, if I/O bandwidth needs are low enough and you need to run something long-running) is a pretty common strategy for speeding up Erlang (and therefore Elixir) applications. It doesn't seem like the X-Plane folks have felt the need to do this, but it's definitely a possibility should they run into some computational bottleneck. Re: GC, the Erlang VM does it per-process ("process" in the EVM's sense, i.e. a preemptively-scheduled userspace/"green" thread rather than an OS process), so not only are GC operations fast, but they also only affect the process being GC'd. The immutability is also a factor here; since data is copied instead of modified in place, it's arguably a lot easier to know when to discard unused objects (namely: for an object to persist within a process, the recursive function(s) backing some process has to explicitly pass it along to the next invocation, so that's a natural place to discard everything else).