4 ms·

Deleting multiplayer from the Unreal engine can save memory

163 points by mariuz 2y ago

bengarney 2y ago
Really interesting analysis of where the data lives… cutting 3-4 textures would save you more memory even in the 100k actor case, though.
- reitzensteinm 2y ago
  Depending on a bunch of factors of how this data is accessed and actors are laid out in memory, it may be more cache friendly which could yield substantial speedups. Or it could do next to nothing, as the data is multiple cache lines long anyway.
  - bengarney 2y ago
    I would not expect much, but you'd have to measure to be sure. If you actually have a million of something you're better off writing a custom manager thing to handle the bulk of the work anyway. For instance, if you're doing a brick building game where users might place a million bricks - maybe you want each brick to be an Actor for certain use cases, but you'd want to centralize all the collision, rendering, update logic. (This is what I did on a project with this exact use case and it worked nicely.)
    - reitzensteinm 2y ago
      I wouldn't expect much either. The potential for speedups would be if there's locality for data on either side of the multiplayer padding, or if the actors have contiguous layout and deleting the data plays better with the CPU's stride prefetching. Significant performance degredation is also possible if at some point a smart (but not wise) developer positioned the data to eliminate false sharing on either side. Agreed that you shouldn't be using this heavy weight paradigm with large amounts of entities. My intention was just to add a bit of color to the idea that saving memory allocations can have implications beyond just the number of bytes you ultimately malloc.
- cma 2y ago
  If the memory savings he got were fully read or fragmented with other stuff on cache lines that are read in every frame (not likely for static world actors), it could be ~10% of CPU memory bandwidth on mobile every frame at 120hz on an lpddr4 phone. A big problem with them is they are so heavyweight you can only spawn a few per frame before causing hitches and have to have pools or instancing to manage things like bullets. I think in their Robo Recall talk they found they could only spawn 10-20 projectile style bullets per frame before running into hitches, and switched to pools and recycling them.
  - teamonkey 2y ago
    Pooling is pretty standard practice though, it would be the go-to solution for any experienced gameplay programmer when dealing with more than a dozen entities (though annoyingly there isn’t a standardised way of doing it in Blueprint).
    - dijit 2y ago
      To be completely fair though, blueprints themselves are oft-maligned for performance. They're fantastic for prototyping, but once you have designed some kind of hot-path most people typically start converting blueprints to code as an optimisation. In such a scenario adding pooling becomes a trivial part of such an effort.
    - cma 2y ago
      Standard practice, but it bit Epic by surprise. You wouldn't think it would be needed at such small numbers. You wouldn't automatically think it would be needed on 3+ghz machines.
  - Pxtl 2y ago
    I've never played with UE and so I'm kinda shocked to learn that there isn't pooling already for objects that have this kind of creation cost.
spockz 2y ago
Besides the memory savings, would there also be performance gains? More actors would now fit in a cache line.
- cma 2y ago
  The actors in unreal are such bloated single inheritance god classes that with a few actor components they take up more like a 4K memory page not part of a cache line, especially in editor builds. But they do have a more optimized entity component system now too. To be fair, a single transform now that things are 64 bit coordinates I think is bigger than a cache line too.
- larstofus 2y ago
  I haven't profiled this specifically, but my guess is that there shouldn't be any measurable performance gains. Most of the time, actors are randomly scattered in memory anyway, so having smaller actors doesn't avoid a lot of cache misses.
out-of-ideas 2y ago
is it me or does the side loop endlessly loading thousands of requests? to at least: - https://public-api.wordpress.com/wp-admin/rest-proxy/#https://widgets.wp.com https://public-api.wordpress.com/wp-admin/rest-proxy/#https:... - https://s0.wp.com/wp-content/js/rlt-proxy.js?m=20240709 https://s0.wp.com/wp-content/js/rlt-proxy.js?m=20240709 blocking these via regex made the page load up really nice and fast edit: formatting
keyle 2y ago
Would this have any effect in prod build though?
- teamonkey 2y ago
  That was my immediate thought on reading the article. Is this in dev or prod builds and which UE version is it referring to? (Not that I expect the UActor code to have changed much but modifying UActor seemed more common in the early 4.x days.)
  - larstofus 2y ago
    Oh, good point. The test was done in the current stable 5.5 release in a development configuration :) Since this is a change to the memory layout of the class, there should be no change due to the build configuration, though.
mwkaufma 2y ago
On ABZÛ we ripped out multiplayer and lots of other cruft from AActor and UPrimitiveComponent - dropping builtin overlap events, which are a kind of anti pattern anyway, not only saved RAM but cut a lot of ghost reads/writes.
- jayd16 2y ago
  What makes overlap events an anti-pattern?
  - mwkaufma 2y ago
    In principle it's a fine idea, but their implementation has so many footguns (race conditions, thundering herds, perf cliffs, etc) it was easier to impl your own simpler alternative.
- rc5150 2y ago
  Whoa! Didn't picture myself seeing a dev who worked on Abzu in the wild here on HN--I very much enjoyed that game, my thanks and high fives to your team for your work! I'm having a kid-in-the-tunnel-meeting-Mean-Joe-Green-in-the-commercial moment, I just started my own game development journey about a week ago so it's neat getting to run across a full-on developer! To stay on topic, I often thought how cool Abzu would have been with multiplayer but it's a good lesson to me that some features that might be desirable might also be a hindrance to some degree. Okay, enough fanboying!
- thaumasiotes 2y ago
  Your presence inspired me to try to look up what the circumflex on "abzû" is supposed to signify. As best I can tell, it's a marker of vowel length. I wonder how that came to be used. It's a traditional way to distinguish eta and omega in transliteration from Greek, but it's not at all a traditional way to mark long vowels in general. (I see that wikipedia says this about Akkadian: > Long vowels are transliterated with a macron (ā, ē, ī, ū) or a circumflex (â, ê, î, û), the latter being used for long vowels arising from the contraction of vowels in hiatus. But it seems odd for an independent root to contain a contracted double vowel. And the page "Abzu" has the circumflex on the Sumerian transliteration too.)
joegibbs 2y ago
You would probably want to avoid having tens of thousands or a hundred thousand actors though, they're pretty heavy regardless. There might be a few reasons why you'd want to have that many but I think ideally you'd want to instance them or have some kind of actor that handles UObjects instead
Fokamul 2y ago
Too bad big companies don't care about this and more. "Morons(gamers) will just buy new hardware, fu hiring engine core devs".
- seivan 2y ago
  [dead]
- maccard 2y ago
  This attitude comes up on here whenever gamedev comes up, and I really dislike it. Here's a quote form the article > I’ve already told you that this method saves 328 bytes per actor, which is not too much at first glance. You can apply the same trick for SceneComponents and save another 32 bytes per SceneComponent. Assuming an average of two SceneComponents per actor, you get up to 392 bytes per actor. Still not an impressive number unless you deal with a lot of actors. A hypothetical example level with 25 000 actors (which is a lot, but not unreasonable) will save about 10 MB. I've a lot of experience with Unreal, and 25k actors is likely to run into a whole host of problems, such that saving 10MB of RAM is likely to be the least of your worries. You'd get more benefit out of removing a single unneeded texture, or compressing a single animation better. One of the reasons developers use unreal (and yes, developers do use Unreal, it's not just "big companies" forcing their poor creatives to use the engine) is _because_ unreal has more man hours of development in a year than a small team would ever be able to put into their own engine. Like any tool it has tradeoffs, and it does have a (measureable) overhead. But to say that companies don't care is just disingenuous
  - speed_spread 2y ago
    Actors are handled by the CPU where shaving 10MB can mean that more things can now fit in the cache leading to dramatic improvement.
    - maccard 2y ago
      If you’re going to make that assertion then back it up with numbers. It could just easily have absolutely no impact whatsoever because your game thread is spending all its time on navigation mesh queries which have nothing to do with actors or UObjects.
jofla_net 2y ago
This was true even in the first version over 20 years ago. In a single player derivative, I remember combing through tons of UScript, unrealscript, stanzas which went something like. "Do this, and if we're in multiplayer, do this too or instead." The code was a messs, but again, good times.
dleslie 2y ago
This has been a pattern since the first release of Unreal Engine. It's how we managed to smoosh it onto PS2 and Xbox.
thenthenthen 2y ago
How to implement A-life in Stalker 2
shadowgovt 2y ago
This is a really good writeup. Something the author doesn't mention is that shrinking your data structures is also helpful for cache cohesion: if your structures are smaller, more of them can fit in smaller CPU caches (even if the game engine is striping resources of the same kind to simplify ripping across them every frame, this can matter). The only counterweight I'd add is that if you later decide to add multiplayer, that is very, very hard if the engine wasn't set up for it from the beginning. Multiplayer adds complexity that exceeds simple things like getting on the network and sending messages; synchronization and prediction are meaningful for a realtime experience and are much easier to get to if you've started from "It's multiplayer under the hood but the server is local" than "We have a single-player realtime game and we're making it multiplayer." But that's not a reason never to do this; not all games need to be multiplayer!

163 points by mariuz 2y ago

Really interesting analysis of where the data lives… cutting 3-4 textures would save you more memory even in the 100k actor case, though.

reitzensteinm 2y ago
Depending on a bunch of factors of how this data is accessed and actors are laid out in memory, it may be more cache friendly which could yield substantial speedups. Or it could do next to nothing, as the data is multiple cache lines long anyway.
- bengarney 2y ago
  I would not expect much, but you'd have to measure to be sure. If you actually have a million of something you're better off writing a custom manager thing to handle the bulk of the work anyway. For instance, if you're doing a brick building game where users might place a million bricks - maybe you want each brick to be an Actor for certain use cases, but you'd want to centralize all the collision, rendering, update logic. (This is what I did on a project with this exact use case and it worked nicely.)
  - reitzensteinm 2y ago
    I wouldn't expect much either. The potential for speedups would be if there's locality for data on either side of the multiplayer padding, or if the actors have contiguous layout and deleting the data plays better with the CPU's stride prefetching. Significant performance degredation is also possible if at some point a smart (but not wise) developer positioned the data to eliminate false sharing on either side. Agreed that you shouldn't be using this heavy weight paradigm with large amounts of entities. My intention was just to add a bit of color to the idea that saving memory allocations can have implications beyond just the number of bytes you ultimately malloc.
cma 2y ago
If the memory savings he got were fully read or fragmented with other stuff on cache lines that are read in every frame (not likely for static world actors), it could be ~10% of CPU memory bandwidth on mobile every frame at 120hz on an lpddr4 phone. A big problem with them is they are so heavyweight you can only spawn a few per frame before causing hitches and have to have pools or instancing to manage things like bullets. I think in their Robo Recall talk they found they could only spawn 10-20 projectile style bullets per frame before running into hitches, and switched to pools and recycling them.
- teamonkey 2y ago
  Pooling is pretty standard practice though, it would be the go-to solution for any experienced gameplay programmer when dealing with more than a dozen entities (though annoyingly there isn’t a standardised way of doing it in Blueprint).
  - dijit 2y ago
    To be completely fair though, blueprints themselves are oft-maligned for performance. They're fantastic for prototyping, but once you have designed some kind of hot-path most people typically start converting blueprints to code as an optimisation. In such a scenario adding pooling becomes a trivial part of such an effort.
  - cma 2y ago
    Standard practice, but it bit Epic by surprise. You wouldn't think it would be needed at such small numbers. You wouldn't automatically think it would be needed on 3+ghz machines.
- Pxtl 2y ago
  I've never played with UE and so I'm kinda shocked to learn that there isn't pooling already for objects that have this kind of creation cost.

spockz 2y ago

Besides the memory savings, would there also be performance gains? More actors would now fit in a cache line.

cma 2y ago
The actors in unreal are such bloated single inheritance god classes that with a few actor components they take up more like a 4K memory page not part of a cache line, especially in editor builds. But they do have a more optimized entity component system now too. To be fair, a single transform now that things are 64 bit coordinates I think is bigger than a cache line too.
larstofus 2y ago
I haven't profiled this specifically, but my guess is that there shouldn't be any measurable performance gains. Most of the time, actors are randomly scattered in memory anyway, so having smaller actors doesn't avoid a lot of cache misses.

out-of-ideas 2y ago

is it me or does the side loop endlessly loading thousands of requests? to at least: - https://public-api.wordpress.com/wp-admin/rest-proxy/#https://widgets.wp.com https://public-api.wordpress.com/wp-admin/rest-proxy/#https:... - https://s0.wp.com/wp-content/js/rlt-proxy.js?m=20240709 https://s0.wp.com/wp-content/js/rlt-proxy.js?m=20240709 blocking these via regex made the page load up really nice and fast edit: formatting

keyle 2y ago

Would this have any effect in prod build though?

teamonkey 2y ago
That was my immediate thought on reading the article. Is this in dev or prod builds and which UE version is it referring to? (Not that I expect the UActor code to have changed much but modifying UActor seemed more common in the early 4.x days.)
- larstofus 2y ago
  Oh, good point. The test was done in the current stable 5.5 release in a development configuration :) Since this is a change to the memory layout of the class, there should be no change due to the build configuration, though.

mwkaufma 2y ago

On ABZÛ we ripped out multiplayer and lots of other cruft from AActor and UPrimitiveComponent - dropping builtin overlap events, which are a kind of anti pattern anyway, not only saved RAM but cut a lot of ghost reads/writes.

jayd16 2y ago
What makes overlap events an anti-pattern?
- mwkaufma 2y ago
  In principle it's a fine idea, but their implementation has so many footguns (race conditions, thundering herds, perf cliffs, etc) it was easier to impl your own simpler alternative.
rc5150 2y ago
Whoa! Didn't picture myself seeing a dev who worked on Abzu in the wild here on HN--I very much enjoyed that game, my thanks and high fives to your team for your work! I'm having a kid-in-the-tunnel-meeting-Mean-Joe-Green-in-the-commercial moment, I just started my own game development journey about a week ago so it's neat getting to run across a full-on developer! To stay on topic, I often thought how cool Abzu would have been with multiplayer but it's a good lesson to me that some features that might be desirable might also be a hindrance to some degree. Okay, enough fanboying!
thaumasiotes 2y ago
Your presence inspired me to try to look up what the circumflex on "abzû" is supposed to signify. As best I can tell, it's a marker of vowel length. I wonder how that came to be used. It's a traditional way to distinguish eta and omega in transliteration from Greek, but it's not at all a traditional way to mark long vowels in general. (I see that wikipedia says this about Akkadian: > Long vowels are transliterated with a macron (ā, ē, ī, ū) or a circumflex (â, ê, î, û), the latter being used for long vowels arising from the contraction of vowels in hiatus. But it seems odd for an independent root to contain a contracted double vowel. And the page "Abzu" has the circumflex on the Sumerian transliteration too.)

joegibbs 2y ago

You would probably want to avoid having tens of thousands or a hundred thousand actors though, they're pretty heavy regardless. There might be a few reasons why you'd want to have that many but I think ideally you'd want to instance them or have some kind of actor that handles UObjects instead

Fokamul 2y ago

Too bad big companies don't care about this and more. "Morons(gamers) will just buy new hardware, fu hiring engine core devs".

seivan 2y ago
[dead]
maccard 2y ago
This attitude comes up on here whenever gamedev comes up, and I really dislike it. Here's a quote form the article > I’ve already told you that this method saves 328 bytes per actor, which is not too much at first glance. You can apply the same trick for SceneComponents and save another 32 bytes per SceneComponent. Assuming an average of two SceneComponents per actor, you get up to 392 bytes per actor. Still not an impressive number unless you deal with a lot of actors. A hypothetical example level with 25 000 actors (which is a lot, but not unreasonable) will save about 10 MB. I've a lot of experience with Unreal, and 25k actors is likely to run into a whole host of problems, such that saving 10MB of RAM is likely to be the least of your worries. You'd get more benefit out of removing a single unneeded texture, or compressing a single animation better. One of the reasons developers use unreal (and yes, developers do use Unreal, it's not just "big companies" forcing their poor creatives to use the engine) is _because_ unreal has more man hours of development in a year than a small team would ever be able to put into their own engine. Like any tool it has tradeoffs, and it does have a (measureable) overhead. But to say that companies don't care is just disingenuous
- speed_spread 2y ago
  Actors are handled by the CPU where shaving 10MB can mean that more things can now fit in the cache leading to dramatic improvement.
  - maccard 2y ago
    If you’re going to make that assertion then back it up with numbers. It could just easily have absolutely no impact whatsoever because your game thread is spending all its time on navigation mesh queries which have nothing to do with actors or UObjects.

jofla_net 2y ago

This was true even in the first version over 20 years ago. In a single player derivative, I remember combing through tons of UScript, unrealscript, stanzas which went something like. "Do this, and if we're in multiplayer, do this too or instead." The code was a messs, but again, good times.

dleslie 2y ago

This has been a pattern since the first release of Unreal Engine. It's how we managed to smoosh it onto PS2 and Xbox.

thenthenthen 2y ago

How to implement A-life in Stalker 2

shadowgovt 2y ago

This is a really good writeup. Something the author doesn't mention is that shrinking your data structures is also helpful for cache cohesion: if your structures are smaller, more of them can fit in smaller CPU caches (even if the game engine is striping resources of the same kind to simplify ripping across them every frame, this can matter). The only counterweight I'd add is that if you later decide to add multiplayer, that is very, very hard if the engine wasn't set up for it from the beginning. Multiplayer adds complexity that exceeds simple things like getting on the network and sending messages; synchronization and prediction are meaningful for a realtime experience and are much easier to get to if you've started from "It's multiplayer under the hood but the server is local" than "We have a single-player realtime game and we're making it multiplayer." But that's not a reason never to do this; not all games need to be multiplayer!