11 ms·

Serialization for C# Games

133 points by jolexxa 2y ago

Madmallard 2y ago
RunUO has an implementation of this and it's like 25 years old but still worked really well
- mentos 2y ago
  Wow clicked into the thread to see if anyone might mention RunUO :) it’s the only exposure I’ve had to serialization in C# I always wondered how it ranked compared to other approaches.
  - w-ll 2y ago
    As someone that also fell in love with C# with RunUO. I never actually looked at the Serialization at the time. Need to spend some time in RunUO or the fork soon. https://github.com/runuo/runuo/blob/master/Server/Serialization.cs https://github.com/runuo/runuo/blob/master/Server/Serializat...
- sanex 2y ago
  Ultima online solved all our problems 25 years ago.
- jolexxa 2y ago
  I really like this implementation, but it's probably worth mentioning here that RunUO and other tools like it are solving the problem at a layer of abstraction beneath what I was introducing here. The serialization system I am providing here actually leverages System.Text.Json for reading and writing data — it's more concerned with helping you represent version-able, upgrade-able data models that are also compatible with the hierarchical state machine implementation I use for managing game state.
interroboink 2y ago
Sometimes, when battling these issues, I wish the Smalltalk-style approach[1][2] was more popular/feasible. Basically, saving the entire state of the VM is a fundamental operation supported by the language. Only truly transient things like network connections require special effort. There are some echoes of this with things like Lua's Pluto/Eris, or serializable continuations in other languages (eg: Perl's Continuity). It's just such a pain to thoroughly handle that sort of stuff without language-level support. And doing a "good enough" approach with some rough edges is usually shippable, so it's hard to build a critical mass of demand for such support. And even if there was, it's very hard to add it to a language/framework/etc that wasn't designed for it to begin with. I've had a decent experience with 'struct string' style approaches, like Lua's string.pack() or Perl's pack()[3]. It's a little brittle, but extremely straightforward and "not framework-y," which suits me. But it leaves out things like program execution state; it's just for plain data. [1] https://en.wikipedia.org/wiki/Smalltalk#Image-based_persistence https://en.wikipedia.org/wiki/Smalltalk#Image-based_persiste... [2] example of using this serializable statefulness for serving web apps: https://en.wikipedia.org/wiki/Seaside_(software) https://en.wikipedia.org/wiki/Seaside_(software) [3] https://perldoc.perl.org/functions/pack https://perldoc.perl.org/functions/pack
- nox101 2y ago
  I don't, in fact I mostly hate serializations systems. IMO they lead to extremely long load times. It might be more work to put the data that actually needs to be saved into some binary blob but it's the difference between a game that loads instantly and a game (like Source games) that takes 10-20 infuriating seconds per level every time you die.
- alexvitkov 2y ago
  A full memory dump for a game will nowadays often be multiple gigabytes, that's a non-starter. Even back in the day, Game Maker had a function to dump the state to disk that was intended for game saves. It sucked - turns out there's a bunch of state that you don't want in your savefile - keybinds, settings, even most game state actually. Save state should be opt-in, not opt-out, and on top of that a VM/memory dump makes it a very big pain to opt-out.
kogir 2y ago
This seems to cover many common pain points, but I’ve written my fair share of .NET serializers and for anything I build now I’d just use protocol buffers. Robust support, handles versioning pretty well, and works cross platform. I’d like to know their reasons for making yet another serializer vs just using pb or thrift.
- jolexxa 2y ago
  This is a good point. I don't think anyone wakes up wanting to make a new serializer. At this point, I was already pretty deep into making and releasing tools for my game projects so doing this didn't seem like such a stretch (although it actually ended up being one of the hardest things I've ever done). A lot of small to mid-size games (which are the focus of the tools I provide) want to save data into JSON, whether it is to be mod-friendly or just somewhat human-friendly to the developer while working on the game. Not familiar with Thrift, but PB is obviously for binary data and has a focus on compactness and performance, which isn't the primary concern on my list of priorities for a serialization system. My primary concern for a serialization system is refactor-friendliness. I want to be able to rework type hierarchies without breaking existing save files, or get as close to that as possible. I suppose you could say I'm only really introducing "half" of a serialization system: the heavy lifting is being split between the introspection generator (for writing metadata at compile time via source generation) and System.Text.Json (which handles a lot of the runtime logic for serializing/deserializing things).
LeonB 2y ago
In my personal projects I’ve been using variations on the same simple code for saving/loadings objects for a decade or so, and have very few problems. The heart of the code is this interface - public interface IStashy<K> { void Save<T>(T t, K id); T Load<T>(K id); IEnumerable<T> LoadAll<T>(); void Delete<T>(K id); K GetNewId<T>(); } And implementations of that are very stable over time. Objects get serialized as json and stored in a folder named after their type. There’s a small number of gotchas, for which I have well known work arounds: - I generally won’t remove a property, but mark it as obsolete and stop using it. - If I’ve added a new Boolean property, I’d tend to name it such that it defaults to false, or if it must default to true, have it stored in a nullable boolean, and if it loads as null (from an older instance of the type), set it to the default. - some convenient types I want to use (as properties) are not serializable, so before saving I’ll copy their data into a serializable type, like an array of key values, then on loading rehydrate that to a dictionary. (I guess this is a harsh performance penalty if you’re doing a lot of it in a game)
- nonethewiser 2y ago
  > I generally won’t remove a property, but mark it as obsolete and stop using it. Presumably because loading will break? > - If I’ve added a new Boolean property, I’d tend to name it such that it defaults to false, or if it must default to true, have it stored in a nullable boolean, and if it loads as null (from an older instance of the type), set it to the default. Why?
  - eropple 2y ago
    `default(Boolean)` is false, so you can load an old object and it'll substitute the default, rather than having to throw an error on a missing property. You could do the same with, say, a new Int32 field, so long as it should default to zero. Similarly, `default(Nullable<Boolean>)` is (wait for it) null, so you can do "oldVal ?? true".
    - LeonB 2y ago
      (I meant to respond to the gp comment here, soz. I agree with everything the parent comment says.) > Presumably because loading will break? I think the object will still load ok, I’m not sure if it would break because it’s been so long since I was in a scenario where I wanted to really delete a property. Normally when I make it obsolete there is also some new property or properties that have replaced it. When loading the object, if the (now obsolete) property is not null, I translate its value into the new property/ies (I.e., “migrate” it into the new properties), then null out the old property value, so that the migration only happens that one time. I guess using something allegedly “simple”, over a long time, only appears simple because you will slowly internalise any idioms you’re using, and they don’t take much thought anymore. Looking back — I’ve used this pattern for over 20 years, across various platforms. I’ve had a backing source that is anything from xml files to json to sqlite to in memory (for rapid tests) in a few languages. Things that seem natural or intuitive (to me) at this point are just habits that are rusted on, whether good or bad. Sometimes I start building fast indexing systems on top of it, or archiving systems or record versioning… and the better tool would be to switch to a db or to a more full featured key value store. But it’s such a lot of fun!
isthiseasymode 2y ago
Naive question: is there a reason why SQLite wouldn’t work for something like this?
- nonethewiser 2y ago
  I wonder the same thing. Perhaps less portable? IE can't package that up in a binary (I have absolutely no idea just spitballing)? And very cursory search suggests maybe there is nothing to that guess: https://www.reddit.com/r/golang/comments/tqffv2/packaging_an_sql_database_in_a_go_binary/ https://www.reddit.com/r/golang/comments/tqffv2/packaging_an... It's an interesting question because I've run into some datascientists that were so used to working in memory with dataframes and similar that they moved mountains to do things like de-duplicate csv's in memory (that they couldn't all fit in at once) where-as they could have done so trivially with sqlite.
- jameshart 2y ago
  Well you still need to solve for what happens when a new version of your app (maybe with a new embedded version of SQLite) loads up an old data file saved by an old version of your app. The old version might not contain all the tables you need, and the ones it has may not have the columns you expect. So you need to run some data migrations on the database. Now you no longer have a serialization problem but instead you have a schema versioning problem.
  - cjonas 2y ago
    Could solve this with a migration framework (I'm sure there is something for sqlLite). I've also done something similar with object/document storage. Store the version of the schema in the record and write a map function for each version from the previous.
  - flohofwoe 2y ago
    > instead you have a schema versioning problem That same versioning problem also exists with other approaches. Having a versioned schema of the savegame format around for version migrations is generally a good idea.
  - 2y ago
flohofwoe 2y ago
If I learned one important lesson from writing savegame systems: don't directly serialize your entire game state on the "game object level" (e.g. don't create a savegame by running a serializer over your game object soup), instead decouple the saved data from your game logic internals, have a clear boundary between your game logic and the savegame-system, keep the saved state as minimal as possible, and reconstruct or default-initialize the rest of the data after loading saved state. With that approach, a language-assisted serialization system also looses a lot of its appeal IMHO (although it can still be useful of course for describing the separate savegame data format). Also: resist the architecture astronaut in you, especially savegame systems are a honey trap for overengineering ;)
- smartis2812 2y ago
  [flagged]
  - whoknowsidont 2y ago
    Please don't put comments like this in threads. There is a button for it.
    - mn99 2y ago
      Spot on
    - hellodang_ 2y ago
      [dead]
- voxic11 2y ago
  If your game engine is built on a data-first architecture like ECS then it can be pretty trivial to directly serialize your game state. I have had good luck with this using bitECS https://github.com/NateTheGreatt/bitECS/blob/master/docs/INTRO.md#-serialization https://github.com/NateTheGreatt/bitECS/blob/master/docs/INT...
  - 2y ago
scotty79 2y ago
How this subject is approached in ECS? Just add Save component that knows how to serialize to all objects that need to be saved and a system that dumps alive objects into a file?
wheybags 2y ago
In my experience, the pain of dealing with changes outweighs the pain of dealing with boilerplate, so it's better to explicitly write out save and load functions manually than rely on reflection. Also means you can do stuff like if(version<x) { load old thing + migrate} else {load new thing} very easily. And it's just code, not magic.
- jayd16 2y ago
  Something like DB schema upgrading would be good but if you have versions you should be able to do that just fine. Reflection and changes are not at odds.
- jolexxa 2y ago
  That's essentially what this system does — it identifies the models and their properties that you've marked as serializable at build-time using source generation, and then allows you to provide a type resolver and converter to System.Text.Json that lets you make upgrade-able models with logic like you just described. The assist from the source generation helps reduce some of the boilerplate you need, but there's no escaping it ultimately.
penetrarthur 2y ago
There's a very good MessagePack serialization library for C#. I've used it in many of the games I worked on. https://github.com/MessagePack-CSharp/MessagePack-CSharp https://github.com/MessagePack-CSharp/MessagePack-CSharp
jayd16 2y ago
I want to like this because it seems well done but I kind of grimace instead. It's not the library's fault. Game engines have some form of serialization already (most of what a game engine does is load a serialized game state into memory, imo). I've found its usually better to try to leverage those systems so you're not building multiple models objects and doing conversions between game engine and serialized types. Engines often do a lot of (design)work to load things directly into memory in such a way that the game engine can use the inflated object immediately without a lot of parsing. It's nice to try to leverage that. Moreover less plugins is less complexity in the build process etc etc. Those desires give me pause when looking at serialization plugins in the context of game engines. Howeve, it's also not entirely feasible to only use the core engine systems in all cases. Often what's available at runtime for a game engine isn't always the same as build time. You might need to read this data outside of the engine and then you're really out of luck...life's so complicated.

133 points by jolexxa 2y ago

Madmallard 2y ago

RunUO has an implementation of this and it's like 25 years old but still worked really well

mentos 2y ago
Wow clicked into the thread to see if anyone might mention RunUO :) it’s the only exposure I’ve had to serialization in C# I always wondered how it ranked compared to other approaches.
- w-ll 2y ago
  As someone that also fell in love with C# with RunUO. I never actually looked at the Serialization at the time. Need to spend some time in RunUO or the fork soon. https://github.com/runuo/runuo/blob/master/Server/Serialization.cs https://github.com/runuo/runuo/blob/master/Server/Serializat...
sanex 2y ago
Ultima online solved all our problems 25 years ago.
jolexxa 2y ago
I really like this implementation, but it's probably worth mentioning here that RunUO and other tools like it are solving the problem at a layer of abstraction beneath what I was introducing here. The serialization system I am providing here actually leverages System.Text.Json for reading and writing data — it's more concerned with helping you represent version-able, upgrade-able data models that are also compatible with the hierarchical state machine implementation I use for managing game state.

interroboink 2y ago

Sometimes, when battling these issues, I wish the Smalltalk-style approach[1][2] was more popular/feasible. Basically, saving the entire state of the VM is a fundamental operation supported by the language. Only truly transient things like network connections require special effort. There are some echoes of this with things like Lua's Pluto/Eris, or serializable continuations in other languages (eg: Perl's Continuity). It's just such a pain to thoroughly handle that sort of stuff without language-level support. And doing a "good enough" approach with some rough edges is usually shippable, so it's hard to build a critical mass of demand for such support. And even if there was, it's very hard to add it to a language/framework/etc that wasn't designed for it to begin with. I've had a decent experience with 'struct string' style approaches, like Lua's string.pack() or Perl's pack()[3]. It's a little brittle, but extremely straightforward and "not framework-y," which suits me. But it leaves out things like program execution state; it's just for plain data. [1] https://en.wikipedia.org/wiki/Smalltalk#Image-based_persistence https://en.wikipedia.org/wiki/Smalltalk#Image-based_persiste... [2] example of using this serializable statefulness for serving web apps: https://en.wikipedia.org/wiki/Seaside_(software) https://en.wikipedia.org/wiki/Seaside_(software) [3] https://perldoc.perl.org/functions/pack https://perldoc.perl.org/functions/pack

nox101 2y ago
I don't, in fact I mostly hate serializations systems. IMO they lead to extremely long load times. It might be more work to put the data that actually needs to be saved into some binary blob but it's the difference between a game that loads instantly and a game (like Source games) that takes 10-20 infuriating seconds per level every time you die.
alexvitkov 2y ago
A full memory dump for a game will nowadays often be multiple gigabytes, that's a non-starter. Even back in the day, Game Maker had a function to dump the state to disk that was intended for game saves. It sucked - turns out there's a bunch of state that you don't want in your savefile - keybinds, settings, even most game state actually. Save state should be opt-in, not opt-out, and on top of that a VM/memory dump makes it a very big pain to opt-out.

kogir 2y ago

This seems to cover many common pain points, but I’ve written my fair share of .NET serializers and for anything I build now I’d just use protocol buffers. Robust support, handles versioning pretty well, and works cross platform. I’d like to know their reasons for making yet another serializer vs just using pb or thrift.

jolexxa 2y ago
This is a good point. I don't think anyone wakes up wanting to make a new serializer. At this point, I was already pretty deep into making and releasing tools for my game projects so doing this didn't seem like such a stretch (although it actually ended up being one of the hardest things I've ever done). A lot of small to mid-size games (which are the focus of the tools I provide) want to save data into JSON, whether it is to be mod-friendly or just somewhat human-friendly to the developer while working on the game. Not familiar with Thrift, but PB is obviously for binary data and has a focus on compactness and performance, which isn't the primary concern on my list of priorities for a serialization system. My primary concern for a serialization system is refactor-friendliness. I want to be able to rework type hierarchies without breaking existing save files, or get as close to that as possible. I suppose you could say I'm only really introducing "half" of a serialization system: the heavy lifting is being split between the introspection generator (for writing metadata at compile time via source generation) and System.Text.Json (which handles a lot of the runtime logic for serializing/deserializing things).

LeonB 2y ago

In my personal projects I’ve been using variations on the same simple code for saving/loadings objects for a decade or so, and have very few problems. The heart of the code is this interface - public interface IStashy<K> { void Save<T>(T t, K id); T Load<T>(K id); IEnumerable<T> LoadAll<T>(); void Delete<T>(K id); K GetNewId<T>(); } And implementations of that are very stable over time. Objects get serialized as json and stored in a folder named after their type. There’s a small number of gotchas, for which I have well known work arounds: - I generally won’t remove a property, but mark it as obsolete and stop using it. - If I’ve added a new Boolean property, I’d tend to name it such that it defaults to false, or if it must default to true, have it stored in a nullable boolean, and if it loads as null (from an older instance of the type), set it to the default. - some convenient types I want to use (as properties) are not serializable, so before saving I’ll copy their data into a serializable type, like an array of key values, then on loading rehydrate that to a dictionary. (I guess this is a harsh performance penalty if you’re doing a lot of it in a game)

nonethewiser 2y ago
> I generally won’t remove a property, but mark it as obsolete and stop using it. Presumably because loading will break? > - If I’ve added a new Boolean property, I’d tend to name it such that it defaults to false, or if it must default to true, have it stored in a nullable boolean, and if it loads as null (from an older instance of the type), set it to the default. Why?
- eropple 2y ago
  `default(Boolean)` is false, so you can load an old object and it'll substitute the default, rather than having to throw an error on a missing property. You could do the same with, say, a new Int32 field, so long as it should default to zero. Similarly, `default(Nullable<Boolean>)` is (wait for it) null, so you can do "oldVal ?? true".
  - LeonB 2y ago
    (I meant to respond to the gp comment here, soz. I agree with everything the parent comment says.) > Presumably because loading will break? I think the object will still load ok, I’m not sure if it would break because it’s been so long since I was in a scenario where I wanted to really delete a property. Normally when I make it obsolete there is also some new property or properties that have replaced it. When loading the object, if the (now obsolete) property is not null, I translate its value into the new property/ies (I.e., “migrate” it into the new properties), then null out the old property value, so that the migration only happens that one time. I guess using something allegedly “simple”, over a long time, only appears simple because you will slowly internalise any idioms you’re using, and they don’t take much thought anymore. Looking back — I’ve used this pattern for over 20 years, across various platforms. I’ve had a backing source that is anything from xml files to json to sqlite to in memory (for rapid tests) in a few languages. Things that seem natural or intuitive (to me) at this point are just habits that are rusted on, whether good or bad. Sometimes I start building fast indexing systems on top of it, or archiving systems or record versioning… and the better tool would be to switch to a db or to a more full featured key value store. But it’s such a lot of fun!

isthiseasymode 2y ago

Naive question: is there a reason why SQLite wouldn’t work for something like this?

nonethewiser 2y ago
I wonder the same thing. Perhaps less portable? IE can't package that up in a binary (I have absolutely no idea just spitballing)? And very cursory search suggests maybe there is nothing to that guess: https://www.reddit.com/r/golang/comments/tqffv2/packaging_an_sql_database_in_a_go_binary/ https://www.reddit.com/r/golang/comments/tqffv2/packaging_an... It's an interesting question because I've run into some datascientists that were so used to working in memory with dataframes and similar that they moved mountains to do things like de-duplicate csv's in memory (that they couldn't all fit in at once) where-as they could have done so trivially with sqlite.
jameshart 2y ago
Well you still need to solve for what happens when a new version of your app (maybe with a new embedded version of SQLite) loads up an old data file saved by an old version of your app. The old version might not contain all the tables you need, and the ones it has may not have the columns you expect. So you need to run some data migrations on the database. Now you no longer have a serialization problem but instead you have a schema versioning problem.
- cjonas 2y ago
  Could solve this with a migration framework (I'm sure there is something for sqlLite). I've also done something similar with object/document storage. Store the version of the schema in the record and write a map function for each version from the previous.
- flohofwoe 2y ago
  > instead you have a schema versioning problem That same versioning problem also exists with other approaches. Having a versioned schema of the savegame format around for version migrations is generally a good idea.
- 2y ago

flohofwoe 2y ago

If I learned one important lesson from writing savegame systems: don't directly serialize your entire game state on the "game object level" (e.g. don't create a savegame by running a serializer over your game object soup), instead decouple the saved data from your game logic internals, have a clear boundary between your game logic and the savegame-system, keep the saved state as minimal as possible, and reconstruct or default-initialize the rest of the data after loading saved state. With that approach, a language-assisted serialization system also looses a lot of its appeal IMHO (although it can still be useful of course for describing the separate savegame data format). Also: resist the architecture astronaut in you, especially savegame systems are a honey trap for overengineering ;)

smartis2812 2y ago
[flagged]
- whoknowsidont 2y ago
  Please don't put comments like this in threads. There is a button for it.
  - mn99 2y ago
    Spot on
  - hellodang_ 2y ago
    [dead]
voxic11 2y ago
If your game engine is built on a data-first architecture like ECS then it can be pretty trivial to directly serialize your game state. I have had good luck with this using bitECS https://github.com/NateTheGreatt/bitECS/blob/master/docs/INTRO.md#-serialization https://github.com/NateTheGreatt/bitECS/blob/master/docs/INT...
- 2y ago

scotty79 2y ago

How this subject is approached in ECS? Just add Save component that knows how to serialize to all objects that need to be saved and a system that dumps alive objects into a file?

wheybags 2y ago

In my experience, the pain of dealing with changes outweighs the pain of dealing with boilerplate, so it's better to explicitly write out save and load functions manually than rely on reflection. Also means you can do stuff like if(version<x) { load old thing + migrate} else {load new thing} very easily. And it's just code, not magic.

jayd16 2y ago
Something like DB schema upgrading would be good but if you have versions you should be able to do that just fine. Reflection and changes are not at odds.
jolexxa 2y ago
That's essentially what this system does — it identifies the models and their properties that you've marked as serializable at build-time using source generation, and then allows you to provide a type resolver and converter to System.Text.Json that lets you make upgrade-able models with logic like you just described. The assist from the source generation helps reduce some of the boilerplate you need, but there's no escaping it ultimately.

penetrarthur 2y ago

There's a very good MessagePack serialization library for C#. I've used it in many of the games I worked on. https://github.com/MessagePack-CSharp/MessagePack-CSharp https://github.com/MessagePack-CSharp/MessagePack-CSharp

jayd16 2y ago

I want to like this because it seems well done but I kind of grimace instead. It's not the library's fault. Game engines have some form of serialization already (most of what a game engine does is load a serialized game state into memory, imo). I've found its usually better to try to leverage those systems so you're not building multiple models objects and doing conversions between game engine and serialized types. Engines often do a lot of (design)work to load things directly into memory in such a way that the game engine can use the inflated object immediately without a lot of parsing. It's nice to try to leverage that. Moreover less plugins is less complexity in the build process etc etc. Those desires give me pause when looking at serialization plugins in the context of game engines. Howeve, it's also not entirely feasible to only use the core engine systems in all cases. Often what's available at runtime for a game engine isn't always the same as build time. You might need to read this data outside of the engine and then you're really out of luck...life's so complicated.