8 ms·

On the game I'm currently working on, it's built very heavily around Lua. So for the save system, we simply fill a large Lua table, and then write that to disk,

On the game I'm currently working on, it's built very heavily around Lua. So for the save system, we simply fill a large Lua table, and then write that to disk, as Lua code. The 'save' file then simply becomes a Lua file that can be read directly into Lua. This is absolutely amazing for debugging purposes. Also you never have to worry about corrupt save files or anything of it's ilk. Development is easier, diagnosing problems is easier, and using a programmatic data structure on the backend means that you can pretty much keep things clean and forward compatible with ease. (Oh, also being able to debug by altering the save file in any way you want is a godsend).

frik 13y ago
So, it's similar to JSON (JavaScript), but valid Lua syntax. local t = {} t = {["foo"] = "bar", [123] = 456} t.foo2 = "bar2"
hootener 13y ago
I've had to write output save file formats for various projects on several occasions, and it never occurred to me to take this approach. Thanks for sharing this, it's one of those ideas that (to me) seems so brilliant in its simplicity that I probably would've never thought of it. Any hiccups in the day-to-day work using this approach? I'm just trying to get a better idea of the workflow since I'm very seriously considering applying it to my next project.
- mrcharles 13y ago
  The biggest hiccup is almost a literal one; serializing large lua structures and then writing them to disk can take a lot of time. But this can largely be mitigated by just saving compiled lua instead of text lua.
Touche 13y ago
That's how people are going to cheat at your game.
- saucetenuto 13y ago
  It's only cheating if the developer disapproves.
- 6d0debc071 13y ago
  Hash the information and include the hash in the file. If the hash and the contents don't match when you try to load it, you can refuse it. If not loading things is important to you, mind.
  - scott_karana 13y ago
    What's to stop people from re-hashing the changed file? :)
    - roryokane 13y ago
      You could hash the file contents plus a salt that is contained within the application binary. Most people wouldn’t know how to extract the salt from the binary so they could get the hash right. Though I guess if people are determined enough to hack your game, one hacker might just publish the salt or a small program to rehash files for you. If a user has enough time on their hands, there’s nothing you can do to stop them from running your software in a VM with a debugger and finding out its secrets; the best you can do is making that hard enough that people won’t bother.
    - 6d0debc071 13y ago
      Lack of knowledge, lack of interest. If you wanted more security, you could keep a secret that you don't include in the saves but do include when you calculate the hash, so that anyone who doesn't have the key is going to get the wrong answer. That's about as far as I'd consider going for relatively trivial data like save games. Though that's, in principle, discoverable if someone's sufficiently interested. After that point, it becomes much simpler for someone to watch the memory associated with your program and extract/alter the values there. (Programs to do that to games are generally called Trainers.) That's not a complicated thing to do unless someone's tried to stop you doing it. There are some techniques to provide some degree of security there. Changing where in memory you place your information each time springs to mind, thus making it more difficult for people to find out where the values are and then share their locations. However, even that's not perfect. Depending on how sure you want to be that no-one's going to alter the values, you're potentially looking at requiring very deep knowledge of security there. After that point the next easiest target may be the program file itself. That said, if you want to get around that sort of problem and you're really serious about it, then running your encryption in an environment that hostile may be making things more difficult than they need to be. You might use a trusted platform module, to try to make the environment you were in less hostile, if one were present on the user's machine. But, honestly, I'd want the information not to be stored or calculated on the user's machine if it were that valuable. Have the user's end be the input, encrypt their signals with your public key, and do the calculations that you needed to be sure of remotely. Though then the user has to trust you. I wouldn't usually advocate that my users trust me that much - not unless we were dealing with a situation where the information we were talking about was entangled with others in some way such that a reasonable argument could be made that they didn't own it, and I was just the best common arbiter I could think of. -she shrugs awkwardly- You can get yourself into a situation where it's probable that the amount of effort someone would have to invest is vastly greater than the likely value of the information fairly easily. But ultimately it's a question of how expensive you want to make things and what that's worth to you. Against a sufficiently dedicated adversary, with a sufficiently valuable target, there are so many unknowns in computer security that I wouldn't even be sure that storing the data on your server would be sufficient ^^;
dbaupp 13y ago
Why does using a Lua-basef format stop the files being corrupted?
- politician 13y ago
  Maybe he means that he doesn't have to deal with bugs in a custom binary serializer.
- Igglyboo 13y ago
  I don't think it does. I think he meant that if a save became corrupted it wouldn't do so silently, it would violently crash the game because of a syntax error.
- Zecc 13y ago
  It doesn't. But it makes them much easier to fix. Edit: Igglyboo has a point too.
- mrcharles 13y ago
  It can only become corrupted by external factors; a lot of games I've worked on, in-game bugs could lead to corrupted saves being written out to disk. Since in this case we are just serializing lua data, unless the serializer itself has a bug, it will always write out correctly, and any issues become issues of game logic rather than anything else.
Tanner 13y ago
That Lua was originally designed as a configuration language becomes really clear when you start doing things like this. Having my code and configuration being separate but equal was really a paradigm shift for me. Also, the Tiled Map Editor exports directly to Lua.
mattgreenrocks 13y ago
Preach it. My favorite persistence code: stuff that has nothing to do with SQL/NoSQL. I leaned heavily on Python's pickle module for serializing a few thousand entities to disk a few years ago. By streaming them to the application at startup time, it remained plenty fast for all datasets it'd encounter. I intended to replace it with SQLite one day, but I never had to. I could just keep them all in memory. I'd probably choose something a bit safer now, but it was hard to beat the simplicity.
- 3pt14159 13y ago
  I used to do that, but pickle bit me once. I think it changes between versioning or something. I had to start the statistical model from scratch.
  - mattgreenrocks 13y ago
    Now that you mention it, I remember running into something like it. It's a big issue. That module had a bunch of scar code to migrate entities as they came up from disk.
  - rcxdude 13y ago
    Yeah, I wouldn't use pickle for anything where backwards or forwards compatibility is important. It is however very convenient for 'I want to send/store this data for a bit' tasks.
TheEzEzz 13y ago
I did this with C# in my last game. All the map/object editors output C# code on save, which was then included in the compiled code on the next build. The beauty is that your "data" files get automatically updated when you refactor your regular code! On top of that loading is faster, because you don't need to worry about fetching a file and parsing it, the whole thing is just compiled code embedded in your executable.
JackC 13y ago
You probably know this, but remember that storing user data as code is a place where you (general "you") have to think very carefully about security. Is there any way that arbitrary code in the file could compromise the user's system? If so, does the user know to treat these data files as executables? Is there any way someone untrusted could ever edit the file without the user's knowledge? Even in combination with other programs the user might be running? Are you sure about all of that? Maybe Lua in particular is sandboxed so that's not a problem (beats me), but in general this is an area where safe high-level languages can all of a sudden turn dangerous. Personally I would rarely find it worth it.
- Guvante 13y ago
  Typically sandboxing is stage one of any lua implementation. You don't need raw IO access and rarely need to print to the screen for instance.
- TelmoMenezes 13y ago
  This is a good point, but I feel that discouraging this type of approach is not the way to go. I apologise in advance for ranting... I hope this is not too off-topic, but instead a "zoom out" on the issue. This touches on something deep and wrong about how we use computers these days. Computers are really good at being computers, and the amplification of intellectual capabilities they afford is tremendous, but this is reserved for a limited few that were persistent enough and learned enough to rediscover the raw computer buried underneath, and what it can do. For example, I dream of a world where everything communicates through s-expressions, all code is data and all data is code. Everything understandable all the way down. Imagine what people from all fields could create with this level of plug-ability and inter-operability. We had a whiff of that with the web so far, but it could be so much more powerful, so much simpler, so much more elegant. All the computer science is there, it's just a social problem. I understand the security issues, but surely limiting the potential of computers is not the solution. There has to be a better way.
  - judk 13y ago
    When my conputer has the potential to erase my bank account, I want to limit its potential.
balls187 13y ago
I did something similar, but used JSON instead (pretty trivial to (de)/serialize LUA tables to JSON. This made it easy to send data to the server, and inspect with standard tools as well.
agumonkey 13y ago
IIRC that's how Office and Photoshop file format started. I think it's a nightmare for compatibility in the end.
ilovecookies 13y ago
While one could say that this is about savefiles for games, I would say it's implications could more be about savefiles for software projects. If you are building the game in LUA, of course LUA is going to be the preferred way to save your game in since you are already using LUA objects and interpreting files in that language will be easy to integrate. If you ever used maven xml configs, java object marshalling or c# xml you would understand the pains of using xml as a file format for software projects and data representation. You have to find a solution that is language agnostic, neither LUA or JSON is.
seivan 13y ago
This sounds a lot like NSCoding for Objective-C (Cocoa). Though you'd still have to define the types/classes and name for each property you want to save. But you could technically save it in a big blob, and then read it into memory as you resume. Could persist to disk as a binary, sql or a plist (xml). I guess the only downside is, that if you got a lot of composite classes all with their own properties and associations (say a graph), there's a lot of manual work to be done.

frik 13y ago

So, it's similar to JSON (JavaScript), but valid Lua syntax. local t = {} t = {["foo"] = "bar", [123] = 456} t.foo2 = "bar2"

hootener 13y ago

I've had to write output save file formats for various projects on several occasions, and it never occurred to me to take this approach. Thanks for sharing this, it's one of those ideas that (to me) seems so brilliant in its simplicity that I probably would've never thought of it. Any hiccups in the day-to-day work using this approach? I'm just trying to get a better idea of the workflow since I'm very seriously considering applying it to my next project.

mrcharles 13y ago
The biggest hiccup is almost a literal one; serializing large lua structures and then writing them to disk can take a lot of time. But this can largely be mitigated by just saving compiled lua instead of text lua.

Touche 13y ago

That's how people are going to cheat at your game.

saucetenuto 13y ago
It's only cheating if the developer disapproves.
6d0debc071 13y ago
Hash the information and include the hash in the file. If the hash and the contents don't match when you try to load it, you can refuse it. If not loading things is important to you, mind.
- scott_karana 13y ago
  What's to stop people from re-hashing the changed file? :)
  - roryokane 13y ago
    You could hash the file contents plus a salt that is contained within the application binary. Most people wouldn’t know how to extract the salt from the binary so they could get the hash right. Though I guess if people are determined enough to hack your game, one hacker might just publish the salt or a small program to rehash files for you. If a user has enough time on their hands, there’s nothing you can do to stop them from running your software in a VM with a debugger and finding out its secrets; the best you can do is making that hard enough that people won’t bother.
  - 6d0debc071 13y ago
    Lack of knowledge, lack of interest. If you wanted more security, you could keep a secret that you don't include in the saves but do include when you calculate the hash, so that anyone who doesn't have the key is going to get the wrong answer. That's about as far as I'd consider going for relatively trivial data like save games. Though that's, in principle, discoverable if someone's sufficiently interested. After that point, it becomes much simpler for someone to watch the memory associated with your program and extract/alter the values there. (Programs to do that to games are generally called Trainers.) That's not a complicated thing to do unless someone's tried to stop you doing it. There are some techniques to provide some degree of security there. Changing where in memory you place your information each time springs to mind, thus making it more difficult for people to find out where the values are and then share their locations. However, even that's not perfect. Depending on how sure you want to be that no-one's going to alter the values, you're potentially looking at requiring very deep knowledge of security there. After that point the next easiest target may be the program file itself. That said, if you want to get around that sort of problem and you're really serious about it, then running your encryption in an environment that hostile may be making things more difficult than they need to be. You might use a trusted platform module, to try to make the environment you were in less hostile, if one were present on the user's machine. But, honestly, I'd want the information not to be stored or calculated on the user's machine if it were that valuable. Have the user's end be the input, encrypt their signals with your public key, and do the calculations that you needed to be sure of remotely. Though then the user has to trust you. I wouldn't usually advocate that my users trust me that much - not unless we were dealing with a situation where the information we were talking about was entangled with others in some way such that a reasonable argument could be made that they didn't own it, and I was just the best common arbiter I could think of. -she shrugs awkwardly- You can get yourself into a situation where it's probable that the amount of effort someone would have to invest is vastly greater than the likely value of the information fairly easily. But ultimately it's a question of how expensive you want to make things and what that's worth to you. Against a sufficiently dedicated adversary, with a sufficiently valuable target, there are so many unknowns in computer security that I wouldn't even be sure that storing the data on your server would be sufficient ^^;

dbaupp 13y ago

Why does using a Lua-basef format stop the files being corrupted?

politician 13y ago
Maybe he means that he doesn't have to deal with bugs in a custom binary serializer.
Igglyboo 13y ago
I don't think it does. I think he meant that if a save became corrupted it wouldn't do so silently, it would violently crash the game because of a syntax error.
Zecc 13y ago
It doesn't. But it makes them much easier to fix. Edit: Igglyboo has a point too.
mrcharles 13y ago
It can only become corrupted by external factors; a lot of games I've worked on, in-game bugs could lead to corrupted saves being written out to disk. Since in this case we are just serializing lua data, unless the serializer itself has a bug, it will always write out correctly, and any issues become issues of game logic rather than anything else.

Tanner 13y ago

That Lua was originally designed as a configuration language becomes really clear when you start doing things like this. Having my code and configuration being separate but equal was really a paradigm shift for me. Also, the Tiled Map Editor exports directly to Lua.

mattgreenrocks 13y ago

Preach it. My favorite persistence code: stuff that has nothing to do with SQL/NoSQL. I leaned heavily on Python's pickle module for serializing a few thousand entities to disk a few years ago. By streaming them to the application at startup time, it remained plenty fast for all datasets it'd encounter. I intended to replace it with SQLite one day, but I never had to. I could just keep them all in memory. I'd probably choose something a bit safer now, but it was hard to beat the simplicity.

3pt14159 13y ago
I used to do that, but pickle bit me once. I think it changes between versioning or something. I had to start the statistical model from scratch.
- mattgreenrocks 13y ago
  Now that you mention it, I remember running into something like it. It's a big issue. That module had a bunch of scar code to migrate entities as they came up from disk.
- rcxdude 13y ago
  Yeah, I wouldn't use pickle for anything where backwards or forwards compatibility is important. It is however very convenient for 'I want to send/store this data for a bit' tasks.

TheEzEzz 13y ago

I did this with C# in my last game. All the map/object editors output C# code on save, which was then included in the compiled code on the next build. The beauty is that your "data" files get automatically updated when you refactor your regular code! On top of that loading is faster, because you don't need to worry about fetching a file and parsing it, the whole thing is just compiled code embedded in your executable.

JackC 13y ago

You probably know this, but remember that storing user data as code is a place where you (general "you") have to think very carefully about security. Is there any way that arbitrary code in the file could compromise the user's system? If so, does the user know to treat these data files as executables? Is there any way someone untrusted could ever edit the file without the user's knowledge? Even in combination with other programs the user might be running? Are you sure about all of that? Maybe Lua in particular is sandboxed so that's not a problem (beats me), but in general this is an area where safe high-level languages can all of a sudden turn dangerous. Personally I would rarely find it worth it.

Guvante 13y ago
Typically sandboxing is stage one of any lua implementation. You don't need raw IO access and rarely need to print to the screen for instance.
TelmoMenezes 13y ago
This is a good point, but I feel that discouraging this type of approach is not the way to go. I apologise in advance for ranting... I hope this is not too off-topic, but instead a "zoom out" on the issue. This touches on something deep and wrong about how we use computers these days. Computers are really good at being computers, and the amplification of intellectual capabilities they afford is tremendous, but this is reserved for a limited few that were persistent enough and learned enough to rediscover the raw computer buried underneath, and what it can do. For example, I dream of a world where everything communicates through s-expressions, all code is data and all data is code. Everything understandable all the way down. Imagine what people from all fields could create with this level of plug-ability and inter-operability. We had a whiff of that with the web so far, but it could be so much more powerful, so much simpler, so much more elegant. All the computer science is there, it's just a social problem. I understand the security issues, but surely limiting the potential of computers is not the solution. There has to be a better way.
- judk 13y ago
  When my conputer has the potential to erase my bank account, I want to limit its potential.

balls187 13y ago

I did something similar, but used JSON instead (pretty trivial to (de)/serialize LUA tables to JSON. This made it easy to send data to the server, and inspect with standard tools as well.

agumonkey 13y ago

IIRC that's how Office and Photoshop file format started. I think it's a nightmare for compatibility in the end.

ilovecookies 13y ago

While one could say that this is about savefiles for games, I would say it's implications could more be about savefiles for software projects. If you are building the game in LUA, of course LUA is going to be the preferred way to save your game in since you are already using LUA objects and interpreting files in that language will be easy to integrate. If you ever used maven xml configs, java object marshalling or c# xml you would understand the pains of using xml as a file format for software projects and data representation. You have to find a solution that is language agnostic, neither LUA or JSON is.

seivan 13y ago

This sounds a lot like NSCoding for Objective-C (Cocoa). Though you'd still have to define the types/classes and name for each property you want to save. But you could technically save it in a big blob, and then read it into memory as you resume. Could persist to disk as a binary, sql or a plist (xml). I guess the only downside is, that if you got a lot of composite classes all with their own properties and associations (say a graph), there's a lot of manual work to be done.