6 ms·

A file format uncracked for 20 years

299 points by todsacerdoti 1y ago

rawling 1y ago
Weird that the one of these with no interaction got bumped rather than https://news.ycombinator.com/item?id=45842851 https://news.ycombinator.com/item?id=45842851
- hcs 1y ago
  I saw the other when it was first posted so it must have made the front page (or second page)
Dwedit 1y ago
Lowercase 'x' is always your dead giveaway that it's ZLIB.
LunicLynx 1y ago
The Xbox had strong requirements for loading times. This is probably a linear (lin) record of how the data was loaded unoptimized from the disk. And just written to a file. So in this file seek doesn’t do anything because seek kills the requirement of 45 sec per loading screen. Instead the logic is as follows: check if a .lin file exists. Yes: open a handle to it and only read from it with fread, what ever currently is at the current file position. No: while reading any file write the read bytes to a .lin file in the order they are read. This gives a highlyy optimized .lin file which can be read from the disk into memory, without creating a better dedicated loading mechanism. So if your really would like to unpack this. The first file being read is most likely the key, as it dictates what comes next. If it is a level model, then the position of the player in it might affect which other files to load etc. In short it’s not a file format in the classical sense, it’s a linear stream of game data.
- landr0id 1y ago
  >This is probably a linear (lin) record of how the data was loaded unoptimized from the disk. Yes, it's buried deep in the details but it's basically just every byte read being written in a linear stream to an output file. I don't know which stage of grief this is, but since I wrote this blog post I've now ported my IDA debugger scripts to a dedicated QEMU plugin which logs all I/O operations and some other metadata. I tried using this technique to statically rewrite files by basically following DataLoad (with unique identifier) -> Seek -> Read patterns. There's some annoying nuance to deal with (like seeking backwards implying that data was read, tested, then discarded) but I got this working. Unfortunately some object types encode absolute offsets in them that need to be touched up, so a couple of object types fail to load correctly in external tooling and the PC build of the game. Now I'm just using this data to completely reimplement the game engine's loading logic from scratch using a custom IO stream which checks the incoming IO operation (seek/read) against what I logged from the game engine to ensure a 1:1 match with how the game loads data.
  - WatchDog 1y ago
    Have you done any analysis of what proportion of the lin file is being read in total? You stated in the blog post, that your goal is to try and find unused content, however if as described, the file is just a record of how the game loads the data, then it won't contain any hidden unused assets, since unused assets would never have been read from the original unoptimised file, and thus never written to this optimized file.
    - landr0id 1y ago
      I agree and don't think there's any unused data. For `common.lin` for instance my parser reads it basically to the end and there's some small amount of data that's unused. I never actually quantified the amount but I'm fairly certain it's <100 bytes. Probably a bug in there. The goal post has shifted so far beyond my original intentional at this point. The devs working on the EnhancedSC mod have a strong desire to port some Xbox assets/maps to PC, so I'm mostly doing it at this point as an attempt to help them out. *On second thought, there's definitely some unused scripting functionality. Script functions which are unused are still included in their parent classes and are loaded if the parent object is loaded, even if never directly called. Whether or not any of this is interesting is another story. Textures and models though will definitely not be present unless they're used in some non-visible way.
harrylepotter 1y ago
ironically this was the game that enabled the savegame exploit with the bert and ernie fonts if i recall correctly
vivzkestrel 1y ago
what about splinter cell conviction, 15 yrs and nobody has figured out its map file format .unr that uses custom unreal engine 2.x. It even has a tool that lets you unpack its UMD files https://github.com/wcolding/UMDModTemplate https://github.com/wcolding/UMDModTemplate The library on github requires this tool unumd https://www.gildor.org/smf/index.php/topic,458.msg15196.html https://www.gildor.org/smf/index.php/topic,458.msg15196.html... The same tool also works for blacklist. I would like to change the type of enemy spawned in the map but I cannot find any assistance on it. UEExplorer doesnt work because it is some kinda custom map file
noufalibrahim 1y ago
This is an interesting post. I've been spending time on a hobby project[1] that requires reading some old archives and game asset files. I didn't have to do any reverse engineering since it's already done by others and documented on on the moddingwiki. However, I did implement the algorithms myself to work with the assets. It's an interesting rabbit hole to go down into and this post makes me appreciate the way in which this kind of forensic analysis is done. 1: https://eye-of-the-gopher.github.io/ https://eye-of-the-gopher.github.io/
ratmeadow 1y ago
> Compressing data means you save space on the disc... If you conveniently ignore the fact that common.lin is duplicated in each map's directory and is the same for every map I tested, which kinda negates part of this. This is an interesting thing I've noticed about game dev, it seems to sometimes live in a weird space of optimisation requirements vs hackiness. Where you'll have stuff like using instruction data as audio to save space, but then forget to compile in release mode or something. Really odd juxtaposition of near-genius-level optimisation with naive inefficiency. I'm assuming it's because, while there may be strict performance requirements, the devs are under the pump and there's so much going on that silly stuff ends up happening?
- monero-xmr 1y ago
  And passion to deliver. Engineers will kill themselves for a game release for no extra money and far less salary than their abilities would demand at a bigcorp. But they love it so they do it, and hack as best they can, to get their art into the world.
  - qingcharles 1y ago
    I bought into that scam. My dream job was video game dev. I spent my whole childhood writing video games. It was how I ended up doing it professionally for near zero money and 80 hour weeks and burning out completely in two years.
- rusk 1y ago
  There was a running theme in mythic quest about the engineers sweating over the system while monetisation just went bolted on a casino. Also happened in GTA5 [0] there was a ridiculous loading glitch that was quite well documented on here a while ago. Also a monetisation bolt on. So you have competing departments one of whom must justify itself by producing a heavily after my system. And another one which is licensed to generate revenue at any cost…… [0] https://news.ycombinator.com/item?id=26296339 https://news.ycombinator.com/item?id=26296339
  - avereveard 1y ago
tapia 1y ago
I'm always amazed by people doing reverse engineering of some country formats. There's a binary format that I've been wanting to reverse engineer, but I don't know exactly how to start. It's for the result file of a proprietary finite element program. Could anyone point me to some resources and also what are the basics that I need to learn to achieve this?
- iberator 1y ago
  > country format Country ?! What's the meaning
  - fainpul 1y ago
    Might be "binary format", autocorrected.
    - antonvs 1y ago
      That's really penguin
  - tapia 1y ago
    Sorry, that was a typo (autocorrected). I meant binary format.
- tralarpa 1y ago
  There are two approaches (sometimes mixed): (a) you reverse engineer the application writing or reading the file. Even without fully understanding the application it can give you valuable information about the format (e.g. "The application calls fwrite in a for loop ten times, maybe those are related to the ten elements that I see on the screen"). (b) you reverse engineer only the file. For example, you change one value in the application and compare the resulting output file. Or the opposite way: you change one value in the file and see what happens in the application when you load it.
- ashdnazg 1y ago
  The bare basics are working with a hex editor and understanding data types - ints, floats, null-terminated strings, length-prefixed strings etc. I'd recommend taking a well documented binary file format (Doom WAD file?), go over the documentation, and see that you manage to see the individual values in the hex editor. Now, after you have a feel for how things might look in hex, look at your own file. Start by saving an empty project from your program and identifying the header, maybe it's compressed? If it's not, change a tiny thing in the program and save again, compare the files to see what changed. Or alternatively change the file a tiny bit and load it. Write a parser and add things as you learn more. If the file isn't intentionally obfuscated, it should probably be just a matter of persevering until you can parse the entire file.
blixt 1y ago
The quirks of field values not matching expectations reminds me of a rabbit hole when I was reverse engineering the Starbound engine[1] and eventually figured out the game was using a flawed implementation of SHA-256 hashing and had to create a replica of it [2]. Originally I used Python [3] which is a really nice language for reverse engineering data formats thanks to its flexibility. [1] Starbounded was supposed to become an editor: https://github.com/blixt/starbounded https://github.com/blixt/starbounded [2] https://github.com/blixt/starbound-sha256 https://github.com/blixt/starbound-sha256 [3] https://github.com/blixt/py-starbound https://github.com/blixt/py-starbound
tomaytotomato 1y ago
Loved Splinter Cell I wonder if any of the original devs will stumble upon the author's article and then remember why they did those weird file offsets. There was a difference in the PC and Xbox versions, so it will be interesting to find out if the author sees any snippets or missing game assets in the Xbox version.
suckow 1y ago
[flagged]
amiga386 1y ago
So if I understand this right: * common.lin contains filenames, so that filename-expansion code in the game can work. But the offsets and sizes associated with the files are garbage * <filename>.lin contains a stream of every byte read from every file, while loading the level <filename>. The stream is then compressed in 16k chunks by zlib. * There is no indication in that stream of which real file was being read, nor the length of each read, nor what seeking was done (if any). All that metadata is gone. * The only way to recover this metadata is to run the game code and log the exact sequence of file opens, seeks, reads. * Alternatively, extract all that Unreal object loader code from the game and reimplement it yourself, so that you can let the contents of the stream drive the correct reading of the stream. The code should be deterministic. This sounds pretty hellish for the game developers, and I bet the debug versions of their game _ignored_ <filename>.lin and used the real source files, but _wrote_ <filename>.lin immediately after every load... any change to the Unreal objects could alter how they were read, and if the data streamed didn't perfectly match up with what was in the real files, you'd be toast. It reminds me of the extreme optimisation that Farbrausch did for .kkrieger -- they built a single binary, then ran and played it under instrumentation, and _any_ code path that wasn't taken was deleted from the binary to make it smaller. They forgot to take any damage in that playthrough, so all the code that applies damage to the player was deleted. Oops!
- debugnik 1y ago
  About .kkrieger's trimming, I had only heard that they forgot to press up on the main menu so it doesn't work, not about the damage thing.
- cedws 1y ago
  Could you explain a bit more about that code path optimisation? Why wouldn’t the compiler eliminate dead code? It seems like a very haphazard blunt force optimisation method.
  - anamexis 1y ago
    The compiler can’t determine which code paths are never used in practice at runtime.
LEEECHIES 1y ago
[dead]
hiimkeks 1y ago
> The entire content of the function is: > retn 4 > That's it. I've seen this before; it's a random() function.
- amiga386 1y ago
  Very good, a reference to https://youtu.be/DUGGJpn2_zY?t=2324 https://youtu.be/DUGGJpn2_zY?t=2324

299 points by todsacerdoti 1y ago

Weird that the one of these with no interaction got bumped rather than https://news.ycombinator.com/item?id=45842851 https://news.ycombinator.com/item?id=45842851

hcs 1y ago
I saw the other when it was first posted so it must have made the front page (or second page)

Dwedit 1y ago

Lowercase 'x' is always your dead giveaway that it's ZLIB.

LunicLynx 1y ago

The Xbox had strong requirements for loading times. This is probably a linear (lin) record of how the data was loaded unoptimized from the disk. And just written to a file. So in this file seek doesn’t do anything because seek kills the requirement of 45 sec per loading screen. Instead the logic is as follows: check if a .lin file exists. Yes: open a handle to it and only read from it with fread, what ever currently is at the current file position. No: while reading any file write the read bytes to a .lin file in the order they are read. This gives a highlyy optimized .lin file which can be read from the disk into memory, without creating a better dedicated loading mechanism. So if your really would like to unpack this. The first file being read is most likely the key, as it dictates what comes next. If it is a level model, then the position of the player in it might affect which other files to load etc. In short it’s not a file format in the classical sense, it’s a linear stream of game data.

landr0id 1y ago
>This is probably a linear (lin) record of how the data was loaded unoptimized from the disk. Yes, it's buried deep in the details but it's basically just every byte read being written in a linear stream to an output file. I don't know which stage of grief this is, but since I wrote this blog post I've now ported my IDA debugger scripts to a dedicated QEMU plugin which logs all I/O operations and some other metadata. I tried using this technique to statically rewrite files by basically following DataLoad (with unique identifier) -> Seek -> Read patterns. There's some annoying nuance to deal with (like seeking backwards implying that data was read, tested, then discarded) but I got this working. Unfortunately some object types encode absolute offsets in them that need to be touched up, so a couple of object types fail to load correctly in external tooling and the PC build of the game. Now I'm just using this data to completely reimplement the game engine's loading logic from scratch using a custom IO stream which checks the incoming IO operation (seek/read) against what I logged from the game engine to ensure a 1:1 match with how the game loads data.
- WatchDog 1y ago
  Have you done any analysis of what proportion of the lin file is being read in total? You stated in the blog post, that your goal is to try and find unused content, however if as described, the file is just a record of how the game loads the data, then it won't contain any hidden unused assets, since unused assets would never have been read from the original unoptimised file, and thus never written to this optimized file.
  - landr0id 1y ago
    I agree and don't think there's any unused data. For `common.lin` for instance my parser reads it basically to the end and there's some small amount of data that's unused. I never actually quantified the amount but I'm fairly certain it's <100 bytes. Probably a bug in there. The goal post has shifted so far beyond my original intentional at this point. The devs working on the EnhancedSC mod have a strong desire to port some Xbox assets/maps to PC, so I'm mostly doing it at this point as an attempt to help them out. *On second thought, there's definitely some unused scripting functionality. Script functions which are unused are still included in their parent classes and are loaded if the parent object is loaded, even if never directly called. Whether or not any of this is interesting is another story. Textures and models though will definitely not be present unless they're used in some non-visible way.

harrylepotter 1y ago

ironically this was the game that enabled the savegame exploit with the bert and ernie fonts if i recall correctly

vivzkestrel 1y ago

what about splinter cell conviction, 15 yrs and nobody has figured out its map file format .unr that uses custom unreal engine 2.x. It even has a tool that lets you unpack its UMD files https://github.com/wcolding/UMDModTemplate https://github.com/wcolding/UMDModTemplate The library on github requires this tool unumd https://www.gildor.org/smf/index.php/topic,458.msg15196.html https://www.gildor.org/smf/index.php/topic,458.msg15196.html... The same tool also works for blacklist. I would like to change the type of enemy spawned in the map but I cannot find any assistance on it. UEExplorer doesnt work because it is some kinda custom map file

noufalibrahim 1y ago

This is an interesting post. I've been spending time on a hobby project[1] that requires reading some old archives and game asset files. I didn't have to do any reverse engineering since it's already done by others and documented on on the moddingwiki. However, I did implement the algorithms myself to work with the assets. It's an interesting rabbit hole to go down into and this post makes me appreciate the way in which this kind of forensic analysis is done. 1: https://eye-of-the-gopher.github.io/ https://eye-of-the-gopher.github.io/

ratmeadow 1y ago

> Compressing data means you save space on the disc... If you conveniently ignore the fact that common.lin is duplicated in each map's directory and is the same for every map I tested, which kinda negates part of this. This is an interesting thing I've noticed about game dev, it seems to sometimes live in a weird space of optimisation requirements vs hackiness. Where you'll have stuff like using instruction data as audio to save space, but then forget to compile in release mode or something. Really odd juxtaposition of near-genius-level optimisation with naive inefficiency. I'm assuming it's because, while there may be strict performance requirements, the devs are under the pump and there's so much going on that silly stuff ends up happening?

monero-xmr 1y ago
And passion to deliver. Engineers will kill themselves for a game release for no extra money and far less salary than their abilities would demand at a bigcorp. But they love it so they do it, and hack as best they can, to get their art into the world.
- qingcharles 1y ago
  I bought into that scam. My dream job was video game dev. I spent my whole childhood writing video games. It was how I ended up doing it professionally for near zero money and 80 hour weeks and burning out completely in two years.
rusk 1y ago
There was a running theme in mythic quest about the engineers sweating over the system while monetisation just went bolted on a casino. Also happened in GTA5 [0] there was a ridiculous loading glitch that was quite well documented on here a while ago. Also a monetisation bolt on. So you have competing departments one of whom must justify itself by producing a heavily after my system. And another one which is licensed to generate revenue at any cost…… [0] https://news.ycombinator.com/item?id=26296339 https://news.ycombinator.com/item?id=26296339
- avereveard 1y ago

tapia 1y ago

I'm always amazed by people doing reverse engineering of some country formats. There's a binary format that I've been wanting to reverse engineer, but I don't know exactly how to start. It's for the result file of a proprietary finite element program. Could anyone point me to some resources and also what are the basics that I need to learn to achieve this?

iberator 1y ago
> country format Country ?! What's the meaning
- fainpul 1y ago
  Might be "binary format", autocorrected.
  - antonvs 1y ago
    That's really penguin
- tapia 1y ago
  Sorry, that was a typo (autocorrected). I meant binary format.
tralarpa 1y ago
There are two approaches (sometimes mixed): (a) you reverse engineer the application writing or reading the file. Even without fully understanding the application it can give you valuable information about the format (e.g. "The application calls fwrite in a for loop ten times, maybe those are related to the ten elements that I see on the screen"). (b) you reverse engineer only the file. For example, you change one value in the application and compare the resulting output file. Or the opposite way: you change one value in the file and see what happens in the application when you load it.
ashdnazg 1y ago
The bare basics are working with a hex editor and understanding data types - ints, floats, null-terminated strings, length-prefixed strings etc. I'd recommend taking a well documented binary file format (Doom WAD file?), go over the documentation, and see that you manage to see the individual values in the hex editor. Now, after you have a feel for how things might look in hex, look at your own file. Start by saving an empty project from your program and identifying the header, maybe it's compressed? If it's not, change a tiny thing in the program and save again, compare the files to see what changed. Or alternatively change the file a tiny bit and load it. Write a parser and add things as you learn more. If the file isn't intentionally obfuscated, it should probably be just a matter of persevering until you can parse the entire file.

blixt 1y ago

The quirks of field values not matching expectations reminds me of a rabbit hole when I was reverse engineering the Starbound engine[1] and eventually figured out the game was using a flawed implementation of SHA-256 hashing and had to create a replica of it [2]. Originally I used Python [3] which is a really nice language for reverse engineering data formats thanks to its flexibility. [1] Starbounded was supposed to become an editor: https://github.com/blixt/starbounded https://github.com/blixt/starbounded [2] https://github.com/blixt/starbound-sha256 https://github.com/blixt/starbound-sha256 [3] https://github.com/blixt/py-starbound https://github.com/blixt/py-starbound

tomaytotomato 1y ago

Loved Splinter Cell I wonder if any of the original devs will stumble upon the author's article and then remember why they did those weird file offsets. There was a difference in the PC and Xbox versions, so it will be interesting to find out if the author sees any snippets or missing game assets in the Xbox version.

suckow 1y ago

[flagged]

amiga386 1y ago

So if I understand this right: * common.lin contains filenames, so that filename-expansion code in the game can work. But the offsets and sizes associated with the files are garbage * <filename>.lin contains a stream of every byte read from every file, while loading the level <filename>. The stream is then compressed in 16k chunks by zlib. * There is no indication in that stream of which real file was being read, nor the length of each read, nor what seeking was done (if any). All that metadata is gone. * The only way to recover this metadata is to run the game code and log the exact sequence of file opens, seeks, reads. * Alternatively, extract all that Unreal object loader code from the game and reimplement it yourself, so that you can let the contents of the stream drive the correct reading of the stream. The code should be deterministic. This sounds pretty hellish for the game developers, and I bet the debug versions of their game _ignored_ <filename>.lin and used the real source files, but _wrote_ <filename>.lin immediately after every load... any change to the Unreal objects could alter how they were read, and if the data streamed didn't perfectly match up with what was in the real files, you'd be toast. It reminds me of the extreme optimisation that Farbrausch did for .kkrieger -- they built a single binary, then ran and played it under instrumentation, and _any_ code path that wasn't taken was deleted from the binary to make it smaller. They forgot to take any damage in that playthrough, so all the code that applies damage to the player was deleted. Oops!

debugnik 1y ago
About .kkrieger's trimming, I had only heard that they forgot to press up on the main menu so it doesn't work, not about the damage thing.
cedws 1y ago
Could you explain a bit more about that code path optimisation? Why wouldn’t the compiler eliminate dead code? It seems like a very haphazard blunt force optimisation method.
- anamexis 1y ago
  The compiler can’t determine which code paths are never used in practice at runtime.

LEEECHIES 1y ago

[dead]

hiimkeks 1y ago

> The entire content of the function is: > retn 4 > That's it. I've seen this before; it's a random() function.

amiga386 1y ago
Very good, a reference to https://youtu.be/DUGGJpn2_zY?t=2324 https://youtu.be/DUGGJpn2_zY?t=2324