4 ms·

I myself reverse engineered two file formats: the Movie Magic Scheduling database files - it's an application-specific project management tool for motion pictur

I myself reverse engineered two file formats: the Movie Magic Scheduling database files - it's an application-specific project management tool for motion picture and TV production - as well as the Zeni 4 Electronic Design Automation (ie. Electronic CAD) Physical Design documents. I've been thinking just in the last few days that it would be cool to write up how I did it. The basic way is to create a document with nothing in it, then a second document with a very small difference. For example in Movie Magic I had one new document that I saved without putting anything in it, then a second where I entered the letter "A" into a certain field. I made hex dumps of them both, then compared the hex. Then I would make some modest changes and additions, such as the letter "B" in a different field, or I would change the "A" in that first field to "AB". This approach only really works if you've had experience implementing file format code; it's particularly helpful to have designed original formats completely from scratch. For these specific reasons I am better than most at file format reverse-engineering, however I don't really have a clue about doing that for network protocols as the Samba folks did. Once I have some guess about the file format, I start writing a file interpreter, typically to dump the binary format into human-readible text. "Actor's Name: A", "Movie Title: B". Rather more importantly, that binary file dumper is chock-full of assertions. It's rather more important to get the assertions right than it is to display the actual file payload values. Loosely speaking, a set of correctly-implemented assertions is, in itself, the documentation for the file format you just reverse-engineered. It's also important to round-trip. So I come up with some very simple text input format, then a filter that creates documents that are readable by Zeni or Movie Magic, then I try to open the files. If that doesn't puke on my shoes, I edit the file a bit, then run that edited file through my human-readible dumping program. Lather, rinse and repeat. It is very tedious and slow but it is quite cool when you discover something that works. I've been puzzling over how I could offer a consulting service where I would reverse-engineer documents as a service to other companies. No doubt there is lots of need for this but I am unclear as to how to market it. The DMCA only covers Digital Restrictions Management. If the DMCA doesn't apply, as it did not in the case of files that you create yourself as with Movie Magic Scheduling or Zeni 4, while there are some procedures one must take care to follow, reverse-engineering is completely legal. Another way to put it, is that among the reasons we have patents, is so that reverse-engineering won't be necessary.

MichaelCrawford 12y ago
I just pulled this out of my ass a little while ago. I can do much better and go into a lot more detail, but not just now. I'll post a far better discussion later after I write up an article for my website. However I figured it would be helpful to post this particular piece so as to stimulate discussion that I can reflect in my upcoming article.
- redtuesday 12y ago
  I'm looking forward to it. Hopefully I can learn from your post. At the moment I proceed like you outlined above, but I'm stuck reverse engineering a save game file format from an old sports simulation game written in VB6. The programm seems to write the savefiles with the put command (__vbaPut3 in the assembly) and reads the savefiles with get (__vbaGet3). Even though this is my first try at reversing a file format, thanks to sweetscapes 010 editor I was able to make sense of one third of the file, but i'm stuck ever since. :D
  - MichaelCrawford 12y ago
    What version of Windows does it run on? If it's one of the DOS-based versions, you'll find junk in your files. That is, suppose your savefile was 512 bytes long, however it actually wrote 510 bytes starting at an offset of 2; the first two bytes might be, say, 0x1234, rather than the 0x0000 that anyone would reasonably expect. There are two ways to deal with this: the most straightforward way would be to use an NT-based Windows - NT4, Win2k, XP &c. But if you game won't run on NT - that's the case for many as they write directly to the video card and sound card - an acceptable solution would be to write a tool that fills up all the free space on your filesystem with zeroes. That's what I myself actually did for Movie Magic, as I was using Mac OS 8.1 at the time - long before we had Mac OS X. And that right there was my most-significant competitive advantage; as far as I have ever been able to tell, no one else has ever figured out to do that. I am willing to spill the beans as nowadays everyone uses at least allegedly "secure" operating systems. Finally you have to be concerned about junk returned by the system memory allocator. Even in *NIX, malloc doesn't initialize its buffers, however the kernel initializes memory to zero when you grow the process memory with setbrk. Loosely speaking, when malloc runs out of memory, it asks the kernel for more memory by increasing the break. The break is the boundary between mapped memory and unmapped memory; it grows upward. The stack starts towards the top of the address space and grows downward. Between the two is unmapped memory; while strictly speaking it is addressable to actually do so will cause a segvio. On a secure system, decrementing the stack pointer or incrementing the break will result in zero-filled memory. On MS-DOS, Windows 98, Mac OS 8 &c. that memory will be full of junk. I don't know how I'd do this on Windows but on the Classic Mac OS it was not hard at all to patch the system, so I'd hack the Memory Manager to always initializes the buffers it returns.

MichaelCrawford 12y ago

I just pulled this out of my ass a little while ago. I can do much better and go into a lot more detail, but not just now. I'll post a far better discussion later after I write up an article for my website. However I figured it would be helpful to post this particular piece so as to stimulate discussion that I can reflect in my upcoming article.

redtuesday 12y ago
I'm looking forward to it. Hopefully I can learn from your post. At the moment I proceed like you outlined above, but I'm stuck reverse engineering a save game file format from an old sports simulation game written in VB6. The programm seems to write the savefiles with the put command (__vbaPut3 in the assembly) and reads the savefiles with get (__vbaGet3). Even though this is my first try at reversing a file format, thanks to sweetscapes 010 editor I was able to make sense of one third of the file, but i'm stuck ever since. :D
- MichaelCrawford 12y ago
  What version of Windows does it run on? If it's one of the DOS-based versions, you'll find junk in your files. That is, suppose your savefile was 512 bytes long, however it actually wrote 510 bytes starting at an offset of 2; the first two bytes might be, say, 0x1234, rather than the 0x0000 that anyone would reasonably expect. There are two ways to deal with this: the most straightforward way would be to use an NT-based Windows - NT4, Win2k, XP &c. But if you game won't run on NT - that's the case for many as they write directly to the video card and sound card - an acceptable solution would be to write a tool that fills up all the free space on your filesystem with zeroes. That's what I myself actually did for Movie Magic, as I was using Mac OS 8.1 at the time - long before we had Mac OS X. And that right there was my most-significant competitive advantage; as far as I have ever been able to tell, no one else has ever figured out to do that. I am willing to spill the beans as nowadays everyone uses at least allegedly "secure" operating systems. Finally you have to be concerned about junk returned by the system memory allocator. Even in *NIX, malloc doesn't initialize its buffers, however the kernel initializes memory to zero when you grow the process memory with setbrk. Loosely speaking, when malloc runs out of memory, it asks the kernel for more memory by increasing the break. The break is the boundary between mapped memory and unmapped memory; it grows upward. The stack starts towards the top of the address space and grows downward. Between the two is unmapped memory; while strictly speaking it is addressable to actually do so will cause a segvio. On a secure system, decrementing the stack pointer or incrementing the break will result in zero-filled memory. On MS-DOS, Windows 98, Mac OS 8 &c. that memory will be full of junk. I don't know how I'd do this on Windows but on the Classic Mac OS it was not hard at all to patch the system, so I'd hack the Memory Manager to always initializes the buffers it returns.