4 ms·

What binary formats are you interested in? For my graduate thesis I had to read a lot about JPG, PNG, SQLite3 and MS-OLE file formats and I could give you some

What binary formats are you interested in? For my graduate thesis I had to read a lot about JPG, PNG, SQLite3 and MS-OLE file formats and I could give you some references to read from and shamelessly link to my github project where you can find some tools related to this (and others) file formats.

OneOneOneOne 12y ago
I work in and around embedded software. Custom API's and internal dumps often contain custom binary formats. A tool that interactively generates a parser for these would be great.
- dr_zoidberg 12y ago
  For "free exploring" of formats I usually dive with the IPython interpreter and a bunch of modules, struct and array are helpful for packing/unpacking of values. With struct you write a string that describes the data and the modules packs/unpacks the bytes. For example you can do: struct.unpack(">4sBhLq", data) and you are asking struct to parse a 19 byte long string, data (it has to be strictly 19 btyes), because you expect the following: * ">" big endian data * "4s" a 4 byte field to read as a string * "B" one unsigned byte * "h" one signed short (2 bytes) * "L" one unsigned long (4 bytes) * "q" one signed quad (8 bytes) I haven't read of anything that's automatic, if you have the API or some docs this is the closest I can think of. Of course, the online help of Python covers this module to greater detail: https://docs.python.org/2/library/struct.html https://docs.python.org/2/library/struct.html
  - andrewchambers 12y ago
    https://github.com/construct/construct https://github.com/construct/construct This is the best thing I have ever used for parsing binary file formats.
    - dr_zoidberg 12y ago
      That looks very interesting!
      - andrewchambers 12y ago
        The examples are really cool, checkout the elf executable binary parser https://github.com/construct/construct/blob/master/construct/formats/executable/elf32.py https://github.com/construct/construct/blob/master/construct... or the png parser https://github.com/construct/construct/blob/master/construct/formats/graphics/png.py https://github.com/construct/construct/blob/master/construct....
    - moyix 12y ago
      Construct is lovely. I've used it on everything from MS debug symbols [1] to talking with USB devices [2]. [1] https://github.com/moyix/pdbparse https://github.com/moyix/pdbparse [2] https://github.com/moyix/fbtools https://github.com/moyix/fbtools
pmorici 12y ago
How would you diagnose an unknown compression format? That's a problem I've encountered recently and I'm hoping there is an easier way than stepping through the reference compressor in a debugger.
- Kalium 12y ago
  If you're looking to find out what it is and what's in it, there are some very flexible unpackers out there. Titanium Core from Reversing Labs is excellent.
  - pmorici 12y ago
    I've already know what it is, and it isn't maleware or a packer. What I'm looking for is something that can do analysis on a bit of data and say what compression scheme is likely used.
    - Kalium 12y ago
      Ah. Well, the product I mentioned can usually identify packing schemes and unpack them. Commercial close-source, though.

OneOneOneOne 12y ago

I work in and around embedded software. Custom API's and internal dumps often contain custom binary formats. A tool that interactively generates a parser for these would be great.

dr_zoidberg 12y ago
For "free exploring" of formats I usually dive with the IPython interpreter and a bunch of modules, struct and array are helpful for packing/unpacking of values. With struct you write a string that describes the data and the modules packs/unpacks the bytes. For example you can do: struct.unpack(">4sBhLq", data) and you are asking struct to parse a 19 byte long string, data (it has to be strictly 19 btyes), because you expect the following: * ">" big endian data * "4s" a 4 byte field to read as a string * "B" one unsigned byte * "h" one signed short (2 bytes) * "L" one unsigned long (4 bytes) * "q" one signed quad (8 bytes) I haven't read of anything that's automatic, if you have the API or some docs this is the closest I can think of. Of course, the online help of Python covers this module to greater detail: https://docs.python.org/2/library/struct.html https://docs.python.org/2/library/struct.html
- andrewchambers 12y ago
  https://github.com/construct/construct https://github.com/construct/construct This is the best thing I have ever used for parsing binary file formats.
  - dr_zoidberg 12y ago
    That looks very interesting!
    - andrewchambers 12y ago
      The examples are really cool, checkout the elf executable binary parser https://github.com/construct/construct/blob/master/construct/formats/executable/elf32.py https://github.com/construct/construct/blob/master/construct... or the png parser https://github.com/construct/construct/blob/master/construct/formats/graphics/png.py https://github.com/construct/construct/blob/master/construct....
  - moyix 12y ago
    Construct is lovely. I've used it on everything from MS debug symbols [1] to talking with USB devices [2]. [1] https://github.com/moyix/pdbparse https://github.com/moyix/pdbparse [2] https://github.com/moyix/fbtools https://github.com/moyix/fbtools

pmorici 12y ago

How would you diagnose an unknown compression format? That's a problem I've encountered recently and I'm hoping there is an easier way than stepping through the reference compressor in a debugger.

Kalium 12y ago
If you're looking to find out what it is and what's in it, there are some very flexible unpackers out there. Titanium Core from Reversing Labs is excellent.
- pmorici 12y ago
  I've already know what it is, and it isn't maleware or a packer. What I'm looking for is something that can do analysis on a bit of data and say what compression scheme is likely used.
  - Kalium 12y ago
    Ah. Well, the product I mentioned can usually identify packing schemes and unpack them. Commercial close-source, though.