3 ms·

I work in and around embedded software. Custom API's and internal dumps often contain custom binary formats. A tool that interactively generates a parser for

I work in and around embedded software. Custom API's and internal dumps often contain custom binary formats. A tool that interactively generates a parser for these would be great.

dr_zoidberg 11y ago
For "free exploring" of formats I usually dive with the IPython interpreter and a bunch of modules, struct and array are helpful for packing/unpacking of values. With struct you write a string that describes the data and the modules packs/unpacks the bytes. For example you can do: struct.unpack(">4sBhLq", data) and you are asking struct to parse a 19 byte long string, data (it has to be strictly 19 btyes), because you expect the following: * ">" big endian data * "4s" a 4 byte field to read as a string * "B" one unsigned byte * "h" one signed short (2 bytes) * "L" one unsigned long (4 bytes) * "q" one signed quad (8 bytes) I haven't read of anything that's automatic, if you have the API or some docs this is the closest I can think of. Of course, the online help of Python covers this module to greater detail: https://docs.python.org/2/library/struct.html https://docs.python.org/2/library/struct.html
- andrewchambers 11y ago
  https://github.com/construct/construct https://github.com/construct/construct This is the best thing I have ever used for parsing binary file formats.
  - dr_zoidberg 11y ago
    That looks very interesting!
    - andrewchambers 11y ago
      The examples are really cool, checkout the elf executable binary parser https://github.com/construct/construct/blob/master/construct/formats/executable/elf32.py https://github.com/construct/construct/blob/master/construct... or the png parser https://github.com/construct/construct/blob/master/construct/formats/graphics/png.py https://github.com/construct/construct/blob/master/construct....
  - moyix 11y ago
    Construct is lovely. I've used it on everything from MS debug symbols [1] to talking with USB devices [2]. [1] https://github.com/moyix/pdbparse https://github.com/moyix/pdbparse [2] https://github.com/moyix/fbtools https://github.com/moyix/fbtools

I work in and around embedded software. Custom API's and internal dumps often contain custom binary formats. A tool that interactively generates a parser for these would be great.

dr_zoidberg 11y ago

For "free exploring" of formats I usually dive with the IPython interpreter and a bunch of modules, struct and array are helpful for packing/unpacking of values. With struct you write a string that describes the data and the modules packs/unpacks the bytes. For example you can do: struct.unpack(">4sBhLq", data) and you are asking struct to parse a 19 byte long string, data (it has to be strictly 19 btyes), because you expect the following: * ">" big endian data * "4s" a 4 byte field to read as a string * "B" one unsigned byte * "h" one signed short (2 bytes) * "L" one unsigned long (4 bytes) * "q" one signed quad (8 bytes) I haven't read of anything that's automatic, if you have the API or some docs this is the closest I can think of. Of course, the online help of Python covers this module to greater detail: https://docs.python.org/2/library/struct.html https://docs.python.org/2/library/struct.html

andrewchambers 11y ago
https://github.com/construct/construct https://github.com/construct/construct This is the best thing I have ever used for parsing binary file formats.
- dr_zoidberg 11y ago
  That looks very interesting!
  - andrewchambers 11y ago
    The examples are really cool, checkout the elf executable binary parser https://github.com/construct/construct/blob/master/construct/formats/executable/elf32.py https://github.com/construct/construct/blob/master/construct... or the png parser https://github.com/construct/construct/blob/master/construct/formats/graphics/png.py https://github.com/construct/construct/blob/master/construct....
- moyix 11y ago
  Construct is lovely. I've used it on everything from MS debug symbols [1] to talking with USB devices [2]. [1] https://github.com/moyix/pdbparse https://github.com/moyix/pdbparse [2] https://github.com/moyix/fbtools https://github.com/moyix/fbtools