2 ms·

I tried several times to build such a thing but gave up every time. It is easy for simple file formats but if you want something universal it gets quite complex

by danbruc 12y ago

I tried several times to build such a thing but gave up every time. It is easy for simple file formats but if you want something universal it gets quite complex pretty quickly. There are file formats where the endianess is specified in the file and not statically known, there are file formats where the bit width of numbers is not statically known, there are (binary) strings delimited with some special marker and there are many different ways to escape the delimiter within the string, there are data blocks starting at fixed offsets, at offsets relative to some other block or the end of the file, there are offsets computed from different values found in the file, compressed blocks, unions of different block types discriminated by a tag, tags that need to be computed from multiple values, ... I would really love such a tool, but a language to describe most of the file formats in the wild is probably going to be (close to) Turing complete and is unfortunately nothing you can hack together over a weekend.

MontagFTB 12y ago
Let me start out by saying I do not claim the current binspector grammar can cover all the cases you've described. Yes, at some point binary formats can get so unwieldy the only way to read them is with some kind of Turing-complete system. What I have discovered, however, is that there are still a vast array of binary formats that can be well described with the format language as it stands. I have tried to devise workarounds to skip past the parts of formats that cannot be handled well. I hope to double-back on those limitations and extend the feature set of the language, but I hope what's in there is enough to get people started. Of the issues you mentioned, Binspector's language can handle dynamic field endianness, dynamic field size, terminators and delimiters, absolute/relative/dynamic offsets, tag-discriminated unions, and lambda calculations. It doesn't do everything, but it does allow a lot.
- danbruc 12y ago
  You already made it far beyond what I ever achieved. I aimed at a XML-based file format description and that gets messy pretty quickly once you have to express expressions. After the initial friction of having to write a parser, going with a custom language seems way more promising. I always wanted to peek under the hood of Wireshark and see how they do it - do they have specialized code for every protocol or do they use an abstract protocol description, too? I never did it. I hope your project matures, there is a lot of space in this niche.

by danbruc 12y ago

MontagFTB 12y ago

Let me start out by saying I do not claim the current binspector grammar can cover all the cases you've described. Yes, at some point binary formats can get so unwieldy the only way to read them is with some kind of Turing-complete system. What I have discovered, however, is that there are still a vast array of binary formats that can be well described with the format language as it stands. I have tried to devise workarounds to skip past the parts of formats that cannot be handled well. I hope to double-back on those limitations and extend the feature set of the language, but I hope what's in there is enough to get people started. Of the issues you mentioned, Binspector's language can handle dynamic field endianness, dynamic field size, terminators and delimiters, absolute/relative/dynamic offsets, tag-discriminated unions, and lambda calculations. It doesn't do everything, but it does allow a lot.

danbruc 12y ago
You already made it far beyond what I ever achieved. I aimed at a XML-based file format description and that gets messy pretty quickly once you have to express expressions. After the initial friction of having to write a parser, going with a custom language seems way more promising. I always wanted to peek under the hood of Wireshark and see how they do it - do they have specialized code for every protocol or do they use an abstract protocol description, too? I never did it. I hope your project matures, there is a lot of space in this niche.