4 ms·

Interesting. I didn't know anyone had come up with a declarative language for binary files.

jchw 1y ago
There's actually more than one, though Kaitai probably has the most maturity of any of them. Various hex editors have their own formats. 010 Editor has C-style binary templates, imhex has a binary pattern language as well. Okteta has Okteta Structure Definitions which can be declared using XML or with JS. Kaitai Struct is the most complete system that has code generation for multiple programming languages and isn't tied to a hex editor or anything else for that matter. That said, I think there's still a ton of room for improvement and innovation. Kaitai has a lot of useful tooling, but I think as it is today it falls a bit short: the code gen is not at the same support level for all languages (most languages are fairly limited), and I think serialization is still mostly experimental. That and there's probably a lot you could do to still make it more expressive and powerful.
- krapht 1y ago
  Have you used Google wuffs?
  - jchw 1y ago
    No, though I am familiar with it. I wouldn't have classified Kaitai and wuffs as being the same category of software, though I can see why you would.
- weinzierl 1y ago
  An adjacent or complementary field is description of data in transit. Wireshark dissectors come to mind. I think it'd be quite useful to unify these fields.
  - jchw 1y ago
    I had been trying to make a Kaitai to Wireshark Dissector compiler in my third party Kaitai implementation[1]. However, the Wireshark emitter is still basically useless for now. It only supports basic structs with basic attrs. I mainly started a third-party Kaitai implementation to experiment a bit with supporting new features in Go, and also just to have a native Go implementation for convenience, since I'm still not very good at Scala. However, once an approach is developed for how exactly to handle emitting to Wireshark it should be purely mechanical to graft on a Wireshark emitter to the upstream Kaitai Struct compiler, too. https://github.com/jchv/zanbato https://github.com/jchv/zanbato
emddudley 1y ago
There's an old XML one called Data Format Description Language (DFDL).
cr125rider 1y ago
How does this compare to how protobuf defines structures?
- mananaysiempre 1y ago
  Completely different problem, completely different solution. Protobuf and its ilk (ASN.1, Cap’n Proto, etc.) have you describe a tree structure, then map that to bytes according to their own sensibilities. Kaitai and its ilk (Wireshark might be a more familliar member of the group) have you describe a bunch of data structures as well as somebody else’s pretty much arbitrary ideas as to how they are to map to bytes, then deal with the results. You can’t use a Protobuf implementation to get EXIF data out of JPEGs, but then you can’t get format evolution guarantees out of Kaitai either. (I hear ASN.1 can somewhat cross the gap using ECN, but as far as I can tell literally nobody uses that in public.)
viraptor 1y ago
Imhex https://imhex.werwolv.net/ https://imhex.werwolv.net/ has another one. Not fully declarative, but that makes some things easier to deal with.
rented_mule 1y ago
In addition to languages, there's a Python library called "construct" that's been around for a long time. It uses a declarative style to make it surprisingly easy to make binary parsers and emitters. https://construct.readthedocs.io/en/latest/intro.html#example https://construct.readthedocs.io/en/latest/intro.html#exampl...
knome 1y ago
You should check out erlangs binary literals.
0points 1y ago
There's a metric ton of them by now. Here's incomplete notes from a couple of years ago: ### kaitai - https://github.com/kaitai-io/kaitai_struct https://github.com/kaitai-io/kaitai_struct - https://github.com/kaitai-io/awesome-kaitai https://github.com/kaitai-io/awesome-kaitai - http://formats.kaitai.io/dos_datetime/index.html http://formats.kaitai.io/dos_datetime/index.html ### Hexinator / Synalyze It! - Universal Parsing Engine - Hexinator is freemium version of Synalyze It! - https://github.com/synalysis/Grammars/blob/master/bitmap.grammar https://github.com/synalysis/Grammars/blob/master/bitmap.gra... ### quickbms - http://aluigi.altervista.org/quickbms.htm http://aluigi.altervista.org/quickbms.htm ## multiex - http://multiex.xentax.com/ http://multiex.xentax.com/ ### Game Extractor by WATTO - http://www.watto.org/game_extractor.html http://www.watto.org/game_extractor.html ### 010 editor templates - https://www.sweetscape.com/010editor/repository/templates/ https://www.sweetscape.com/010editor/repository/templates/ ### hex fiend templates - https://github.com/HexFiend/HexFiend/tree/master/templates https://github.com/HexFiend/HexFiend/tree/master/templates ### malcat - has some form of binary templates - https://malcat.fr/ https://malcat.fr/ ### Andys Binary Folding Editor - http://www.nyangau.org/be/be.htm http://www.nyangau.org/be/be.htm ### winhex templates - https://www.x-ways.net/winhex/templates/index.html https://www.x-ways.net/winhex/templates/index.html ### TRiD - file identifier - TrID is an utility designed to identify file types from their binary signatures. - https://mark0.net/soft-trid-e.html https://mark0.net/soft-trid-e.html ### GNU file - https://github.com/file/file https://github.com/file/file ### Noesis - Noesis is a tool for previewing and converting between hundreds of model, image, and animation formats. - http://richwhitehouse.com/index.php?content=inc_projects.php&showproject=91 http://richwhitehouse.com/index.php?content=inc_projects.php... - https://github.com/RoadTrain/noesis-plugins https://github.com/RoadTrain/noesis-plugins - https://github.com/RoadTrain/noesis-plugins-official https://github.com/RoadTrain/noesis-plugins-official ### Ninja ripper - extract individual models from DirectX 3D games, while they are running - https://ninjaripper.com/ https://ninjaripper.com/ ### Unpakke - http://www.nullsecurity.org/unpakke http://www.nullsecurity.org/unpakke ### Camoto online-only universal game modding tool - https://moddingwiki.shikadi.net/wiki/Camoto https://moddingwiki.shikadi.net/wiki/Camoto - https://camoto.shikadi.net/ https://camoto.shikadi.net/
- dloss 1y ago
  Great list! Will incorporate some of those into my list of tools for binary parsing: https://github.com/dloss/binary-parsing https://github.com/dloss/binary-parsing

jchw 1y ago

There's actually more than one, though Kaitai probably has the most maturity of any of them. Various hex editors have their own formats. 010 Editor has C-style binary templates, imhex has a binary pattern language as well. Okteta has Okteta Structure Definitions which can be declared using XML or with JS. Kaitai Struct is the most complete system that has code generation for multiple programming languages and isn't tied to a hex editor or anything else for that matter. That said, I think there's still a ton of room for improvement and innovation. Kaitai has a lot of useful tooling, but I think as it is today it falls a bit short: the code gen is not at the same support level for all languages (most languages are fairly limited), and I think serialization is still mostly experimental. That and there's probably a lot you could do to still make it more expressive and powerful.

krapht 1y ago
Have you used Google wuffs?
- jchw 1y ago
  No, though I am familiar with it. I wouldn't have classified Kaitai and wuffs as being the same category of software, though I can see why you would.
weinzierl 1y ago
An adjacent or complementary field is description of data in transit. Wireshark dissectors come to mind. I think it'd be quite useful to unify these fields.
- jchw 1y ago
  I had been trying to make a Kaitai to Wireshark Dissector compiler in my third party Kaitai implementation[1]. However, the Wireshark emitter is still basically useless for now. It only supports basic structs with basic attrs. I mainly started a third-party Kaitai implementation to experiment a bit with supporting new features in Go, and also just to have a native Go implementation for convenience, since I'm still not very good at Scala. However, once an approach is developed for how exactly to handle emitting to Wireshark it should be purely mechanical to graft on a Wireshark emitter to the upstream Kaitai Struct compiler, too. https://github.com/jchv/zanbato https://github.com/jchv/zanbato

emddudley 1y ago

There's an old XML one called Data Format Description Language (DFDL).

cr125rider 1y ago

How does this compare to how protobuf defines structures?

mananaysiempre 1y ago
Completely different problem, completely different solution. Protobuf and its ilk (ASN.1, Cap’n Proto, etc.) have you describe a tree structure, then map that to bytes according to their own sensibilities. Kaitai and its ilk (Wireshark might be a more familliar member of the group) have you describe a bunch of data structures as well as somebody else’s pretty much arbitrary ideas as to how they are to map to bytes, then deal with the results. You can’t use a Protobuf implementation to get EXIF data out of JPEGs, but then you can’t get format evolution guarantees out of Kaitai either. (I hear ASN.1 can somewhat cross the gap using ECN, but as far as I can tell literally nobody uses that in public.)

viraptor 1y ago

Imhex https://imhex.werwolv.net/ https://imhex.werwolv.net/ has another one. Not fully declarative, but that makes some things easier to deal with.

rented_mule 1y ago

In addition to languages, there's a Python library called "construct" that's been around for a long time. It uses a declarative style to make it surprisingly easy to make binary parsers and emitters. https://construct.readthedocs.io/en/latest/intro.html#example https://construct.readthedocs.io/en/latest/intro.html#exampl...

knome 1y ago

You should check out erlangs binary literals.

0points 1y ago

There's a metric ton of them by now. Here's incomplete notes from a couple of years ago: ### kaitai - https://github.com/kaitai-io/kaitai_struct https://github.com/kaitai-io/kaitai_struct - https://github.com/kaitai-io/awesome-kaitai https://github.com/kaitai-io/awesome-kaitai - http://formats.kaitai.io/dos_datetime/index.html http://formats.kaitai.io/dos_datetime/index.html ### Hexinator / Synalyze It! - Universal Parsing Engine - Hexinator is freemium version of Synalyze It! - https://github.com/synalysis/Grammars/blob/master/bitmap.grammar https://github.com/synalysis/Grammars/blob/master/bitmap.gra... ### quickbms - http://aluigi.altervista.org/quickbms.htm http://aluigi.altervista.org/quickbms.htm ## multiex - http://multiex.xentax.com/ http://multiex.xentax.com/ ### Game Extractor by WATTO - http://www.watto.org/game_extractor.html http://www.watto.org/game_extractor.html ### 010 editor templates - https://www.sweetscape.com/010editor/repository/templates/ https://www.sweetscape.com/010editor/repository/templates/ ### hex fiend templates - https://github.com/HexFiend/HexFiend/tree/master/templates https://github.com/HexFiend/HexFiend/tree/master/templates ### malcat - has some form of binary templates - https://malcat.fr/ https://malcat.fr/ ### Andys Binary Folding Editor - http://www.nyangau.org/be/be.htm http://www.nyangau.org/be/be.htm ### winhex templates - https://www.x-ways.net/winhex/templates/index.html https://www.x-ways.net/winhex/templates/index.html ### TRiD - file identifier - TrID is an utility designed to identify file types from their binary signatures. - https://mark0.net/soft-trid-e.html https://mark0.net/soft-trid-e.html ### GNU file - https://github.com/file/file https://github.com/file/file ### Noesis - Noesis is a tool for previewing and converting between hundreds of model, image, and animation formats. - http://richwhitehouse.com/index.php?content=inc_projects.php&showproject=91 http://richwhitehouse.com/index.php?content=inc_projects.php... - https://github.com/RoadTrain/noesis-plugins https://github.com/RoadTrain/noesis-plugins - https://github.com/RoadTrain/noesis-plugins-official https://github.com/RoadTrain/noesis-plugins-official ### Ninja ripper - extract individual models from DirectX 3D games, while they are running - https://ninjaripper.com/ https://ninjaripper.com/ ### Unpakke - http://www.nullsecurity.org/unpakke http://www.nullsecurity.org/unpakke ### Camoto online-only universal game modding tool - https://moddingwiki.shikadi.net/wiki/Camoto https://moddingwiki.shikadi.net/wiki/Camoto - https://camoto.shikadi.net/ https://camoto.shikadi.net/

dloss 1y ago
Great list! Will incorporate some of those into my list of tools for binary parsing: https://github.com/dloss/binary-parsing https://github.com/dloss/binary-parsing