3 ms·
I contributed a number of file formats a few years ago (and attempted numerous others) but ran into a number of problems with certain file formats: 1. It's not
by dhx 5y ago
I contributed a number of file formats a few years ago (and attempted numerous others) but ran into a number of problems with certain file formats:
1. It's not possible to read from the file until a multiple byte termination sequence is detected. [1]
2. You can't read sections of a file where the termination condition is the presence of a sequence of bytes denoting the next unrelated section of the file (and you don't want to consume/read these bytes) [2]
3. The WebIDE at the time couldn't handle very large file format specifications such as Photoshop (PSD) [3]
4. Files containing compressed or encrypted sections require a compression/encryption algorithm to be hardcoded into Kaitai struct libraries for each programming language it can output to.
The WebIDE I particularly liked as it makes it easy to get started and share results. I also liked how Kaitai Struct allows easy definition of constraints (simple ones at least) into the file format specification so that you can say "this section of the file shall have a size not exceeding header.length * 2 bytes".
Some alternative binary file format specification attempts for those interested in seeing alternatives, each with their own set of problems/pros/cons:
1. 010 Editor [4]
2. Synalysis [5]
3. hachoir [6]
4. DFDL [7]
[1] https://github.com/kaitai-io/kaitai_struct/issues/158 https://github.com/kaitai-io/kaitai_struct/issues/158
[2] https://github.com/kaitai-io/kaitai_struct/issues/156 https://github.com/kaitai-io/kaitai_struct/issues/156
[3] https://raw.githubusercontent.com/davidhicks/kaitai_struct_formats/41ca1f0d80a7821db7a44d0c7809974e6afdfc3d/image/psd.ksy https://raw.githubusercontent.com/davidhicks/kaitai_struct_f...
[4] https://www.sweetscape.com/010editor/repository/templates/ https://www.sweetscape.com/010editor/repository/templates/
[5] https://github.com/synalysis/Grammars https://github.com/synalysis/Grammars
[6] https://github.com/vstinner/hachoir/tree/main/hachoir/parser https://github.com/vstinner/hachoir/tree/main/hachoir/parser
[7] https://github.com/DFDLSchemas/ https://github.com/DFDLSchemas/
- alophawen 5y agoNot able to decode your references. What does this refer to: > 4. Files containing compressed or encrypted sections require a compression/encryption algorithm to be hardcoded into Kaitai struct libraries for each programming language it can output to. Text mentions "Kaitai", but references 4. DFDL [7] or [4] https://www.sweetscape.com/010editor/repository/templates/ https://www.sweetscape.com/010editor/repository/templates/ is unrelated.
- dhx 5y agoSorry for the confusing format. "[X]" is the notation used for references but "X." is just an item in an ordered list (nothing to do with references). The comment about compressed and encrypted sections of files relates to the limited built-in algorithms that can be found at [1]. Any additional algorithms, for example, AES-XTS, would need to be added as custom processes and a corresponding function added for each language runtime Kaitai supports. "DFDL [7]" refers to the reference "[7] https://github.com/DFDLSchemas/ https://github.com/DFDLSchemas/". "4. ..." doesn't have any special meeting [1] https://raw.githubusercontent.com/kaitai-io/kaitai_struct_compiler/HEAD/shared/src/main/scala/io/kaitai/struct/format/ProcessExpr.scala https://raw.githubusercontent.com/kaitai-io/kaitai_struct_co...