11 ms·
I do love kaitai and recently contributed a grammar, but both the title and the copy talk about writing. > Reading and writing binary formats is hard... Kaitai
by TACIXAT 6y ago
I do love kaitai and recently contributed a grammar, but both the title and the copy talk about writing.
> Reading and writing binary formats is hard... Kaitai Struct tries to make this job easier...
Kaitai cannot write data back out. [1] This is a major limitation for me. It would be nice to use it as a mutation engine for fuzzing, but without being able to write it back out, it is mostly just beneficial for analysis.
1. https://doc.kaitai.io/faq.html#writing https://doc.kaitai.io/faq.html#writing
- kevinherron 6y agoYikes, huge limitation. Guess I won't be looking into this any further.
- imtringued 6y agoIt would be a CS breakthrough if it could... You're asking a lot...
- quiescant_dodo 6y agoIs this a known theoretical CS problem?
- jacb 6y agoI might be missing something - why would this be a breakthrough? It sounds complicated to generate the interfaces, sure, but is there a theoretical problem blocking this, or just practical?
- convolvatron 6y agoI've looked at this problem quite a bit over the years...I agree with you completely. there isn't anything fundamental here, just the normal cultural adoption issues, usability, etc. There may be some compilation/complexity issues around formats with variable length fields and self-description, but certainly less problematic than general purpose programming. I really wish though that there were more traction here, as I really believe that we should be quite prepared do deal with bits and not protobufs by default. nothing wrong with protobufs for quite a number of uses. I just don't know why people are so afraid of and/or biased against bit strings.
- joshspankit 6y agoThrough this thread I really feel like I’m missing something. Are we not talking about writing binary files that conform to the spec? Like, in the case of a GIF: simply writing valid garbage data should produce a file that presents as a valid GIF with noise for the image. Similarly: reading the file through the parser and writing it out unmodified should create an identical file (assuming no stenography). Right?
- spullara 6y agoI honestly don't get it either. The inverse of the read spec is the write spec. My guess having not dug deeply is that they don't distinguish between required and optional fields, that said, they should still be able to write what they have based on the read spec but could potentially be still an invalid file.
- IceDane 6y agoYou're right. The person saying this would be a breakthrough doesn't understand what this is doing. There are already similarly declarative tools which can accomplish this. Haskell has binary parsing libraries which work similarly and give you both reading and writing capabilities.
- armsforsharks 6y agoIt's a really interesting idea! I'm also surprised there hasn't been much traction here. I've started a Rust library for this: https://github.com/sharksforarms/deku https://github.com/sharksforarms/deku It's a declarative bit-level symmetrical reader writer generator library.
- afranchuk 6y agoYeah I was also confused. I wrote a bidirectional parser/writer layer for yaml in a haskell program I had at work. The yaml structure mappings were all declarative in the code and even allowed documentation for the structures to be printed out. It's not that hard once you define the primitive bidirectional (higher-order) mapping function to go from a `Configurable a` to a `Configurable b`, the rest kind of unfolds from there.
- HelloNurse 6y agoBinary file formats can be vastly more convoluted than YAML. For example, consider roundtripping ZIP archives or PDF documents, or both at the same time (see https://www.alchemistowl.org/pocorgtfo/ https://www.alchemistowl.org/pocorgtfo/).
- kevinherron 6y agoAre we talking about different things? ASN.1 does more or less what I was hoping Kaitai can do. Where's the breakthrough? I just want to be able to describe network protocols in some "language" and generate code that can serialize/deserialize it.
- comex 6y agoKaitai is designed to describe arbitrary, preexisting binary formats. You can’t do that with ASN.1.
- kevinherron 6y agoI know, I was just using it as kind of an example of what I want and of a similar problem. Maybe not the right example.
- OJFord 6y agoI think I'm missing something, but rust's `serde`?
- touisteur 6y agoErr. I'm not sure what asn.1 is missing for this? I've seen lots of people use asn.1 exactly for this (i.e. writing a grammar for an arbitrary pre-existing binary format not readily described in asn.1).
- ithkuil 6y agoasn.1 distinguishes between schema and encoding; there are many binary encodings and you can technically devise a custom one that would let you describe the high level structure with an asn.1 grammar and then lay out the actual bits with a custom encoding format so that it matches the pre-existing format you're writing the new serde for). This may work as many formats have this leveled approach. E.g. the lowest level of the spec may tell something about how to encode integers (all integers are 32-bit big endian, or varlen encoded ...), sequences (ength prefixed, or terminated by a sentinel. Any chances you have some reference to what you saw?
- IceDane 6y agoHow exactly would it be a cs breakthrough? Are you sure you understand what this does?
- nextaccountic 6y agoThis breakthrough happened already. This is called bidirectional or invertible parsing. See it discussed here https://news.ycombinator.com/item?id=16392654 https://news.ycombinator.com/item?id=16392654 And this paper https://dl.acm.org/doi/10.1145/1863523.1863525 https://dl.acm.org/doi/10.1145/1863523.1863525 "Invertible syntax descriptions: unifying parsing and pretty printing" And this Haskell library https://hackage.haskell.org/package/roundtrip https://hackage.haskell.org/package/roundtrip among others
- iab 6y agoWow - thanks for the heads-up
- floppy123 6y agoAny other lib that also generates Encoder/Writer code?
- comex 6y agoThe Construct library for Python can do it: https://construct.readthedocs.io/en/latest/intro.html#example https://construct.readthedocs.io/en/latest/intro.html#exampl... I’ve long searched for something better than Construct, but so far I have yet to find even an equal.
- wpietri 6y agoOoh, exciting! I built a parser [1] for AIS messages [2], a quirky ship-to-ship protocol. My lower-level stuff always felt clumsy to me. I'll have to see if this cleans it up. [1] https://github.com/wpietri/simpleais https://github.com/wpietri/simpleais [2] https://gpsd.gitlab.io/gpsd/AIVDM.html https://gpsd.gitlab.io/gpsd/AIVDM.html
- attheicearcade 6y agoIt can, but it can get incredibly slow for large formats. I was using it to reverse engineer some binary game formats but the parser would take a couple of minutes to complete. I rewrote it using struct and that time dropped to a few seconds. Useful for probing an unknown format, but I prefer the 010 editor since it’s more interactive.
- armsforsharks 6y agoI have been working on Deku: a declarative binary reading and writing: bit-level, symmetric, serialization/deserialization library. https://github.com/sharksforarms/deku https://github.com/sharksforarms/deku
- jchw 6y agoI wrote a sort-of adjacent library for Go at one point. I’m a bit stuck trying to figure out exactly what to do on 2.0 but it has a lot of Kaitai like features including an expression language for transforming things (on master version) and it supports writing structures out. https://github.com/go-restruct/restruct https://github.com/go-restruct/restruct
- layoutIfNeeded 6y agoI guess the submission title is incorrect then: >Kaitai: Describe the structure of data, not how you read or write it
- junon 6y agoYep this is why I dropped it. I started working on an alternative that supported writing but didn't follow through since I didn't think many people were interested in Kaitai to begin with.