8 ms·
Kaitai: Describe the structure of data, not how you read or write it
- nullspace 6y agoThis is very interesting, but just curious, when would you use something like this versus, say, protobufs?
- vander_elst 6y agoIf I understand it correctly, katai generates for you the reader as well, while protobuf generates only the data container. Katai is more a sort of protobuf + grpc for reading data.
- lima 6y agoProtobufs generate readers/writers for various languages (i.e. serialization/deserializiation). gRPC is an RPC framework that uses Protobufs.
- jackjeff 6y agoBig difference is Katai is designed to parse existing binary formats like PNG, JPEG, MIDI, WAV files. Things like Protobuf can serialize/deserialize any data but it’s very opiniated about how to do this. You won’t convince it to work with existing file formats.
- uerg 6y agoIf you try to speak a protocol which is not yours.
- UncleEntity 6y agoKaitai Struct can read any 'ol random data format while protobufs (I assume) are only able to read/write a specific protocol. I was playing with it a while back along with wasm and got it to decode all the individual opcodes (along with the rest of the file) but it turned out to be really, really slow in the generated python version. C++ probably has much better performance but I haven't actually tested the difference.
- tubs 6y agoLast time I looked at it, it generated sub-objects as pointers allocated with new, which was a bit meh.
- karlding 6y agoProtocol Buffers and Kaitai structs solve different problems, although they both deal with serialized data. With a protobuf, you don't really care about how the data that you're serializing into the binary buffer is represented in the binary format. All you care about is that your data can be serialized and deserialized. Conversely, Kaitai allows you to control and specify the representation of the serialized data. This allows you to specify arbitrary formats (like image formats, for example). Essentially you start with the serialization format instead of starting with the deserialized data (or like how the title of this submission says, "describe the structure of data"). As such, you can somewhat describe protobufs [0] using Kaitai structs, but the converse is not necessarily true. There's actually a section under their FAQ [1] with a more in-depth response to this. [0] https://formats.kaitai.io/google_protobuf/index.html https://formats.kaitai.io/google_protobuf/index.html [1] https://doc.kaitai.io/faq.html#vs-protobuf https://doc.kaitai.io/faq.html#vs-protobuf
- imtringued 6y agoYou can use it to reverse engineer specifications for arbitrary binary data.
- whiskypeters 6y agoi found the Kaitai toolset (IDE, compiler) to be useful in parsing or "deserialization" for proprietary financial protocols. if you are just reading data (especially in an environment with clients in different languages) it's a strong recommend, however lack of support for "serialization" means that you will still need to roll your own encoders. ultimately we created our own tooling/DSL for encoding as well as generating KSY files to generate clients as part of our builds edit: i forgot to add - there is an issue for serialization on kaitai github repo for some time [1], with some interesting discussion around the implementation challenges 1. https://github.com/kaitai-io/kaitai_struct/issues/27 https://github.com/kaitai-io/kaitai_struct/issues/27
- TACIXAT 6y agoI do love kaitai and recently contributed a grammar, but both the title and the copy talk about writing. > Reading and writing binary formats is hard... Kaitai Struct tries to make this job easier... Kaitai cannot write data back out. [1] This is a major limitation for me. It would be nice to use it as a mutation engine for fuzzing, but without being able to write it back out, it is mostly just beneficial for analysis. 1. https://doc.kaitai.io/faq.html#writing https://doc.kaitai.io/faq.html#writing
- kevinherron 6y agoYikes, huge limitation. Guess I won't be looking into this any further.
- imtringued 6y agoIt would be a CS breakthrough if it could... You're asking a lot...
- quiescant_dodo 6y agoIs this a known theoretical CS problem?
- jacb 6y agoI might be missing something - why would this be a breakthrough? It sounds complicated to generate the interfaces, sure, but is there a theoretical problem blocking this, or just practical?
- convolvatron 6y agoI've looked at this problem quite a bit over the years...I agree with you completely. there isn't anything fundamental here, just the normal cultural adoption issues, usability, etc. There may be some compilation/complexity issues around formats with variable length fields and self-description, but certainly less problematic than general purpose programming. I really wish though that there were more traction here, as I really believe that we should be quite prepared do deal with bits and not protobufs by default. nothing wrong with protobufs for quite a number of uses. I just don't know why people are so afraid of and/or biased against bit strings.
- karavelov 6y agoNo support for sum types? I didn't see even simple unions covered.
- ta988 6y agoDo you have any use case for these?
- adsharma 6y agohttps://adsharma.github.io/flattools-11222020.html https://adsharma.github.io/flattools-11222020.html supports unions and 4 popular type safe languages. The idea is that you'd write decorators in those languages to implement functionality similar to ksy or write a template to generate ksy and reuse Kaitai tool chain.
- mturk 6y agoI love Kaitai, and use it extensively (outside my day job) for exploring binary file formats of things like DOS games and quilting patterns. About a year ago I wrote a blog entry about using it for scientific data: https://matthewturk.github.io/post/kaitai-struct-scientific-data/ https://matthewturk.github.io/post/kaitai-struct-scientific-... . Recently I learned about some pretty cool work using it with respect to high-energy physics: https://osf.io/2sner/ https://osf.io/2sner/
- aappleby 6y agoCan Kaitai Struct store its metadata in a binary form that's parseable with Kaitai Struct? Seems like there's an obvious bootstrapping task there that would then make the custom .ksy format irrelevant.
- atbpaca 6y agoGuess it is similar to Apache Daffodil.
- nayuki 6y agoThe first thing that comes to my mind in this context is that "kaitai" means "want to write" (書いたい) in Japanese. It also has many other meanings: https://jisho.org/search/kaitai https://jisho.org/search/kaitai . Maybe the authors had disassembly (解体) in mind.
- denial 6y agoJust a correction to point away from want to write. It's 書きたい (kakitai) for want to write (and it's probably not want to buy 買いたい).
- nayuki 6y agoThanks for catching my blind spot there. I think what happened was that I used the -te form (書いて) and stripped off -te, but didn't conjugate the verb properly starting from 書く.
- ddevault 6y agoThis is a use-case for which a DSL is well-suited. They would be wise to abandon YAML.
- purpleidea 6y agoI think that I agree of you, and I'm obviously fond of clever DSL's cough mgmt config cough, but it's not clear what specifics you had in mind here. Share away if you're interested!
- pie_flavor 6y agoIf it was something that looked like YAML, but wasn't, that'd be fine. YAML in its simplest form is a neat idea. Indentation is simple and obvious. Same as it doesn't matter that TOML looks a heck of a lot like INI. But using YAML itself is terrible, because YAML is a colossally bloated language that does crazy things at the wrong points and has a million syntaxes that step on each others' toes.
- ddevault 6y agoYAML has far too many features and doesn't let you structure data well for this use-case. There's no concise way to provide typings for its fields, for instance - note how every field requires two lines in the examples, far too verbose. I don't really know what to tell you other than that YAML is not well-suited to this domain, and designing a domain-specific language which meets the particular needs of the system, and only just, would be better. You could base it off of C structs or Rust types or s-expressions or whatever else, the exact syntax isn't important so long as it allows you to concisely and precisely specify the semantics of the tool.
- zoom6628 6y agoI can s this being hugely useful to bring able to generate code to read wider range of formats instead of having to include/embed dozens of 3rd party libs to read the files. Could be strong tool for commercial product dev that have legal or technical limitation on use of 3rd party code.
- billfruit 6y agoIs there support for C in the pipeline? That would be nice, because lots of binary data manipulation in C even now, and it could use some quality-of-life updates.
- pabs3 6y agoI note that the Kaitai compiler is written in Scala and built using sbt, both of which are unfortunately not bootstrappable. https://bootstrappable.org/projects/jvm-languages.html https://bootstrappable.org/projects/jvm-languages.html
- rockwotj 6y agoSeems like a bummer that the stream interfaces are blocking for C++ and java :/
- beders 6y agoThis looks interesting, but is there an alternative for expressing the format? YAML is just plain awful.
- adsharma 6y agoI like the flatbuffer syntax as an IDL. It should be fairly trivial to write a tool to generate ksy https://adsharma.github.io/flattools-11222020.html https://adsharma.github.io/flattools-11222020.html
- Mikhail_Edoshin 6y agoIn ID3 tags there's such a thing as unsynchronization: the MP3 syncword is 11 bits set to one so for the tag data not to be mistaken for an MPR3 frame it must not have a similar sequence of bits. The solution was to replace every 0xFF in tag data with 0xFF 0x00. Or not to replace, as this mistake may only be made by old players that do not understand ID3. So there's a special setting for this that may occur in two places: in the whole tag or in an individual frame within a tag. The logic itself is simple, as you read data byte-by-byte you need to check check if the previous byte was 0xFF and the frame or tag is marked as unsychronized. Yet it's not that simple to describe this declaratively. I wonder if Kaitai can actually do this. From what I see Kaitai does have at least part of ID3 described, but it doesn't seem to actually do unsynchronization, as far as I can tell from the code.
- aliswe 6y agoI recently started making a (hopefully) version-independent blender converter using Kaitai: https://github.com/bjorn-ali-goransson/wz-pie-converter https://github.com/bjorn-ali-goransson/wz-pie-converter (Its for 3d models for the FOSS game Warzone 2100)
- nickdothutton 6y agoThe industry rediscovers DEC RMS?