4 ms·

I think OpenAPI is a rather arbitrary and approximate description of models built upon HTTP with JSON, but that's constrained enough, and the problems solved ar

I think OpenAPI is a rather arbitrary and approximate description of models built upon HTTP with JSON, but that's constrained enough, and the problems solved are often similar enough, for such a solution to be useful. Actually maybe JSON schema would be more useful to consider here, since that is about formats, rather than interfaces or protocols. With binary formats, on the other hand, there's neither data model nor usage constraints like that, and a generic tool to handle all the things would be just the Turing machine -- and then we have a bunch of languages for that. As for kaitai struct and similar attempts to handle arbitrary binary formats, I hear of those from time to time, and in some cases they seem to be useful, but they too should be either Turing-complete (and then it probably makes more sense to use general-purpose languages) or would fail to be useful occasionally.

NovemberWhiskey 5y ago
This seems obviously to be the right answer to me. Think about handling a ZIP file: you can't start at the beginning of the file and work forward from there. No; you have to start at/near the end of file and scan for an end-of-central-directory record based on a signature (it's not at a fixed offset due to a variable-length comment field). That then enables you to find the actual offset for the central directory itself; which in turns allow you to find the files themselves.
woodrowbarlow 5y ago
(author here) aye, this is a crucial point. any description language for binary formats quickly approaches turing complete, and i think this is what leads to so many of the "transpile-first parser languages". i'm impressed how well kaitai has managed to wrangle that tendency towards turing-completeness; they do have an embedded expression language for attribute binding, and they do have 'repeat' and 'switch' constructs, but the language is astoundingly declarative. i think something very similar to kaitai could work if the tools to work with the files programmatically were ergonomic and available in many different languages. e.g., if the kaitai compiler were a native compiled library with lots of packaged FFI wrappers for many languages.
- defanor 5y ago
  Well, nothing stops Turing-complete languages from being declarative: lambda calculus is both, for instance, as well as some of the languages based on it (in particular parser combinators in Haskell and similar languages are neat and flexible). And there's indeed a practice of embedding some languages into others, though I think it ends up quite awkward to use, and in this case potential limitations (actually I don't know if kaitai is Turing-complete) may be undesirable. I imagine in many cases it may be simpler to ship, say, a parser/decoder (and/or printer/encoder) library providing a C API that is easy to call via FFI, rather than a description in a less common language, to be used with a particular library. But then again, probably that'd have its uses too.

NovemberWhiskey 5y ago

This seems obviously to be the right answer to me. Think about handling a ZIP file: you can't start at the beginning of the file and work forward from there. No; you have to start at/near the end of file and scan for an end-of-central-directory record based on a signature (it's not at a fixed offset due to a variable-length comment field). That then enables you to find the actual offset for the central directory itself; which in turns allow you to find the files themselves.

woodrowbarlow 5y ago

(author here) aye, this is a crucial point. any description language for binary formats quickly approaches turing complete, and i think this is what leads to so many of the "transpile-first parser languages". i'm impressed how well kaitai has managed to wrangle that tendency towards turing-completeness; they do have an embedded expression language for attribute binding, and they do have 'repeat' and 'switch' constructs, but the language is astoundingly declarative. i think something very similar to kaitai could work if the tools to work with the files programmatically were ergonomic and available in many different languages. e.g., if the kaitai compiler were a native compiled library with lots of packaged FFI wrappers for many languages.

defanor 5y ago
Well, nothing stops Turing-complete languages from being declarative: lambda calculus is both, for instance, as well as some of the languages based on it (in particular parser combinators in Haskell and similar languages are neat and flexible). And there's indeed a practice of embedding some languages into others, though I think it ends up quite awkward to use, and in this case potential limitations (actually I don't know if kaitai is Turing-complete) may be undesirable. I imagine in many cases it may be simpler to ship, say, a parser/decoder (and/or printer/encoder) library providing a C API that is easy to call via FFI, rather than a description in a less common language, to be used with a particular library. But then again, probably that'd have its uses too.