8 ms·

Why isn't there a Swagger/OpenAPI for binary formats?

101 points by woodrowbarlow 5y ago

panick21_ 5y ago
gRPC
- woodrowbarlow 5y ago
  grpc is a layer on top of protobuf
  - randomdata 5y ago
    Extending it to define transport concerns, akin to Swagger/OpenAPI.
  - pianoben 5y ago
    In my view, gRPC is the closest "binary" analog to OpenAPI, which is a transport-layer concern first and foremost with discoverability features. If your problem space is "a way to assign meaning to arbitrary bitfields", I don't feel that OpenAPI is a very good comparison. Its purview is "representing web APIs in terms of structured JSON/XML payloads". That is, OpenAPI also deals in logical and not physical representation. (unless it specifies UTF-8 or something - it might, I don't know!)
    - woodrowbarlow 5y ago
      you know, i think you're right. on reflection, a comparison to json schema[1] (rather than openapi) would be more appropriate, as mentioned be defanor in another comment. 1: https://json-schema.org/ https://json-schema.org/
    - cbsmith 5y ago
      > In my view, gRPC is the closest "binary" analog to OpenAPI, which is a transport-layer concern first and foremost with discoverability features. Well, it and every other binary RPC technology...
SonOfLilit 5y ago
Swagger is, I think, named to reference SWIG, the tool everyone used to use to generate language bindings for C libraries, which is this. SWIG is built on top of C .h files, which are the _real_ Swagger for binary formats.
datavirtue 5y ago
Shhhh....we are still in stealth mode.
emteycz 5y ago
Protobuf, MessagePack?
- woodrowbarlow 5y ago
  (author here, the blog is under an alias) those are serialization formats, they don't allow for defining the physical representation of the data. for instance, you can't write a protobuf schema that de-serializes a .gif header.
  - kortex 5y ago
    What exactly do you mean by "defining the physical representation of the data"? Like, "here is a type called uint32, it is a 32 bit little endian unsigned number. Here is struct Foo, it has two uint32 fields". Like, that level of granularity? Basically Katai struct but can also be used to serialize, not just deserialize?
synack 5y ago
RecordFlux https://github.com/Componolit/RecordFlux https://github.com/Componolit/RecordFlux
- woodrowbarlow 5y ago
  (author here, the blog is under an alias) oh, interesting! i hadn't seen this one yet. i think this one would fall under the category "transpile-first languages" (like wuffs or spicy); i.e. it is useful for portable parsers but faces obstacles for generating ancillary tooling.
jayd16 5y ago
Does COM count?
ysleepy 5y ago
Automatic generation of wireshark dissectors would be amazing. I wrote a few in Lua, but it still felt second class citizen to built-in C dissectors, loading them was annoying and the UI to use them was too. A good dissector will annotate fields with semantic meaning and group and nest them, giving wireshark the precise bit offset+length where a value came from. Those are some different requirements than other parsers and reuse is therefore a bit difficult.
- woodrowbarlow 5y ago
  preach! by the way: wireshark can load ASN.1 source files directly. https://www.wireshark.org/docs/wsdg_html_chunked/CreatingAsn1Dissectors.html https://www.wireshark.org/docs/wsdg_html_chunked/CreatingAsn...
- hombre_fatal 5y ago
  Would something like this help? https://github.com/joushx/kaitai-to-wireshark https://github.com/joushx/kaitai-to-wireshark Happened to just see this article when reading Kaitai docs: https://www.incibe-cert.es/en/blog/understanding-industrial-network-traffic-dissectors-and-lua-and-kaitai https://www.incibe-cert.es/en/blog/understanding-industrial-... I was checking out Kaitai the other day (generate binary parsers from declarative structures) and noticed its tutorials section has great content on reverse-engineering binary formats with it: https://doc.kaitai.io https://doc.kaitai.io. Might be interesting to anyone interested in OP's question. I like the first blog link that reverses the binary format of one of those interactive anime novels from an .exe: https://hackernoon.com/reverse-engineering-visual-novels-101-d0bc3bf7ab8 https://hackernoon.com/reverse-engineering-visual-novels-101... Really cool walkthrough of how to do such a thing regardless of whether you use Kaitai to do it.
pjmlp 5y ago
Hello DCE RPC, SOM, COM, CORBA,.....
- cbsmith 5y ago
  Yeah... it's kind of amazing reading articles like this.
senko 5y ago
Well, there's ASN.1: https://en.m.wikipedia.org/wiki/ASN.1 https://en.m.wikipedia.org/wiki/ASN.1 Welcome to hell, abandon all hope, etc.
- woodrowbarlow 5y ago
  (author here, the blog is under an alias) ASN.1 is in a similar category to protobuf: it doesn't allow definition of the physical representation of the data. (it does support multiple serialization profiles, but ultimately ASN.1 data can only be serialized in a few different ways.) you can't write an ASN.1 file to parse arbitrary binary formats found in the wild.
  - tjalfi 5y ago
    ASN.1's Encoding Control Notation[0] allows bit-level encoding control and can be used to write ASN.1 specifications for existing protocols. [0] https://en.wikipedia.org/wiki/Encoding_Control_Notation https://en.wikipedia.org/wiki/Encoding_Control_Notation
alophawen 5y ago
This interests me a lot. If blog author reads this, is there some way to reach you in pm?
- woodrowbarlow 5y ago
  that's me, please see my profile. :) (i've added an edit on the bottom of the article)
  - alophawen 5y ago
    You got mail
defanor 5y ago
I think OpenAPI is a rather arbitrary and approximate description of models built upon HTTP with JSON, but that's constrained enough, and the problems solved are often similar enough, for such a solution to be useful. Actually maybe JSON schema would be more useful to consider here, since that is about formats, rather than interfaces or protocols. With binary formats, on the other hand, there's neither data model nor usage constraints like that, and a generic tool to handle all the things would be just the Turing machine -- and then we have a bunch of languages for that. As for kaitai struct and similar attempts to handle arbitrary binary formats, I hear of those from time to time, and in some cases they seem to be useful, but they too should be either Turing-complete (and then it probably makes more sense to use general-purpose languages) or would fail to be useful occasionally.
- NovemberWhiskey 5y ago
  This seems obviously to be the right answer to me. Think about handling a ZIP file: you can't start at the beginning of the file and work forward from there. No; you have to start at/near the end of file and scan for an end-of-central-directory record based on a signature (it's not at a fixed offset due to a variable-length comment field). That then enables you to find the actual offset for the central directory itself; which in turns allow you to find the files themselves.
- woodrowbarlow 5y ago
  (author here) aye, this is a crucial point. any description language for binary formats quickly approaches turing complete, and i think this is what leads to so many of the "transpile-first parser languages". i'm impressed how well kaitai has managed to wrangle that tendency towards turing-completeness; they do have an embedded expression language for attribute binding, and they do have 'repeat' and 'switch' constructs, but the language is astoundingly declarative. i think something very similar to kaitai could work if the tools to work with the files programmatically were ergonomic and available in many different languages. e.g., if the kaitai compiler were a native compiled library with lots of packaged FFI wrappers for many languages.
  - defanor 5y ago
deleted 5y ago
[deleted]
Jtsummers 5y ago
I don't know OpenAPI and only did a quick read. Does it support any formats other than JSON and XML? If not, then it hardly seems like an appropriate analog to what you're asking for. Can it support s-expr based formats? Plain text keyword:value pairs? Any arbitrary input? Because that's what you're asking for for "binary formats". Which is a wide-open field. To have a universal system for that you will need something closer to the parser description formats that you have already rejected. Just like you'd have to do to properly support any arbitrary plain text format instead of just JSON and XML.
- deleted 5y ago
  [deleted]
mkl95 5y ago
GRPC has a reflection protocol [1] that can be used to build Swagger style tools [2] [1] https://github.com/grpc/grpc/blob/master/doc/server-reflection.md#grpc-server-reflection-protocol https://github.com/grpc/grpc/blob/master/doc/server-reflecti... [2] https://github.com/grpc-swagger/grpc-swagger https://github.com/grpc-swagger/grpc-swagger
- woodrowbarlow 5y ago
  (author here) i did not know about the reflection protocol. maybe this could be a model for how kaitai struct could avoid the "centralized bottleneck" they have now in terms of ecosystem growth.
- luxurytent 5y ago
  You can see that reflection in action through tools like grpcui: https://github.com/fullstorydev/grpcui https://github.com/fullstorydev/grpcui
Khelavaster 5y ago
It's SignalR or WCF with binary serialization!
octagons 5y ago
Tangential to the rest of the conversation, but I often use/write binary templates [0] for 010 Editor [1]. The GUI-driven approach to navigating structures (such as PE headers) can be very useful. I wish there was a similar tool that allowed me to simply highlight and bookmark blocks of binary data when I'm dealing with an unknown file format, such as a firmware dump for something exotic and/or proprietary. [0] https://www.sweetscape.com/010editor/templates.html https://www.sweetscape.com/010editor/templates.html [1] https://www.sweetscape.com/010editor/ https://www.sweetscape.com/010editor/
nemetroid 5y ago
> both of these tools are only interested in allowing the author to model the logical representation of the data, not the physical. they are useful if you have data and want to automatically convert that into a file format or protocol, but not useful if you want to describe an existing file format or protocol. OpenAPI isn't necessarily great at this either. If your JSON API wasn't designed with OpenAPI in mind (or in a way that's similar to typical OpenAPI designs), it might not be possible to write an OpenAPI spec that describes the format well. But "type: object" is valid, so it's always possible to write a "spec". I think the question ("why is there no schema language for binary formats?") is valid, but that OpenAPI isn't as encompassing as it's laid out to be.
- emj 5y ago
  Yes and that's a fundamentally different problem; I think this is because there is always some kind of state in a protocol. SOAP has this problem, and also YANG[1] and tools that try to replace SNMP. You can never abstract away the state that the consumer and producer of the protocol has. There are lots of tools to disect, debug, resend and parse network protocols like IP and TCP but in those protocols we know there is a state built in that can not be broken, so we do not build OpenApi for them. OpenApi specifications that need three or four requests to reason with are a lot harder to work with than ones that are only one request, and these are different beast they become protocols. In my personal experience the binary specification often are protocols with a lot more implicit state in them, e.g. binary diff of in memory struct etc. You quickly lose the usefullness of Swagger the more state you have. That said there are lots of good example of binary protocols that try to be stateless. Conclussion: This blog post need to be scoped a bit better, what kind of binary communications do you want an OpenApi for and how is that different from what exists today? As soon as you have to encode the state of the application in the specification of the protocol (like TCP above) things get really complicated. [1] https://en.wikipedia.org/wiki/YANG https://en.wikipedia.org/wiki/YANG
Flocular 5y ago
DFDL ("Daffodil") https://en.wikipedia.org/wiki/Data_Format_Description_Language https://en.wikipedia.org/wiki/Data_Format_Description_Langua... was made pretty much exactly for describing/parsing binary formats in a schema-like format. It is however based on XML and not JSON.
divbzero 5y ago
OP might be conflating OpenAPI [1] which specifies HTTP endpoints of APIs with JSON Schema [2] which specifies data schemas of JSON documents. OpenAPI specs often use the application/json media type with schemas specified using JSON Schema, but they can also use other media types including binary formats. What OP is really asking is: Why isn’t there {JSON Schema,XML Schema,.proto files} for arbitrary binary formats? Kaitai [3] mentioned by OP and sibling comments looks interesting, don’t think I’ve seen something like it before. [1]: https://swagger.io/specification/ https://swagger.io/specification/ [2]: https://json-schema.org/ https://json-schema.org/ [3]: https://kaitai.io/ https://kaitai.io/
onetimeuse74 5y ago
you are looking for katai struct: https://kaitai.io/ https://kaitai.io/ katai allows you to declaratively define a binary format bit for bit - they even have an IDE
- billfruit 5y ago
  Can it parse a binary file/stream containing heterogeneous binary structures? Documentation seems not that clear regarding this. Also the structure definition seems to needs compiling, what if there is change of structure at runtime? Can it handle it? Is it possible to for an application to have editable structures and KaiTai to parse them without needing to recompile the whole application (I'm speaking here in a java context).
CMCDragonkai 5y ago
I did research into session types to try and solve this exact problem. However the theory still new and implementations are lacking. Again session types encode the logic, you still need some sort of structure/content language to describe the messages. Session types should be able to encode complex protocols and combined with kaitai, maybe something can emerge.
DannyBee 5y ago
The author sort of answers the title question without appearing to realize it. The answer is: "The things that exist are good enough for what people are usually trying to do" It's unclear what the author's real use case is. The reason proto/etc suffice for most people is that they do not care about the physical format, and control it. As a result, there is no need to define it explicitly or separately, and mostly people don't care to. The folks that need to control the physical format are often those reading other people's formats. In a lot of cases, those producing the format don't care if you can read it or not :). They are often part of some ecosystem (say, photoshop, or whatever). As a result, they wouldn't use an "openapi for binary formats" even if it existed. For lots of other kinds of standardized formats to read (video, image, etc), these days the processing is harder than the parsing. Not that it's worth nothing, mind you, but that i doubt you would find a lot of people care enough to think it's worth it. You also start to get into fairly different kinds of formats quickly (IE network protocol formats vs file formats), and those differences often lend themselves to different abstractions (IE you often want to be able to stream decode network formats, but may care less about stream decoding file formats, etc). The market that remains to be served well here seems pretty small. Certainly there are folks who want to control the physical format for some reason (whether they need to or not), etc. But again, that feels like a small fraction of cases. As a result, nobody felt enough of a need for OpenAPI for binary formats to push very hard on it. (and attempts at doing so in the past have failed for lack of value)
billfruit 5y ago
Can KaiTai parse a binary file/stream containing heterogeneous binary structures? Documentation seems not that clear regarding this. Also the structure defenition seems to needs compiling, what if there is change of structure at runtime? Can it handle it? Is it possible to for an application to have editable structures and KaiTai to parse them without needing to recompile the whole application (I'm speaking here in a java context).
HNHatesUsers 5y ago
deleted 5y ago
[deleted]
zeckalpha 5y ago
Erlang has some special syntax for this and the people who use it seem to like it. Here’s a paper on adding binary comprehensions (like Python’s list comprehensions): http://user.it.uu.se/~pergu/papers/erlang05.pdf http://user.it.uu.se/~pergu/papers/erlang05.pdf
jchw 5y ago
My project Restruct[1] does Kaitai-like things but also supports serialization. Unfortunately, it only supports Go and only deals with Go struct tags rather than YAML manifests. Still, it totally can be used for serialization. I use it to sketch out quick projects against arbitrary binary formats. Two examples: one, parsing PNG headers to implement a quick binwalk-like program for just PNG that looks for the IEND chunk to extract accurately[2], two, a program that splits FL Studio FLP projects by playlist track[3]. I feel like I’ve self-promoted Restruct like four times on Hacker News, and I feel kind of bad because it could use improvements and even some bug fixes and I never seem to get around to it. Oh well. It’s still useful for me, I hope it’s useful for others, too. That said, Kaitai has a fairly clear path towards adding serialization from a design PoV; many things that would be calculated for parsing structures in deserialization could just become checks/assertions in serialization. As an example, checking that an expression calculates out to the expected value would be a reasonable approach. Reversible expressions could be implemented for some cases, too, if you want it to do more of the heavy lifting. I think the biggest obstacle is actually implementing it, and frankly my Scala is too weak to help with such a relatively big undertaking. I’ve also played with the rust nom library, which implements functional programming style parser combinators. It is quite cool how it can express fairly complex grammars and binary formats pretty much equally well, albeit optimizing it effectively requires serious magic that I do not think nom has. (I assume in Haskell, the same thing can be done with mind-boggling optimization power.) It also doesn’t handle serialization, but it’s still worth noting for robust deserialization. [1]: https://github.com/go-restruct/restruct https://github.com/go-restruct/restruct [2]: https://github.com/jchv/pngextract https://github.com/jchv/pngextract [3]: https://github.com/jchv/flsplit https://github.com/jchv/flsplit
manish_gill 5y ago
http://www.jemarch.net/poke http://www.jemarch.net/poke
catern 5y ago
I've long looked for something like this for my day job, which involves implementing many different custom binary network protocols. One new project in this space is poke: http://www.jemarch.net/poke http://www.jemarch.net/poke It can describe arbitrary formats for both reading and writing, and it's usable programmatically, so I think it can solve the problem!
olodus 5y ago
Another one from google: flatbuffers. We use this at work mostly to be able to be ISSU proof in out data models. It has some useful features for this like checking that the evolution of a data model is in line with ISSU compliancy guidelines, which we check in CI.
sandreas 5y ago
Maybe kaitai struct is what you are looking for? ;) https://kaitai.io/ https://kaitai.io/

101 points by woodrowbarlow 5y ago

panick21_ 5y ago

gRPC

woodrowbarlow 5y ago
grpc is a layer on top of protobuf
- randomdata 5y ago
  Extending it to define transport concerns, akin to Swagger/OpenAPI.
- pianoben 5y ago
  In my view, gRPC is the closest "binary" analog to OpenAPI, which is a transport-layer concern first and foremost with discoverability features. If your problem space is "a way to assign meaning to arbitrary bitfields", I don't feel that OpenAPI is a very good comparison. Its purview is "representing web APIs in terms of structured JSON/XML payloads". That is, OpenAPI also deals in logical and not physical representation. (unless it specifies UTF-8 or something - it might, I don't know!)
  - woodrowbarlow 5y ago
    you know, i think you're right. on reflection, a comparison to json schema[1] (rather than openapi) would be more appropriate, as mentioned be defanor in another comment. 1: https://json-schema.org/ https://json-schema.org/
  - cbsmith 5y ago
    > In my view, gRPC is the closest "binary" analog to OpenAPI, which is a transport-layer concern first and foremost with discoverability features. Well, it and every other binary RPC technology...

SonOfLilit 5y ago

Swagger is, I think, named to reference SWIG, the tool everyone used to use to generate language bindings for C libraries, which is this. SWIG is built on top of C .h files, which are the _real_ Swagger for binary formats.

datavirtue 5y ago

Shhhh....we are still in stealth mode.

emteycz 5y ago

Protobuf, MessagePack?

woodrowbarlow 5y ago
(author here, the blog is under an alias) those are serialization formats, they don't allow for defining the physical representation of the data. for instance, you can't write a protobuf schema that de-serializes a .gif header.
- kortex 5y ago
  What exactly do you mean by "defining the physical representation of the data"? Like, "here is a type called uint32, it is a 32 bit little endian unsigned number. Here is struct Foo, it has two uint32 fields". Like, that level of granularity? Basically Katai struct but can also be used to serialize, not just deserialize?

synack 5y ago

RecordFlux https://github.com/Componolit/RecordFlux https://github.com/Componolit/RecordFlux

woodrowbarlow 5y ago
(author here, the blog is under an alias) oh, interesting! i hadn't seen this one yet. i think this one would fall under the category "transpile-first languages" (like wuffs or spicy); i.e. it is useful for portable parsers but faces obstacles for generating ancillary tooling.

jayd16 5y ago

Does COM count?

ysleepy 5y ago

Automatic generation of wireshark dissectors would be amazing. I wrote a few in Lua, but it still felt second class citizen to built-in C dissectors, loading them was annoying and the UI to use them was too. A good dissector will annotate fields with semantic meaning and group and nest them, giving wireshark the precise bit offset+length where a value came from. Those are some different requirements than other parsers and reuse is therefore a bit difficult.

woodrowbarlow 5y ago
preach! by the way: wireshark can load ASN.1 source files directly. https://www.wireshark.org/docs/wsdg_html_chunked/CreatingAsn1Dissectors.html https://www.wireshark.org/docs/wsdg_html_chunked/CreatingAsn...
hombre_fatal 5y ago
Would something like this help? https://github.com/joushx/kaitai-to-wireshark https://github.com/joushx/kaitai-to-wireshark Happened to just see this article when reading Kaitai docs: https://www.incibe-cert.es/en/blog/understanding-industrial-network-traffic-dissectors-and-lua-and-kaitai https://www.incibe-cert.es/en/blog/understanding-industrial-... I was checking out Kaitai the other day (generate binary parsers from declarative structures) and noticed its tutorials section has great content on reverse-engineering binary formats with it: https://doc.kaitai.io https://doc.kaitai.io. Might be interesting to anyone interested in OP's question. I like the first blog link that reverses the binary format of one of those interactive anime novels from an .exe: https://hackernoon.com/reverse-engineering-visual-novels-101-d0bc3bf7ab8 https://hackernoon.com/reverse-engineering-visual-novels-101... Really cool walkthrough of how to do such a thing regardless of whether you use Kaitai to do it.

pjmlp 5y ago

Hello DCE RPC, SOM, COM, CORBA,.....

cbsmith 5y ago
Yeah... it's kind of amazing reading articles like this.

senko 5y ago

Well, there's ASN.1: https://en.m.wikipedia.org/wiki/ASN.1 https://en.m.wikipedia.org/wiki/ASN.1 Welcome to hell, abandon all hope, etc.

woodrowbarlow 5y ago
(author here, the blog is under an alias) ASN.1 is in a similar category to protobuf: it doesn't allow definition of the physical representation of the data. (it does support multiple serialization profiles, but ultimately ASN.1 data can only be serialized in a few different ways.) you can't write an ASN.1 file to parse arbitrary binary formats found in the wild.
- tjalfi 5y ago
  ASN.1's Encoding Control Notation[0] allows bit-level encoding control and can be used to write ASN.1 specifications for existing protocols. [0] https://en.wikipedia.org/wiki/Encoding_Control_Notation https://en.wikipedia.org/wiki/Encoding_Control_Notation

alophawen 5y ago

This interests me a lot. If blog author reads this, is there some way to reach you in pm?

woodrowbarlow 5y ago
that's me, please see my profile. :) (i've added an edit on the bottom of the article)
- alophawen 5y ago
  You got mail

defanor 5y ago

I think OpenAPI is a rather arbitrary and approximate description of models built upon HTTP with JSON, but that's constrained enough, and the problems solved are often similar enough, for such a solution to be useful. Actually maybe JSON schema would be more useful to consider here, since that is about formats, rather than interfaces or protocols. With binary formats, on the other hand, there's neither data model nor usage constraints like that, and a generic tool to handle all the things would be just the Turing machine -- and then we have a bunch of languages for that. As for kaitai struct and similar attempts to handle arbitrary binary formats, I hear of those from time to time, and in some cases they seem to be useful, but they too should be either Turing-complete (and then it probably makes more sense to use general-purpose languages) or would fail to be useful occasionally.

NovemberWhiskey 5y ago
This seems obviously to be the right answer to me. Think about handling a ZIP file: you can't start at the beginning of the file and work forward from there. No; you have to start at/near the end of file and scan for an end-of-central-directory record based on a signature (it's not at a fixed offset due to a variable-length comment field). That then enables you to find the actual offset for the central directory itself; which in turns allow you to find the files themselves.
woodrowbarlow 5y ago
(author here) aye, this is a crucial point. any description language for binary formats quickly approaches turing complete, and i think this is what leads to so many of the "transpile-first parser languages". i'm impressed how well kaitai has managed to wrangle that tendency towards turing-completeness; they do have an embedded expression language for attribute binding, and they do have 'repeat' and 'switch' constructs, but the language is astoundingly declarative. i think something very similar to kaitai could work if the tools to work with the files programmatically were ergonomic and available in many different languages. e.g., if the kaitai compiler were a native compiled library with lots of packaged FFI wrappers for many languages.
- defanor 5y ago

deleted 5y ago

[deleted]

Jtsummers 5y ago

I don't know OpenAPI and only did a quick read. Does it support any formats other than JSON and XML? If not, then it hardly seems like an appropriate analog to what you're asking for. Can it support s-expr based formats? Plain text keyword:value pairs? Any arbitrary input? Because that's what you're asking for for "binary formats". Which is a wide-open field. To have a universal system for that you will need something closer to the parser description formats that you have already rejected. Just like you'd have to do to properly support any arbitrary plain text format instead of just JSON and XML.

deleted 5y ago
[deleted]

mkl95 5y ago

GRPC has a reflection protocol [1] that can be used to build Swagger style tools [2] [1] https://github.com/grpc/grpc/blob/master/doc/server-reflection.md#grpc-server-reflection-protocol https://github.com/grpc/grpc/blob/master/doc/server-reflecti... [2] https://github.com/grpc-swagger/grpc-swagger https://github.com/grpc-swagger/grpc-swagger

woodrowbarlow 5y ago
(author here) i did not know about the reflection protocol. maybe this could be a model for how kaitai struct could avoid the "centralized bottleneck" they have now in terms of ecosystem growth.
luxurytent 5y ago
You can see that reflection in action through tools like grpcui: https://github.com/fullstorydev/grpcui https://github.com/fullstorydev/grpcui

Khelavaster 5y ago

It's SignalR or WCF with binary serialization!

octagons 5y ago

Tangential to the rest of the conversation, but I often use/write binary templates [0] for 010 Editor [1]. The GUI-driven approach to navigating structures (such as PE headers) can be very useful. I wish there was a similar tool that allowed me to simply highlight and bookmark blocks of binary data when I'm dealing with an unknown file format, such as a firmware dump for something exotic and/or proprietary. [0] https://www.sweetscape.com/010editor/templates.html https://www.sweetscape.com/010editor/templates.html [1] https://www.sweetscape.com/010editor/ https://www.sweetscape.com/010editor/

nemetroid 5y ago

> both of these tools are only interested in allowing the author to model the logical representation of the data, not the physical. they are useful if you have data and want to automatically convert that into a file format or protocol, but not useful if you want to describe an existing file format or protocol. OpenAPI isn't necessarily great at this either. If your JSON API wasn't designed with OpenAPI in mind (or in a way that's similar to typical OpenAPI designs), it might not be possible to write an OpenAPI spec that describes the format well. But "type: object" is valid, so it's always possible to write a "spec". I think the question ("why is there no schema language for binary formats?") is valid, but that OpenAPI isn't as encompassing as it's laid out to be.

emj 5y ago
Yes and that's a fundamentally different problem; I think this is because there is always some kind of state in a protocol. SOAP has this problem, and also YANG[1] and tools that try to replace SNMP. You can never abstract away the state that the consumer and producer of the protocol has. There are lots of tools to disect, debug, resend and parse network protocols like IP and TCP but in those protocols we know there is a state built in that can not be broken, so we do not build OpenApi for them. OpenApi specifications that need three or four requests to reason with are a lot harder to work with than ones that are only one request, and these are different beast they become protocols. In my personal experience the binary specification often are protocols with a lot more implicit state in them, e.g. binary diff of in memory struct etc. You quickly lose the usefullness of Swagger the more state you have. That said there are lots of good example of binary protocols that try to be stateless. Conclussion: This blog post need to be scoped a bit better, what kind of binary communications do you want an OpenApi for and how is that different from what exists today? As soon as you have to encode the state of the application in the specification of the protocol (like TCP above) things get really complicated. [1] https://en.wikipedia.org/wiki/YANG https://en.wikipedia.org/wiki/YANG

Flocular 5y ago

DFDL ("Daffodil") https://en.wikipedia.org/wiki/Data_Format_Description_Language https://en.wikipedia.org/wiki/Data_Format_Description_Langua... was made pretty much exactly for describing/parsing binary formats in a schema-like format. It is however based on XML and not JSON.

divbzero 5y ago

OP might be conflating OpenAPI [1] which specifies HTTP endpoints of APIs with JSON Schema [2] which specifies data schemas of JSON documents. OpenAPI specs often use the application/json media type with schemas specified using JSON Schema, but they can also use other media types including binary formats. What OP is really asking is: Why isn’t there {JSON Schema,XML Schema,.proto files} for arbitrary binary formats? Kaitai [3] mentioned by OP and sibling comments looks interesting, don’t think I’ve seen something like it before. [1]: https://swagger.io/specification/ https://swagger.io/specification/ [2]: https://json-schema.org/ https://json-schema.org/ [3]: https://kaitai.io/ https://kaitai.io/

onetimeuse74 5y ago

you are looking for katai struct: https://kaitai.io/ https://kaitai.io/ katai allows you to declaratively define a binary format bit for bit - they even have an IDE

billfruit 5y ago
Can it parse a binary file/stream containing heterogeneous binary structures? Documentation seems not that clear regarding this. Also the structure definition seems to needs compiling, what if there is change of structure at runtime? Can it handle it? Is it possible to for an application to have editable structures and KaiTai to parse them without needing to recompile the whole application (I'm speaking here in a java context).

CMCDragonkai 5y ago

I did research into session types to try and solve this exact problem. However the theory still new and implementations are lacking. Again session types encode the logic, you still need some sort of structure/content language to describe the messages. Session types should be able to encode complex protocols and combined with kaitai, maybe something can emerge.

DannyBee 5y ago

The author sort of answers the title question without appearing to realize it. The answer is: "The things that exist are good enough for what people are usually trying to do" It's unclear what the author's real use case is. The reason proto/etc suffice for most people is that they do not care about the physical format, and control it. As a result, there is no need to define it explicitly or separately, and mostly people don't care to. The folks that need to control the physical format are often those reading other people's formats. In a lot of cases, those producing the format don't care if you can read it or not :). They are often part of some ecosystem (say, photoshop, or whatever). As a result, they wouldn't use an "openapi for binary formats" even if it existed. For lots of other kinds of standardized formats to read (video, image, etc), these days the processing is harder than the parsing. Not that it's worth nothing, mind you, but that i doubt you would find a lot of people care enough to think it's worth it. You also start to get into fairly different kinds of formats quickly (IE network protocol formats vs file formats), and those differences often lend themselves to different abstractions (IE you often want to be able to stream decode network formats, but may care less about stream decoding file formats, etc). The market that remains to be served well here seems pretty small. Certainly there are folks who want to control the physical format for some reason (whether they need to or not), etc. But again, that feels like a small fraction of cases. As a result, nobody felt enough of a need for OpenAPI for binary formats to push very hard on it. (and attempts at doing so in the past have failed for lack of value)

billfruit 5y ago

Can KaiTai parse a binary file/stream containing heterogeneous binary structures? Documentation seems not that clear regarding this. Also the structure defenition seems to needs compiling, what if there is change of structure at runtime? Can it handle it? Is it possible to for an application to have editable structures and KaiTai to parse them without needing to recompile the whole application (I'm speaking here in a java context).

HNHatesUsers 5y ago

deleted 5y ago

[deleted]

zeckalpha 5y ago

Erlang has some special syntax for this and the people who use it seem to like it. Here’s a paper on adding binary comprehensions (like Python’s list comprehensions): http://user.it.uu.se/~pergu/papers/erlang05.pdf http://user.it.uu.se/~pergu/papers/erlang05.pdf

jchw 5y ago

My project Restruct[1] does Kaitai-like things but also supports serialization. Unfortunately, it only supports Go and only deals with Go struct tags rather than YAML manifests. Still, it totally can be used for serialization. I use it to sketch out quick projects against arbitrary binary formats. Two examples: one, parsing PNG headers to implement a quick binwalk-like program for just PNG that looks for the IEND chunk to extract accurately[2], two, a program that splits FL Studio FLP projects by playlist track[3]. I feel like I’ve self-promoted Restruct like four times on Hacker News, and I feel kind of bad because it could use improvements and even some bug fixes and I never seem to get around to it. Oh well. It’s still useful for me, I hope it’s useful for others, too. That said, Kaitai has a fairly clear path towards adding serialization from a design PoV; many things that would be calculated for parsing structures in deserialization could just become checks/assertions in serialization. As an example, checking that an expression calculates out to the expected value would be a reasonable approach. Reversible expressions could be implemented for some cases, too, if you want it to do more of the heavy lifting. I think the biggest obstacle is actually implementing it, and frankly my Scala is too weak to help with such a relatively big undertaking. I’ve also played with the rust nom library, which implements functional programming style parser combinators. It is quite cool how it can express fairly complex grammars and binary formats pretty much equally well, albeit optimizing it effectively requires serious magic that I do not think nom has. (I assume in Haskell, the same thing can be done with mind-boggling optimization power.) It also doesn’t handle serialization, but it’s still worth noting for robust deserialization. [1]: https://github.com/go-restruct/restruct https://github.com/go-restruct/restruct [2]: https://github.com/jchv/pngextract https://github.com/jchv/pngextract [3]: https://github.com/jchv/flsplit https://github.com/jchv/flsplit

manish_gill 5y ago

http://www.jemarch.net/poke http://www.jemarch.net/poke

catern 5y ago

I've long looked for something like this for my day job, which involves implementing many different custom binary network protocols. One new project in this space is poke: http://www.jemarch.net/poke http://www.jemarch.net/poke It can describe arbitrary formats for both reading and writing, and it's usable programmatically, so I think it can solve the problem!

olodus 5y ago

Another one from google: flatbuffers. We use this at work mostly to be able to be ISSU proof in out data models. It has some useful features for this like checking that the evolution of a data model is in line with ISSU compliancy guidelines, which we check in CI.

sandreas 5y ago

Maybe kaitai struct is what you are looking for? ;) https://kaitai.io/ https://kaitai.io/