8 ms·

Fq: Jq for Binary Formats

rahimiali 5y ago
It's interesting to see how they introduce a new binary format in their catalogue. I was expected to find a domain specific language to define the grammar of binary bitstreams, maybe as a context free grammar. Instead, they built a nice library of routines that helps them design custom parsers by hand for each new format.
- deleted 5y ago
  [deleted]
- mongol 5y ago
  I wonder if it would support also non-binary formats. The tool could evolve to handle json, yaml, xml, ini etc...
  - wwader 5y ago
    Hi, fq actually do support json so other similar text formats could work the same way. But it's currently implemented in a big hacky way, it's just a big blob that happens to be work as normal JSON. I've done some attempts at implementing it as a normal fq decoder but it's hard to figure out a way to represent the whitspace between values etc, ends up very clunky or not very user friendly. Any suggestions are very welcomed.
AzzieElbab 5y ago
I am sure it is awesome but the name of this utility is somewhat unfortunate
- CyberShadow 5y ago
  This related project, on the other hand, embraced it (for better or for worse): https://github.com/jzelinskie/faq https://github.com/jzelinskie/faq
- zeroimpl 5y ago
  I’d have gone with bq myself
  - fullstop 5y ago
    I know that jokes don't go over well here, but... I'd make an improved one, one which is better. And I'd name it bbq.
    - hinkley 5y ago
      And I will write a fuzzer for bbq called omgwtf.
  - dinvlad 5y ago
    D'oh! Big ol BigQuery CLI would like to have a word
    - snidane 5y ago
      then bbq.
- pokstad 5y ago
  Don’t see the issue. I naturally pronounce this “eff-queue”. You’ve really got to work hard to make it vulgar.
- aaron_m04 5y ago
  Yeah, two syllables is too long. Let's just sound it out...
  - philsnow 5y ago
    ... 'feek' ? If it were 'fk' sure, but the Q on the end makes me think of all the English words that come from French and end in 'ique', like technique. 'fq' looks like 'feek' to me.
- coldtea 5y ago
  Only in puritan cultures
  - AzzieElbab 5y ago
    Or passive aggressive…
- wwader 5y ago
  :) i didn't choose it to be provocative or so, apologies if that is the case. I've always pronounced jq yay-queue so fq is eff-queue for me. Also f and q can be written with one hand on qwerty which is nice and quick
pxeger1 5y ago
Well fq too!
zx2c4 5y ago
Relatedly, check out GNU Poke: http://www.jemarch.net/poke http://www.jemarch.net/poke
- pabs3 5y ago
  Also Kaitai Struct: https://kaitai.io/ https://kaitai.io/ And the other things mentioned in the fq README: https://github.com/HexFiend/HexFiend https://github.com/HexFiend/HexFiend https://github.com/binspector/binspector https://github.com/binspector/binspector
  - pdpi 5y ago
    It’s interesting how people work. Seeing that section of the readme with a laundry list of alternatives made me want to try fq even more. It tells you that the author actually cares about the problem space.
    - wwader 5y ago
      Hi! yes i'm very interested in binary analysis and decoders in general and fq was not built with the intention to compete or replace anything. I usually use fq together with lots of other tools, they all fill different purposes. The more the merrier!
    - pabs3 5y ago
      The listing of alternatives in the README really should be standard practice for open source projects. OTOH, some maintainers don't like to do that when they haven't evaluated the projects. Perhaps they could still add them with a disclaimer though.
  - dloss 5y ago
    Nice! Some other tools and parsers: https://github.com/dloss/binary-parsing https://github.com/dloss/binary-parsing
    - wwader 5y ago
userbinator 5y ago
For something that is supposed to be an analog of jq, there is a notable omission from the list of formats: ASN.1.
- naikrovek 5y ago
  note that the tool linked is for binary files. ASN.1 is text, isn't it?
  - stormbrew 5y ago
    I assume they meant the binary encodings of asn.1 like BER/DER.
    - crehn 5y ago
      And tangentially PEM, which is Base64-encoded DER.
      - fanf2 5y ago
        That’s what I thought until recently, but it turns out that PEM refers to just base64 wrapped with ——BEGIN—— and ——END—— lines, and the encapsulated data does not have to be DER. https://datatracker.ietf.org/doc/html/rfc7468 https://datatracker.ietf.org/doc/html/rfc7468
- dikei 5y ago
  Can't blame him. ASN.1 is one of the most complicated binary formats, with so many encoding rules that there's no free decoder that can process them all.
  - chii 5y ago
    Except for interoperability with existing systems, is there any reason why anyone would use this ASN.1 protocol/format? What's it good for?
blondin 5y ago
this is quite an interesting project! combining kaitai structs or similar with the command line. however, i am a little disappointed that the jq syntax was chosen. jq has a very non-intuitive syntax. there are more intuitive query syntaxes out there. (linq or even basic sql come to mind.)
- nine_k 5y ago
  Linq is a bit more wordy, and SQL is sadly not very composable.
- wwader 5y ago
  Yes i can empathize with finding jq hard to understand, it's quite different and took a while to grasp. The reason i choosed it anyway was that after prototyping some common type of queries i would like to do (basic value access in deep structures, multiple recursive traverse with filtering, transform objects and arrays) in various languages, jq was more or less then only one that felt terse enough. Also i think it's quite nice that you can output to JSON and then load into whatever language or environment you want Maybe there are some alternatives i should look at?
lido 5y ago
> This project would not have been possible without itchyny's jq implementation gojq. Another approach is to take the convert binary to object part of your code, output that as JSON on stdout and feed that into jq. Basically, a binary front end + jq = fq
- jrockway 5y ago
  I wrote a small script to convert CSVs to JSON strictly to use jq on the output. Querying things like your GCP bill with jq is quite enjoyable. gojq is also nice. I work with a lot of structured logs and wrapped jq with a little bit of format-understanding and output sugar to make looking at and analyzing such logs an enjoyable experience: https://github.com/jrockway/json-logs https://github.com/jrockway/json-logs
  - darrenf 5y ago
    > I wrote a small script to convert CSVs to JSON strictly to use jq on the output Note that you can use jq to consume simple CSVs (and produce them) without anything else. There’s an entry in the cookbook wiki https://github.com/stedolan/jq/wiki/Cookbook#convert-a-csv-file-with-headers-to-json https://github.com/stedolan/jq/wiki/Cookbook#convert-a-csv-f... - I posted some usage examples a few months back https://news.ycombinator.com/item?id=27379423 https://news.ycombinator.com/item?id=27379423
  - awild 5y ago
    Miller Csv can process json in record format and has a much saner DSL in my experience. https://github.com/johnkerl/miller https://github.com/johnkerl/miller
- cglong 5y ago
  I wish this was only the binary front end so I could pick my parser (e.g. PowerShell). I see fq seems to support sending the whole JSON to stdout; I wonder if there's a way to make this the default behavior: # JSON for whole file fq tovalue file
- donio 5y ago
buryat 5y ago
why not bq?
chris_wot 5y ago
So is this meant for any binary formats?
- linuxdude314 5y ago
  Of course not. It can only work on formats that the team head written parsers for.
  - chris_wot 5y ago
    I ask because I’d be interested in helping write an EMF+ filter
- kitd 5y ago
  Supported formats: https://github.com/wader/fq/blob/master/doc/formats.md https://github.com/wader/fq/blob/master/doc/formats.md They should probably make this a bit more prominent. It's an impressive list for a new project.
  - heinrich5991 5y ago
    Permanent link (press 'y' on any Github link): https://github.com/wader/fq/blob/eb4a6fdbd6ef3a09fc59802e96e108b993ac84ca/doc/formats.md https://github.com/wader/fq/blob/eb4a6fdbd6ef3a09fc59802e96e...
  - wwader 5y ago
    Thanks! any suggestion how to make it more prominent?
gnubison 5y ago
Interesting project. Unfortunate that its name conflicts with one of nq’s executables (https://github.com/leahneukirchen/nq https://github.com/leahneukirchen/nq), but I’m not sure anything can be done about it.
- benatkin 5y ago
  IMO ones that only have one non-prefixed executable take precedence over ones that don't, even when the one with multiple non-prefixed executables is older.
Cayde-6 5y ago
Not to be confused with https://github.com/circonus-labs/fq https://github.com/circonus-labs/fq, the message queue.
deleted 5y ago
[deleted]
nitrogen 5y ago
This looks incredible. I'm on my phone so I haven't tried this, but it looks like this supports slicing into MP3 bitstreams? That would have saved me a month of research and tons of development back in 2013.
- wwader 5y ago
  Hi, it depends a bit, if the mp3 stream uses bit reservoir it might be tricky to to "pure" remuxing with any tool. fq:s mp3_frame decoder do try to know what bits are parts of the current frame or part of a future frame but not sure how much that helps. If the stream does not use reservoir you should be able to slice using fq '.frames[100:200][]' file.mp3 > sliced.mp3 or something similar.
hawk_ 5y ago
It says it supports protobuf. Is there a protobuf file format i.e. for multiple records or do they mean a single protobuf record file?
- sandermvanvliet 5y ago
  Typically people separate protobuf messages by a length value of the next message. Perhaps that’s what they did. Also protobuf can contain embedded messages or even just the binary representation of a list of embedded messages.
  - wwader 5y ago
    Hi, currently the protobuf support can either decode the wire-format or in some cases a format decoder uses protobuf as subformat and passes it a "schema" so it can do some more fancy decoding. But yes it would be interesting adding support for reading protobuf schemas somehow.
    - Cloudef 5y ago
      I wrote protobuf parser in ragel for work.. its still used to replace reflection as c++ protobuf implementation explodes our binaries to huge sizes.
arendtio 5y ago
I am alternating between WOW and wtf. Pretty cool stuff. I just wonder how on earth you want to be able to support all the binary formats out there. I mean, jq supports json, not all structured text data, like json, xml, csv, ini, ...
- luto 5y ago
  apparently they went ahead and implemented a bunch of them in Go: https://github.com/wader/fq/tree/master/format https://github.com/wader/fq/tree/master/format
  - wwader 5y ago
    Hi, i can give some background how i ended up with go instead of using something more declarative. Maybe 1.5 years ago i start to prototype different approaches for what query language to use (sql, jsonpath, my own basic jq version and few more) and what language to implement decoders in (lisp, kaitai, tcl, "scripted" go, normal go and some more). What i found was that for my use cases, detailed parsing of big media files, anything scripted was just too slow. I did look into translating kaitai etc into something compiled which would probably be fast, but next on my list was i wanted was to be able to select and decode subformats in quite complicated ways (like mp4 samples), flexible ways of demux and join blob to decode, calculate checksums, samples counts in various way. All felt clunky or hard to fit into a purely declarative description. But i was also biased towards go as i had good experience using it and know that it would probably be fast enough (turn out smart memory usage is probably the main speed factor for fq when you keep track of lots of things). Also it would provide good tooling like IDE support, refactoring (gopls gofmt -r, rf) and it's a reasonably strongly typed language i think. Last but not least the quick build times really fits my way of working, usually use lots of watchexec etc. For query language i didn't prototype much, i know i really wanted jq as i had already used it extensively and know it was very powerful and had a terse syntax when working with structured data. I had some ideas of maybe using the C-version of jq via bindings or somehow let fq be tool that you used like this 'fq file | jq ... | fq' but it just felt strange and not very user friendly. Then i found gojq and i just felt that i have to make it work somehow, even if it would require lots of hard work and change to it (see https://github.com/wader/gojq/commits/fq https://github.com/wader/gojq/commits/fq, the JQValue change it probably to most interesting and support or custom iterators/functions that has been merged). And it turned out much better than i would expected, large parts becuse gojq's code is very nice and author has been very helpful. There is more things i would like that talk about but i think this is long enough for now :) But all that said i think you could use kaitai or something similar together with fq:s decode API if you want. I also have some ideas and plans on supporting writing deocders in jq, hopefully will get some time for that next year.
lindig 5y ago
I’d like to see support for FIT files as they are emitted by Garmin fitness devices. It’s a clever binary format that in-stream defines the format of records which then contain the actual measurements which may be scaled for more compact representation. These multiple layers make the format not obvious to parse but the tool supports already an impressive list of formats that probably use similar techniques.
rav 5y ago
Does this support out-of-tree format decoders? From an initial glance it looks like all decoders are in-tree and written in golang. We have a lot of internal binary formats at $WORK that I would like to use this on...
- masklinn 5y ago
  Seems unlikely since the decoders are defined not just in-tree but in "host" code, the definitions are neither data-driven nor a DSL. So it would require some sort of native (Go) plugins system, which I understand is about as bad as in Rust owing to there being no standard ABI (or plugins system for that matter). therefore the way to have bespoke / internal formats would be to maintain an internal fork of the tool.
  - akavel 5y ago
    "(...) some sort of native (Go) plugins system (...)" See: https://pkg.go.dev/plugin https://pkg.go.dev/plugin
- wwader 5y ago
  Hi! yes it's kind of support but in a very go:ish at the moment. You can use fq as submodule, import/register your own format decoders and then run cli.Main. More or less what https://github.com/wader/fq/blob/master/fq.go https://github.com/wader/fq/blob/master/fq.go does. I have private version of fq for work with some proprietary formats that does this and it works great. One issue is that the decoder and format API might change, not sure i can give any stability guarantees atm and i want to evolve a bit more. Also it would be great to be able to hook into existing formats more in some way. In the future i hope to support writing decoders in jq and or support some declarative format like kaitai.
ape4 5y ago
Hopefully json will be superseded someday. Cool tool.
- wwader 5y ago
  Interestingly fq works by being kind of a superset of JSON/jq. It has types that can behave as jq values when needs but then with special functions or key accessors can be something else.
leshenka 5y ago
This looks like wireshark's panel for inspecting packets

It's interesting to see how they introduce a new binary format in their catalogue. I was expected to find a domain specific language to define the grammar of binary bitstreams, maybe as a context free grammar. Instead, they built a nice library of routines that helps them design custom parsers by hand for each new format.

deleted 5y ago
[deleted]
mongol 5y ago
I wonder if it would support also non-binary formats. The tool could evolve to handle json, yaml, xml, ini etc...
- wwader 5y ago
  Hi, fq actually do support json so other similar text formats could work the same way. But it's currently implemented in a big hacky way, it's just a big blob that happens to be work as normal JSON. I've done some attempts at implementing it as a normal fq decoder but it's hard to figure out a way to represent the whitspace between values etc, ends up very clunky or not very user friendly. Any suggestions are very welcomed.

AzzieElbab 5y ago

I am sure it is awesome but the name of this utility is somewhat unfortunate

CyberShadow 5y ago
This related project, on the other hand, embraced it (for better or for worse): https://github.com/jzelinskie/faq https://github.com/jzelinskie/faq
zeroimpl 5y ago
I’d have gone with bq myself
- fullstop 5y ago
  I know that jokes don't go over well here, but... I'd make an improved one, one which is better. And I'd name it bbq.
  - hinkley 5y ago
    And I will write a fuzzer for bbq called omgwtf.
- dinvlad 5y ago
  D'oh! Big ol BigQuery CLI would like to have a word
  - snidane 5y ago
    then bbq.
pokstad 5y ago
Don’t see the issue. I naturally pronounce this “eff-queue”. You’ve really got to work hard to make it vulgar.
aaron_m04 5y ago
Yeah, two syllables is too long. Let's just sound it out...
- philsnow 5y ago
  ... 'feek' ? If it were 'fk' sure, but the Q on the end makes me think of all the English words that come from French and end in 'ique', like technique. 'fq' looks like 'feek' to me.
coldtea 5y ago
Only in puritan cultures
- AzzieElbab 5y ago
  Or passive aggressive…
wwader 5y ago
:) i didn't choose it to be provocative or so, apologies if that is the case. I've always pronounced jq yay-queue so fq is eff-queue for me. Also f and q can be written with one hand on qwerty which is nice and quick

pxeger1 5y ago

Well fq too!

zx2c4 5y ago

Relatedly, check out GNU Poke: http://www.jemarch.net/poke http://www.jemarch.net/poke

pabs3 5y ago
Also Kaitai Struct: https://kaitai.io/ https://kaitai.io/ And the other things mentioned in the fq README: https://github.com/HexFiend/HexFiend https://github.com/HexFiend/HexFiend https://github.com/binspector/binspector https://github.com/binspector/binspector
- pdpi 5y ago
  It’s interesting how people work. Seeing that section of the readme with a laundry list of alternatives made me want to try fq even more. It tells you that the author actually cares about the problem space.
  - wwader 5y ago
    Hi! yes i'm very interested in binary analysis and decoders in general and fq was not built with the intention to compete or replace anything. I usually use fq together with lots of other tools, they all fill different purposes. The more the merrier!
  - pabs3 5y ago
    The listing of alternatives in the README really should be standard practice for open source projects. OTOH, some maintainers don't like to do that when they haven't evaluated the projects. Perhaps they could still add them with a disclaimer though.
- dloss 5y ago
  Nice! Some other tools and parsers: https://github.com/dloss/binary-parsing https://github.com/dloss/binary-parsing
  - wwader 5y ago

userbinator 5y ago

For something that is supposed to be an analog of jq, there is a notable omission from the list of formats: ASN.1.

naikrovek 5y ago
note that the tool linked is for binary files. ASN.1 is text, isn't it?
- stormbrew 5y ago
  I assume they meant the binary encodings of asn.1 like BER/DER.
  - crehn 5y ago
    And tangentially PEM, which is Base64-encoded DER.
    - fanf2 5y ago
      That’s what I thought until recently, but it turns out that PEM refers to just base64 wrapped with ——BEGIN—— and ——END—— lines, and the encapsulated data does not have to be DER. https://datatracker.ietf.org/doc/html/rfc7468 https://datatracker.ietf.org/doc/html/rfc7468
dikei 5y ago
Can't blame him. ASN.1 is one of the most complicated binary formats, with so many encoding rules that there's no free decoder that can process them all.
- chii 5y ago
  Except for interoperability with existing systems, is there any reason why anyone would use this ASN.1 protocol/format? What's it good for?

blondin 5y ago

this is quite an interesting project! combining kaitai structs or similar with the command line. however, i am a little disappointed that the jq syntax was chosen. jq has a very non-intuitive syntax. there are more intuitive query syntaxes out there. (linq or even basic sql come to mind.)

nine_k 5y ago
Linq is a bit more wordy, and SQL is sadly not very composable.
wwader 5y ago
Yes i can empathize with finding jq hard to understand, it's quite different and took a while to grasp. The reason i choosed it anyway was that after prototyping some common type of queries i would like to do (basic value access in deep structures, multiple recursive traverse with filtering, transform objects and arrays) in various languages, jq was more or less then only one that felt terse enough. Also i think it's quite nice that you can output to JSON and then load into whatever language or environment you want Maybe there are some alternatives i should look at?

lido 5y ago

> This project would not have been possible without itchyny's jq implementation gojq. Another approach is to take the convert binary to object part of your code, output that as JSON on stdout and feed that into jq. Basically, a binary front end + jq = fq

jrockway 5y ago
I wrote a small script to convert CSVs to JSON strictly to use jq on the output. Querying things like your GCP bill with jq is quite enjoyable. gojq is also nice. I work with a lot of structured logs and wrapped jq with a little bit of format-understanding and output sugar to make looking at and analyzing such logs an enjoyable experience: https://github.com/jrockway/json-logs https://github.com/jrockway/json-logs
- darrenf 5y ago
  > I wrote a small script to convert CSVs to JSON strictly to use jq on the output Note that you can use jq to consume simple CSVs (and produce them) without anything else. There’s an entry in the cookbook wiki https://github.com/stedolan/jq/wiki/Cookbook#convert-a-csv-file-with-headers-to-json https://github.com/stedolan/jq/wiki/Cookbook#convert-a-csv-f... - I posted some usage examples a few months back https://news.ycombinator.com/item?id=27379423 https://news.ycombinator.com/item?id=27379423
- awild 5y ago
  Miller Csv can process json in record format and has a much saner DSL in my experience. https://github.com/johnkerl/miller https://github.com/johnkerl/miller
cglong 5y ago
I wish this was only the binary front end so I could pick my parser (e.g. PowerShell). I see fq seems to support sending the whole JSON to stdout; I wonder if there's a way to make this the default behavior: # JSON for whole file fq tovalue file
donio 5y ago

buryat 5y ago

why not bq?

chris_wot 5y ago

So is this meant for any binary formats?

linuxdude314 5y ago
Of course not. It can only work on formats that the team head written parsers for.
- chris_wot 5y ago
  I ask because I’d be interested in helping write an EMF+ filter
kitd 5y ago
Supported formats: https://github.com/wader/fq/blob/master/doc/formats.md https://github.com/wader/fq/blob/master/doc/formats.md They should probably make this a bit more prominent. It's an impressive list for a new project.
- heinrich5991 5y ago
  Permanent link (press 'y' on any Github link): https://github.com/wader/fq/blob/eb4a6fdbd6ef3a09fc59802e96e108b993ac84ca/doc/formats.md https://github.com/wader/fq/blob/eb4a6fdbd6ef3a09fc59802e96e...
- wwader 5y ago
  Thanks! any suggestion how to make it more prominent?

gnubison 5y ago

Interesting project. Unfortunate that its name conflicts with one of nq’s executables (https://github.com/leahneukirchen/nq https://github.com/leahneukirchen/nq), but I’m not sure anything can be done about it.

benatkin 5y ago
IMO ones that only have one non-prefixed executable take precedence over ones that don't, even when the one with multiple non-prefixed executables is older.

Cayde-6 5y ago

Not to be confused with https://github.com/circonus-labs/fq https://github.com/circonus-labs/fq, the message queue.

deleted 5y ago

[deleted]

nitrogen 5y ago

This looks incredible. I'm on my phone so I haven't tried this, but it looks like this supports slicing into MP3 bitstreams? That would have saved me a month of research and tons of development back in 2013.

wwader 5y ago
Hi, it depends a bit, if the mp3 stream uses bit reservoir it might be tricky to to "pure" remuxing with any tool. fq:s mp3_frame decoder do try to know what bits are parts of the current frame or part of a future frame but not sure how much that helps. If the stream does not use reservoir you should be able to slice using fq '.frames[100:200][]' file.mp3 > sliced.mp3 or something similar.

hawk_ 5y ago

It says it supports protobuf. Is there a protobuf file format i.e. for multiple records or do they mean a single protobuf record file?

sandermvanvliet 5y ago
Typically people separate protobuf messages by a length value of the next message. Perhaps that’s what they did. Also protobuf can contain embedded messages or even just the binary representation of a list of embedded messages.
- wwader 5y ago
  Hi, currently the protobuf support can either decode the wire-format or in some cases a format decoder uses protobuf as subformat and passes it a "schema" so it can do some more fancy decoding. But yes it would be interesting adding support for reading protobuf schemas somehow.
  - Cloudef 5y ago
    I wrote protobuf parser in ragel for work.. its still used to replace reflection as c++ protobuf implementation explodes our binaries to huge sizes.

arendtio 5y ago

I am alternating between WOW and wtf. Pretty cool stuff. I just wonder how on earth you want to be able to support all the binary formats out there. I mean, jq supports json, not all structured text data, like json, xml, csv, ini, ...

luto 5y ago
apparently they went ahead and implemented a bunch of them in Go: https://github.com/wader/fq/tree/master/format https://github.com/wader/fq/tree/master/format
- wwader 5y ago
  Hi, i can give some background how i ended up with go instead of using something more declarative. Maybe 1.5 years ago i start to prototype different approaches for what query language to use (sql, jsonpath, my own basic jq version and few more) and what language to implement decoders in (lisp, kaitai, tcl, "scripted" go, normal go and some more). What i found was that for my use cases, detailed parsing of big media files, anything scripted was just too slow. I did look into translating kaitai etc into something compiled which would probably be fast, but next on my list was i wanted was to be able to select and decode subformats in quite complicated ways (like mp4 samples), flexible ways of demux and join blob to decode, calculate checksums, samples counts in various way. All felt clunky or hard to fit into a purely declarative description. But i was also biased towards go as i had good experience using it and know that it would probably be fast enough (turn out smart memory usage is probably the main speed factor for fq when you keep track of lots of things). Also it would provide good tooling like IDE support, refactoring (gopls gofmt -r, rf) and it's a reasonably strongly typed language i think. Last but not least the quick build times really fits my way of working, usually use lots of watchexec etc. For query language i didn't prototype much, i know i really wanted jq as i had already used it extensively and know it was very powerful and had a terse syntax when working with structured data. I had some ideas of maybe using the C-version of jq via bindings or somehow let fq be tool that you used like this 'fq file | jq ... | fq' but it just felt strange and not very user friendly. Then i found gojq and i just felt that i have to make it work somehow, even if it would require lots of hard work and change to it (see https://github.com/wader/gojq/commits/fq https://github.com/wader/gojq/commits/fq, the JQValue change it probably to most interesting and support or custom iterators/functions that has been merged). And it turned out much better than i would expected, large parts becuse gojq's code is very nice and author has been very helpful. There is more things i would like that talk about but i think this is long enough for now :) But all that said i think you could use kaitai or something similar together with fq:s decode API if you want. I also have some ideas and plans on supporting writing deocders in jq, hopefully will get some time for that next year.

lindig 5y ago

I’d like to see support for FIT files as they are emitted by Garmin fitness devices. It’s a clever binary format that in-stream defines the format of records which then contain the actual measurements which may be scaled for more compact representation. These multiple layers make the format not obvious to parse but the tool supports already an impressive list of formats that probably use similar techniques.

rav 5y ago

Does this support out-of-tree format decoders? From an initial glance it looks like all decoders are in-tree and written in golang. We have a lot of internal binary formats at $WORK that I would like to use this on...

masklinn 5y ago
Seems unlikely since the decoders are defined not just in-tree but in "host" code, the definitions are neither data-driven nor a DSL. So it would require some sort of native (Go) plugins system, which I understand is about as bad as in Rust owing to there being no standard ABI (or plugins system for that matter). therefore the way to have bespoke / internal formats would be to maintain an internal fork of the tool.
- akavel 5y ago
  "(...) some sort of native (Go) plugins system (...)" See: https://pkg.go.dev/plugin https://pkg.go.dev/plugin
wwader 5y ago
Hi! yes it's kind of support but in a very go:ish at the moment. You can use fq as submodule, import/register your own format decoders and then run cli.Main. More or less what https://github.com/wader/fq/blob/master/fq.go https://github.com/wader/fq/blob/master/fq.go does. I have private version of fq for work with some proprietary formats that does this and it works great. One issue is that the decoder and format API might change, not sure i can give any stability guarantees atm and i want to evolve a bit more. Also it would be great to be able to hook into existing formats more in some way. In the future i hope to support writing decoders in jq and or support some declarative format like kaitai.

ape4 5y ago

Hopefully json will be superseded someday. Cool tool.

wwader 5y ago
Interestingly fq works by being kind of a superset of JSON/jq. It has types that can behave as jq values when needs but then with special functions or key accessors can be something else.

leshenka 5y ago

This looks like wireshark's panel for inspecting packets