26 ms·
Fq: Jq for Binary Formats
- deleted 3y ago[deleted]
- candiodari 3y agoLOVE the name.
- dotancohen 3y agoI didn't even think of that until you mentioned it!
- nonethewiser 3y agoReminds me of this emphatically best middle school Fuqing #1 Middle School https://g.co/kgs/HPtt5n https://g.co/kgs/HPtt5n
- jacknews 3y agothe name kind of sucks, why 'f' q? F for ... 'FU' ('teenage snigger'). It should be 'bq', binary query, after 'jq' json query. Cool project none-the-less. The comment about 'programmatic documentation' of binary formats is very interesting, maybe some kind of 'binary description markup' could be part of this?
- pcthrowaway 3y agoNo, FQ!!
- jxf 3y agoThis looks beautiful. When there's more data I bet it's going to be a great tool to hook up to LLMs. "Draw the first frame of this video using fq."
- fulafel 3y agoPreviously (2021): https://news.ycombinator.com/item?id=29657094 https://news.ycombinator.com/item?id=29657094
- dang 3y agoThanks! Macroexpanded: Fq: Jq for Binary Formats - https://news.ycombinator.com/item?id=29657094 https://news.ycombinator.com/item?id=29657094 - Dec 2021 (81 comments)
- majkinetor 3y agoAwesome. It would be good if some form of externally plugable binary format specification is doable in the future. As far as I can see, if the binary format is not supported OTB, you can't use this tool.
- nyberg 3y agohttp://www.jemarch.net/poke http://www.jemarch.net/poke might be interesting in that case.
- wakeupcall 3y agoI second poke. It's an amazing tool for debugging in general. It's relatively rare to look into standardized binary formats (you'll likely look directly into a library at that point), unless you're writing a writer/parser/decoder yourself and need to double-check the output. When developing with general binary data in mind, poke is much more useful.
- notpushkin 3y agoKaitai Struct might be a good choice for that: https://kaitai.io/ https://kaitai.io/
- theozaurus 3y agoI've found Kaitai struct to be an absolute joy to use on things like EEPROM dumps from car computers.
- leoh 3y agoHas anyone found anything incredible for arbitrary UTF-8 data?
- dotancohen 3y agoHonestly, VIM.
- usr1106 3y agoFOSDEM presentation by the author earlier this year: https://fosdem.org/2023/schedule/event/bintools_fq/ https://fosdem.org/2023/schedule/event/bintools_fq/
- sillysaurusx 3y agofqing finally. It always seemed strange that there wasn’t any central database of binary parsers that everyone could contribute to. Nearly every file format is fully documented, but none of the docs are programmatic. I was trying to rip some sounds from a wii game called Rhythm Heaven, and it’s ridiculous how primitive the tech is. By that I mean the programming community’s tech. If you want to extract some assets, you’d better be running Windows, and you’ll need to download some random exe from mediafire made by Jared, a 13yo that coded the extractor in C in his spare time. This is only a very slight exaggeration; Windows being a requirement isn’t. Hopefully projects like this will standardize all binary formats once and for all. Actually, this is a good opportunity to ask: how would one contribute a binary parser to fq? If I wrote one for wii sound files, can I just submit a PR or is there some other process? EDIT: https://github.com/wader/fq/blob/master/doc/dev.md https://github.com/wader/fq/blob/master/doc/dev.md documents the development side of things. I’m more interested in the project itself — if someone puts in the work to make a decoder for an obscure binary format, will it get merged (assuming it’s high quality) or is this only for popular formats?
- andoma 3y agoBeing a long time personal friend with the author I can assure you the more obscure the better :-) His interest in esoteric things and solutions are "well documented" if you browse around his github repos. Some examples: https://github.com/wader/jqjq https://github.com/wader/jqjq https://github.com/wader/catgolf https://github.com/wader/catgolf
- sillysaurusx 3y agoWonderful, thank you so much! Such a cool project.
- Akronymus 3y agoI am RE-ing some binary file formats for games. Should those be contributed as well?
- wwader 3y agofun fact: fq is kind of the spiritual predecessor of https://github.com/wader/flac.tcl https://github.com/wader/flac.tcl, you can see traces of it in flac_frame.go in fq, was used to prototype some things :)
- pabs3 3y agoI love that this includes a section in the README about other tools that are similar or related to fq. Every open source project should list its competitors.
- tough 3y agoRight? Not doing so is either disingenuous or amateurish
- tpoacher 3y agoAgreed. I try to do the same with my stuff. Honest, transparent market research / placement improves your offering, rather than diminishes it.
- wwader 3y agoThanks for the kind words! without the other tools i don't think fq would even exist, they inspired and show what was possible.
- suavesito 3y agoNot only that, the README has a section called _Hopes_, the second bullet point being > Inspire people to create similar tools.
- wwader 3y agoThanks for the kind words! they inspired me so i hope i can inspire others
- SkyPuncher 3y agoI don’t even necessarily see listing alternatives as competition. Often alternatives solve overlapping, but different problems.
- el_oni 3y agoI wouldnt even say the are necessarily competitors. More like a flathead screwdriver to compliment your phillips in most of those cases
- eftel 3y agoI’m currently trying to make sense of a binary file that is used by a proprietary program to import data. The file is generated on a server out of my control but I’m able to see that some kind of key is being sent alongside the data. (To encrypt it?) How would one approach something like this? Where could I look for freelancers who are able to help with this?
- scrollaway 3y agoThe most straightforward way will be to reverse engineer the program that imports the data. Look for reverse engineer freelancers. Many of them in the video game space.
- xmcqdpt2 3y agoDo you have access to the program that reads the data? If so, you can use a debugger to step through the parser for the file, even if symbols are stripped [1]. You can breakpoint on syscalls, such as when the file gets opened [2] and then step through and look around memory for the decrypted version. If you have an idea of what the file should contain you can probably identify patterns this way. I'm not an expert on this topic at all though. [1] Of course you then have less information but it's still possible to see the assembly while the file gets parsed. See for example, http://felix.abecassis.me/2012/08/gdb-debugging-stripped-binaries/ http://felix.abecassis.me/2012/08/gdb-debugging-stripped-bin... [2] https://sourceware.org/gdb/onlinedocs/gdb/Set-Catchpoints.html#Set-Catchpoints https://sourceware.org/gdb/onlinedocs/gdb/Set-Catchpoints.ht...
- xvilka 3y agoFor this kind of task, using low-level debugger tools is probably better. Rizin[1][2]/Cutter[3][4] could help. We also have GSoC participant this year who works hard on improving debuginfo and debugging support[5]. I personally also like Binary Ninja, they recently made their debugger stable enough[6]. [1] https://rizin.re/ https://rizin.re/ [2] https://github.com/rizinorg/rizin https://github.com/rizinorg/rizin [3] https://cutter.re/ https://cutter.re/ [4] https://github.com/rizinorg/cutter https://github.com/rizinorg/cutter [5] https://rizin.re/posts/gsoc-2023-announcement/ https://rizin.re/posts/gsoc-2023-announcement/ [6] https://binary.ninja/2023/05/03/3.4-finally-freed.html#debugger-improvements https://binary.ninja/2023/05/03/3.4-finally-freed.html#debug...
- NackerHughes 3y agoIf the ‘j’ in `jq` stands for ‘JSON’, what does the ‘f’ stand for?
- zulu-inuoe 3y agoGood question. Maybe 'format' or just 'file'?
- jensenbox 3y agoIt stands for the sound you make when you cat a binary file.
- wwader 3y agoHi author here, i think format mostly but also i thought about fq as file(1) on steroids in the beginning :)
- pmoriarty 3y agoToo bad that jq has such a shitty, convoluted syntax. Could have definitely chosen one of the many alternatives to jq that also worked with JSON but did so in a much clearer and more elegant way.
- hedora 3y agoCan you name an alternative to jq with better syntax?
- dmoura 3y agoI prefer a SQL-like format. It’s not as complete but it cover most of the day-to-day use cases. Take a look at https://github.com/dcmoura/spyql https://github.com/dcmoura/spyql (I am the author). Congrats on fq!
- wwader 3y agoThanks! i actually experimented a bit with an SQL-like interface for while, dump things into sqlite and use that as query engine. Problem usually was that file formats tend to be mix of array and tree structures more then relational and at least standard SQL is not great for that. Maybe some graph-SQL dialect could work?
- benibela 3y agoThere is XPath 3.1
- xp84 3y agoCommenting to follow because I’m curious what alternatives you mean. I thought a lot of people liked jq and I only just finally got around to installing it, so if there’s a much better way I’d like to hear it.
- wwader 3y agoHi author here. Slide 3 from my FOSDEM talks explains my own very subjects reasons for choosing jq https://fosdem.org/2023/schedule/event/bintools_fq/attachments/slides/5660/export/events/attachments/bintools_fq/slides/5660/fq_fosdem_2023.pdf https://fosdem.org/2023/schedule/event/bintools_fq/attachmen... To sum up: Very CLI friendly syntax, generators/backtracking makes working with tree structures very convenient. But now after writing lots of jq i even thing writing multi line programs in jq to be quite nice :) but i can recommend IDE help, see end of slide 17
- _9hey 3y agoOh nice! We got a public gqui now
- jeffbee 3y agoLooks like not really. The proto support is pretty basic. It can’t print floats and doubles and doesn’t parse groups or packed fields. It doesn’t use a descriptor database so it can only print the field number, not its name, and it can only differentiate nested messages if the user calls ‘|protobuf’ on what is otherwise considered a string.
- wwader 3y agoHi author here. Yeap currently only decodes the "wire" format. Actually it can decode using a schema but not parsed from a proto file, this is used by other formats using protobuf as a subformat then they pass a schema internally using go types. But proper proto schema support would be nice.
- wwader 3y agoHi fq author here. What is gqui? some internal google thingy?
- brabel 3y agoI've written a parser of Java class files (which works in any JVM language as they all compile to the same bytecode format). It was surprisingly easy! Maybe that could be useful to analyse class files in jq??!
- wwader 3y agoHi fq author here! please do if you want, would be a great addition.
- jensenbox 3y agoDid I miss something or is there no Ubuntu or Debian installer? I certainly know how to download a file and add it to my path (or put it in my personal bin directory) but sure would be nice to have a super simple installer. LOL - please do not make a snap or whatever the hell the "cool kids" use. I certainly wouldn't want to advocate continued use of that pattern for utility functions.
- 2h 3y agoThis has got to be the pettiest gripes I have heard in a while. It's already built, you literally just extract and run: https://github.com/wader/fq/releases/tag/v0.6.0 https://github.com/wader/fq/releases/tag/v0.6.0
- dima55 3y agoIt's already in stock Debian.
- fullspectrumdev 3y agoYou can always submit a PR that adds releases support for Debian or RPM packages or whatever format you like. I might consider doing that next week, just realised I have no idea how a RPM is made, and I haven’t thought about making a Debian package in years.
- piperswe 3y agoSince this is written in Go, it's almost trivial to use fpm [1] to generate a variety of packages. Alternatively you can use nfpm [2] if you don't want to have to deal with installing Ruby & a gem. [1]: https://github.com/jordansissel/fpm https://github.com/jordansissel/fpm [2]: https://github.com/goreleaser/nfpm https://github.com/goreleaser/nfpm
- wwader 3y agoHi author here. Releases support as in package .deb and .rpm myself for each relese?
- sillystuff 3y ago
- natch 3y agoIn my mind the ‘f’ here stands for ‘f yeah’
- _pmf_ 3y agoI'd also like to throw https://github.com/WerWolv/ImHex https://github.com/WerWolv/ImHex in the mix here.
- bombolo 3y agoWhat about GNU Poke? http://www.jemarch.net/poke http://www.jemarch.net/poke
- dima55 3y agoThat looks very similar. Any idea what the differences are?
- bombolo 3y agoNo idea. But i know poke has a rather big community
- wwader 3y agoHi fq author here and also acquaintance with the gnu poke author. One difference is that fq is focused in decoding and querying while poke might be more focused on editing and runtime modeling of formats. But we have both inspired each others think. Here is video of me demoing fq and talking to jose (gnu poke author) about it https://www.youtube.com/watch?v=GJOq_b0eb-s https://www.youtube.com/watch?v=GJOq_b0eb-s he also organized the FOSDEM binary tools track.
- qwefqgq3223 3y agoThis is what I like about powershell. It passes objects via pipeline and if you need to query or filter something, you don't need to learn millions of different tolls (jq, xmlstarlet, etc.) - just use programming language features for everything.
- deleted 3y ago[deleted]
- bkyan 3y agoCould this be used as a library in golang?
- wwader 3y agoHi author here. There is no stable API at the moment and it depends a bit what you want to do. If you mean write own private decoders it is possible but i can't guarantee the API will change, see this old twitter thread https://twitter.com/TimMattison/status/1600871136027627521 https://twitter.com/TimMattison/status/1600871136027627521 If you mean using existing decoders and access the result i think you probably do in theory. That is kind of how the interp packages in fq is implemented, it implements a jq interface + some fq bells and whistles using the decode value structure.
- Patrickmi 3y agoAm surprised, it’s been a day now and I haven’t seen RIIR alternative wow