4 ms·

fqing finally. It always seemed strange that there wasn’t any central database of binary parsers that everyone could contribute to. Nearly every file format is

fqing finally. It always seemed strange that there wasn’t any central database of binary parsers that everyone could contribute to. Nearly every file format is fully documented, but none of the docs are programmatic. I was trying to rip some sounds from a wii game called Rhythm Heaven, and it’s ridiculous how primitive the tech is. By that I mean the programming community’s tech. If you want to extract some assets, you’d better be running Windows, and you’ll need to download some random exe from mediafire made by Jared, a 13yo that coded the extractor in C in his spare time. This is only a very slight exaggeration; Windows being a requirement isn’t. Hopefully projects like this will standardize all binary formats once and for all. Actually, this is a good opportunity to ask: how would one contribute a binary parser to fq? If I wrote one for wii sound files, can I just submit a PR or is there some other process? EDIT: https://github.com/wader/fq/blob/master/doc/dev.md https://github.com/wader/fq/blob/master/doc/dev.md documents the development side of things. I’m more interested in the project itself — if someone puts in the work to make a decoder for an obscure binary format, will it get merged (assuming it’s high quality) or is this only for popular formats?

andoma 3y ago
Being a long time personal friend with the author I can assure you the more obscure the better :-) His interest in esoteric things and solutions are "well documented" if you browse around his github repos. Some examples: https://github.com/wader/jqjq https://github.com/wader/jqjq https://github.com/wader/catgolf https://github.com/wader/catgolf
- sillysaurusx 3y ago
  Wonderful, thank you so much! Such a cool project.
- Akronymus 3y ago
  I am RE-ing some binary file formats for games. Should those be contributed as well?
- wwader 3y ago
  fun fact: fq is kind of the spiritual predecessor of https://github.com/wader/flac.tcl https://github.com/wader/flac.tcl, you can see traces of it in flac_frame.go in fq, was used to prototype some things :)
grumbel 3y ago
There is QuickBMS[1], which covers quite a few game related formats. [1] http://aluigi.altervista.org/quickbms.htm http://aluigi.altervista.org/quickbms.htm
realjhol 3y ago
Like wireshark directors but more general
nonethewiser 3y ago
> I was trying to rip some sounds from a wii game called Rhythm Heaven, and it’s ridiculous how primitive the tech is. This seems like a pretty niche need with only some hobbyists motivated enough to work on it. Is there a broader application than your use case? Otherwise I think thats why the existing software for this isnt great.
- sillysaurusx 3y ago
  Yeah, I didn’t mean to sound entitled. I only meant I was excited for projects like fq to shake things up. When I was writing a parser for the sound file I was thinking “hmm… this really feels like duplicate work.” On the other hand, it’s surprising to me that “grab sounds from a wii game” is so niche! My gut felt like it would be slightly more complicated than unzipping a tarball, but my gut didn’t expect it to be a programming challenge worthy of a small competition.
  - foobarian 3y ago
    It’s all fun and games until Nintendo sends in a cease and desist notice
    - sillysaurusx 3y ago
      Seriously. Nintendo is an evil corporation. I was about to write “close to,” but they crossed the line when they sent someone to prison and garnished his wages for the rest of his life.
      - ozim 3y ago
        Not just someone but dude was not ripping or cracking stuff for fun. He made business out of it and what’s worse he added ransomware to scam his “customers”. POS deserved all of it if not more.
        sillysaurusx 3y ago
        If he added ransomware, that’s a bit different. Making a business out of it isn’t that bad (think about it in terms of people going to prison for assault vs merely making some money), but the ransomware would be. Still, garnishing someone’s wages the rest of his life seems out of proportion. But I admit it’s harder to defend someone that made a livelihood out of holding peoples’ data hostage.
wongarsu 3y ago
There is the 010 Editor, at heart a cross-platform scriptable hex editor with a template language [1]. It has a central template repository [2] as well as templates around the internet (e.g. 3, 4). But it being a paid tool means there are fewer template contributions from 13 year olds, which if we are all honest make up the majority of unpaid open source contributions - they simply have more spare time. 1: https://www.sweetscape.com/010editor/ https://www.sweetscape.com/010editor/ 2: https://www.sweetscape.com/010editor/repository/templates/ https://www.sweetscape.com/010editor/repository/templates/ 3: https://github.com/tge-was-taken/010-Editor-Templates/tree/master/templates https://github.com/tge-was-taken/010-Editor-Templates/tree/m... 4: https://wiki.redmodding.org/cyberpunk-2077-modding/modding-know-how/modding-tools/010-editor https://wiki.redmodding.org/cyberpunk-2077-modding/modding-k...
- overengineer 3y ago
  What is wrong with contributions from 13 year olds?
  - xp84 3y ago
    I don’t think it was implied that there’s anything bad, I read it as the opposite.
  - oxygen_crisis 3y ago
    There aren't enough of them, because the tool costs money, and 13-year-olds can't spend money on software as readily as adults.
- nness 3y ago
  010editor's templating language looks very interesting (and something I had been trying to accomplish with my own mixed-bag of tools when reverse-engineering). I suppose as a hobbyist, the price is a hard one to get over...
- GuB-42 3y ago
  Do 13 year olds care about licenses and pay for the software they use? I certainly didn't when I was 13. And 010 editor is about as well protected as WinRAR. I have a hard time imagining a 13 year old using that particular tool buying it. In fact there is a good chance it is used by the crackers themselves.
BeefWellington 3y ago
Binwalk is my go-to for that kind of thing usually.
- fullspectrumdev 3y ago
  Try unblob sometime, it’s a more modern, maintained alternative (not a fork). A company called OneKey that do some firmware security stuff maintain it, and generally it’s pretty good.
kazinator 3y ago
> It always seemed strange that there wasn’t any central database of binary parsers that everyone could contribute to. file utility with /etc/magic database
- loeg 3y ago
  Yeah, that was my thought as well, although I don't know that file/magic get into much structure beyond identifying the file format.
stelonix 3y ago
I used to be from the romhacking community back in the 2000s and due to usage of Windows, open source/foss wasn't even known to most people. The culture of Windows programmers is way more focused on freeware/binaries. Still waiting to this day for FuSoYa to release the source code of Lunar Magic. About a central database of binary parsers, I've been wanting this for ages too. The closest I ever found was augeas, but that's for configuration files.
- xp84 3y ago
  Yeah that’s a mysterious one. Such an incredible achievement, and it enabled so, so much creativity, and free (as in beer) to all as far as I know. I hope FuSoYa does open source it someday.
- marlin 3y ago
  > About a central database of binary parsers, I've been wanting this for ages too. The closest I ever found was augeas, but that's for configuration files. I'm working on something, that is a open template format for binary file formats. It is usable today as a universal file extractor, with some bugs and limitations. Check it out at https://github.com/martinlindhe/feng https://github.com/martinlindhe/feng
erichdongubler 3y ago
It seems strange, because it's not reality! Forensic tools like FTK and Autopsy have had a plug-in framework for these forever, speaking as a former contributor to the former. There's also Kaitai Struct. I'm sure other communities have popped up that I haven't heard of, too. There's lots of interest in unifying forensic parsing under open work.
- marlin 3y ago
  I'm working on something, that is a open template format for binary file formats. It is usable today as a universal file extractor, with some bugs and limitations. Check it out at https://github.com/martinlindhe/feng https://github.com/martinlindhe/feng
BoppreH 3y ago
> It always seemed strange that there wasn’t any central database of binary parsers that everyone could contribute to. Fully agree on the need for such a database. The problem is that it's been tried, but lacked traction. The latest one I've seen is Kaitai format files[1], that can be used in visualizers or to auto-generate parsers. [1] https://formats.kaitai.io/ https://formats.kaitai.io/
- fullspectrumdev 3y ago
  Thanks for the reminder about Katai, I’ve been meaning to look at it specifically for something but forgot what it was called.
panzi 3y ago
That's also my experience. People releasing their tools on obscure forums, usually without source code and no version control. Almost enire communities that haven't heard of GitHub. Though usually the tools work in wine... but are next to useless. GUI only and very clumsy to use. So many clicks for each single file to extract, no batch operations, no command line interface. A big WTF. How can you do any modding with those tools?
- 59nadir 3y ago
  It's hard to not read this as elitist and entitled, to be honest. There are many people who know that GitHub exists but don't want to use it, and they also don't care for you having their source code. It's a choice they make. This entitlement is interesting because the solution obviously is to just make your own tools and not be a lazy library/tool hunter all the time.
  - panzi 3y ago
    If they do less work, i.e. not write a GUI, but just provide a command line tool you can suddenly automate things and it all gets much easier to use. Anyway, I then wrote my own tools and put them on GitHub. And documented the reverse engineered file formats. And then got pull requests from people that want to help! I was just voicing my bewilderment at this other culture. Don't understand why one would want to live like that.
amenghra 3y ago
I miss the ResEdit days of Macintosh days (i.e. MacOS 7-8-9). You could see/steal/modify/hack the visual assets of most binaries. You could remap keyboard shortcuts, modify menus, etc.
- mpweiher 3y ago
  Well, app wrappers make this even simpler: just poke inside with a file browser.
  - mh- 3y ago
    And them being signed (and those signatures being validated) makes it highly impractical.
    - mpweiher 3y ago
      What the technology giveth, the policy taketh away...
jchw 3y ago
Kaitai Struct is the better way to go for an ecosystem solution, but the tooling could certainly use improvements. In addition to 010 Editor, there's also KDE's Okteta. It does not have a lot of good OSDs and the OSD format/scripting for specifying formats is a little anemic (I'd like to help improve it if I can find time...) but it's very serviceable and a decent open source alternative to what 010 has. Shameless plug, I made a decent Windows EXE/PE OSD for Okteta. (It's even got a bit of support for NE16 executables, just for fun.) This entire genre of tools has been a long time point of interest for me. In addition to making a couple OSDs and contributing some tiny improvements to Kaitai, I also have my own binary schema library for Go, restruct, which, biased or not, remains my favorite way to poke at arbitrary formats, since it's really easy to sketch stuff out and read and write to files quickly. It's basically Go's encoding/binary but with struct tags for more advanced things.
- marlin 3y ago
  > This entire genre of tools has been a long time point of interest for me. Same for me! It turned into a long journey and I am working on a solution that I am very happy about. Somewhere mid-journey I learned about kaitai struct and lost a bit of steam seeing it was similar. But I think my offering is superior in a more simple template format with less programming required and a nice cli app. I am yet to announce it publicly, but i been meaning to for so long already. If you would like to check it out I would be happy! You can use it to map / view content from a format there is a template for. Alot of common formats is already included and you can extend it using your own templates. https://github.com/martinlindhe/feng https://github.com/martinlindhe/feng
  - joshspankit 3y ago
    As someone who had a great time with Kaitai, may I suggest that you write an interface so that fq can be used with any format that Kaitai understands (and any that people add in future)
- mindv0rtex 3y ago
  The functionality that I am personally interested in from a binary parsing framework like Kaitai is generating an encoder implementation in addition to a decoder one. In other words, given a description of a binary format, I would like to be able to construct an instance of a class whose memory layout matches the format. For instance, if the format has an int n, then an array `a` of size `n`, and then a double `d`, it would be awesome to be able to construct a corresponding object with fields `n`, `a` and `d` and when I change `n`, then the size of `a` changes accordingly. And then, if I pass a pointer to this object to the decoder, it would be able to parse it correctly, as if the memory representation of the object came from some external buffer.
wwader 3y ago
Hi, sorry for the delay, on vacation. There is no process really more then convincing me :) and i think i will accept any decoder that is for a format used in public, standardized or proprietary. I do want to add some kind of runtime format support and i'm working adding kaitai support but it's not ready yet, it's not an easy thing to do :) but i've made very good progress. ideally it will be something like: fq -d format.ksy <query> file Subscribe or keep an eye on this issue for updates https://github.com/wader/fq/issues/627 https://github.com/wader/fq/issues/627 And feel free to ask any questions!

andoma 3y ago

Being a long time personal friend with the author I can assure you the more obscure the better :-) His interest in esoteric things and solutions are "well documented" if you browse around his github repos. Some examples: https://github.com/wader/jqjq https://github.com/wader/jqjq https://github.com/wader/catgolf https://github.com/wader/catgolf

sillysaurusx 3y ago
Wonderful, thank you so much! Such a cool project.
Akronymus 3y ago
I am RE-ing some binary file formats for games. Should those be contributed as well?
wwader 3y ago
fun fact: fq is kind of the spiritual predecessor of https://github.com/wader/flac.tcl https://github.com/wader/flac.tcl, you can see traces of it in flac_frame.go in fq, was used to prototype some things :)

grumbel 3y ago

There is QuickBMS[1], which covers quite a few game related formats. [1] http://aluigi.altervista.org/quickbms.htm http://aluigi.altervista.org/quickbms.htm

realjhol 3y ago

Like wireshark directors but more general

nonethewiser 3y ago

> I was trying to rip some sounds from a wii game called Rhythm Heaven, and it’s ridiculous how primitive the tech is. This seems like a pretty niche need with only some hobbyists motivated enough to work on it. Is there a broader application than your use case? Otherwise I think thats why the existing software for this isnt great.

sillysaurusx 3y ago
Yeah, I didn’t mean to sound entitled. I only meant I was excited for projects like fq to shake things up. When I was writing a parser for the sound file I was thinking “hmm… this really feels like duplicate work.” On the other hand, it’s surprising to me that “grab sounds from a wii game” is so niche! My gut felt like it would be slightly more complicated than unzipping a tarball, but my gut didn’t expect it to be a programming challenge worthy of a small competition.
- foobarian 3y ago
  It’s all fun and games until Nintendo sends in a cease and desist notice
  - sillysaurusx 3y ago
    Seriously. Nintendo is an evil corporation. I was about to write “close to,” but they crossed the line when they sent someone to prison and garnished his wages for the rest of his life.
    - ozim 3y ago
      Not just someone but dude was not ripping or cracking stuff for fun. He made business out of it and what’s worse he added ransomware to scam his “customers”. POS deserved all of it if not more.
      - sillysaurusx 3y ago
        If he added ransomware, that’s a bit different. Making a business out of it isn’t that bad (think about it in terms of people going to prison for assault vs merely making some money), but the ransomware would be. Still, garnishing someone’s wages the rest of his life seems out of proportion. But I admit it’s harder to defend someone that made a livelihood out of holding peoples’ data hostage.

wongarsu 3y ago

There is the 010 Editor, at heart a cross-platform scriptable hex editor with a template language [1]. It has a central template repository [2] as well as templates around the internet (e.g. 3, 4). But it being a paid tool means there are fewer template contributions from 13 year olds, which if we are all honest make up the majority of unpaid open source contributions - they simply have more spare time. 1: https://www.sweetscape.com/010editor/ https://www.sweetscape.com/010editor/ 2: https://www.sweetscape.com/010editor/repository/templates/ https://www.sweetscape.com/010editor/repository/templates/ 3: https://github.com/tge-was-taken/010-Editor-Templates/tree/master/templates https://github.com/tge-was-taken/010-Editor-Templates/tree/m... 4: https://wiki.redmodding.org/cyberpunk-2077-modding/modding-know-how/modding-tools/010-editor https://wiki.redmodding.org/cyberpunk-2077-modding/modding-k...

overengineer 3y ago
What is wrong with contributions from 13 year olds?
- xp84 3y ago
  I don’t think it was implied that there’s anything bad, I read it as the opposite.
- oxygen_crisis 3y ago
  There aren't enough of them, because the tool costs money, and 13-year-olds can't spend money on software as readily as adults.
nness 3y ago
010editor's templating language looks very interesting (and something I had been trying to accomplish with my own mixed-bag of tools when reverse-engineering). I suppose as a hobbyist, the price is a hard one to get over...
GuB-42 3y ago
Do 13 year olds care about licenses and pay for the software they use? I certainly didn't when I was 13. And 010 editor is about as well protected as WinRAR. I have a hard time imagining a 13 year old using that particular tool buying it. In fact there is a good chance it is used by the crackers themselves.

BeefWellington 3y ago

Binwalk is my go-to for that kind of thing usually.

fullspectrumdev 3y ago
Try unblob sometime, it’s a more modern, maintained alternative (not a fork). A company called OneKey that do some firmware security stuff maintain it, and generally it’s pretty good.

kazinator 3y ago

> It always seemed strange that there wasn’t any central database of binary parsers that everyone could contribute to. file utility with /etc/magic database

loeg 3y ago
Yeah, that was my thought as well, although I don't know that file/magic get into much structure beyond identifying the file format.

stelonix 3y ago

I used to be from the romhacking community back in the 2000s and due to usage of Windows, open source/foss wasn't even known to most people. The culture of Windows programmers is way more focused on freeware/binaries. Still waiting to this day for FuSoYa to release the source code of Lunar Magic. About a central database of binary parsers, I've been wanting this for ages too. The closest I ever found was augeas, but that's for configuration files.

xp84 3y ago
Yeah that’s a mysterious one. Such an incredible achievement, and it enabled so, so much creativity, and free (as in beer) to all as far as I know. I hope FuSoYa does open source it someday.
marlin 3y ago
> About a central database of binary parsers, I've been wanting this for ages too. The closest I ever found was augeas, but that's for configuration files. I'm working on something, that is a open template format for binary file formats. It is usable today as a universal file extractor, with some bugs and limitations. Check it out at https://github.com/martinlindhe/feng https://github.com/martinlindhe/feng

erichdongubler 3y ago

It seems strange, because it's not reality! Forensic tools like FTK and Autopsy have had a plug-in framework for these forever, speaking as a former contributor to the former. There's also Kaitai Struct. I'm sure other communities have popped up that I haven't heard of, too. There's lots of interest in unifying forensic parsing under open work.

marlin 3y ago
I'm working on something, that is a open template format for binary file formats. It is usable today as a universal file extractor, with some bugs and limitations. Check it out at https://github.com/martinlindhe/feng https://github.com/martinlindhe/feng

BoppreH 3y ago

> It always seemed strange that there wasn’t any central database of binary parsers that everyone could contribute to. Fully agree on the need for such a database. The problem is that it's been tried, but lacked traction. The latest one I've seen is Kaitai format files[1], that can be used in visualizers or to auto-generate parsers. [1] https://formats.kaitai.io/ https://formats.kaitai.io/

fullspectrumdev 3y ago
Thanks for the reminder about Katai, I’ve been meaning to look at it specifically for something but forgot what it was called.

panzi 3y ago

That's also my experience. People releasing their tools on obscure forums, usually without source code and no version control. Almost enire communities that haven't heard of GitHub. Though usually the tools work in wine... but are next to useless. GUI only and very clumsy to use. So many clicks for each single file to extract, no batch operations, no command line interface. A big WTF. How can you do any modding with those tools?

59nadir 3y ago
It's hard to not read this as elitist and entitled, to be honest. There are many people who know that GitHub exists but don't want to use it, and they also don't care for you having their source code. It's a choice they make. This entitlement is interesting because the solution obviously is to just make your own tools and not be a lazy library/tool hunter all the time.
- panzi 3y ago
  If they do less work, i.e. not write a GUI, but just provide a command line tool you can suddenly automate things and it all gets much easier to use. Anyway, I then wrote my own tools and put them on GitHub. And documented the reverse engineered file formats. And then got pull requests from people that want to help! I was just voicing my bewilderment at this other culture. Don't understand why one would want to live like that.

amenghra 3y ago

I miss the ResEdit days of Macintosh days (i.e. MacOS 7-8-9). You could see/steal/modify/hack the visual assets of most binaries. You could remap keyboard shortcuts, modify menus, etc.

mpweiher 3y ago
Well, app wrappers make this even simpler: just poke inside with a file browser.
- mh- 3y ago
  And them being signed (and those signatures being validated) makes it highly impractical.
  - mpweiher 3y ago
    What the technology giveth, the policy taketh away...

jchw 3y ago

Kaitai Struct is the better way to go for an ecosystem solution, but the tooling could certainly use improvements. In addition to 010 Editor, there's also KDE's Okteta. It does not have a lot of good OSDs and the OSD format/scripting for specifying formats is a little anemic (I'd like to help improve it if I can find time...) but it's very serviceable and a decent open source alternative to what 010 has. Shameless plug, I made a decent Windows EXE/PE OSD for Okteta. (It's even got a bit of support for NE16 executables, just for fun.) This entire genre of tools has been a long time point of interest for me. In addition to making a couple OSDs and contributing some tiny improvements to Kaitai, I also have my own binary schema library for Go, restruct, which, biased or not, remains my favorite way to poke at arbitrary formats, since it's really easy to sketch stuff out and read and write to files quickly. It's basically Go's encoding/binary but with struct tags for more advanced things.

marlin 3y ago
> This entire genre of tools has been a long time point of interest for me. Same for me! It turned into a long journey and I am working on a solution that I am very happy about. Somewhere mid-journey I learned about kaitai struct and lost a bit of steam seeing it was similar. But I think my offering is superior in a more simple template format with less programming required and a nice cli app. I am yet to announce it publicly, but i been meaning to for so long already. If you would like to check it out I would be happy! You can use it to map / view content from a format there is a template for. Alot of common formats is already included and you can extend it using your own templates. https://github.com/martinlindhe/feng https://github.com/martinlindhe/feng
- joshspankit 3y ago
  As someone who had a great time with Kaitai, may I suggest that you write an interface so that fq can be used with any format that Kaitai understands (and any that people add in future)
mindv0rtex 3y ago
The functionality that I am personally interested in from a binary parsing framework like Kaitai is generating an encoder implementation in addition to a decoder one. In other words, given a description of a binary format, I would like to be able to construct an instance of a class whose memory layout matches the format. For instance, if the format has an int n, then an array `a` of size `n`, and then a double `d`, it would be awesome to be able to construct a corresponding object with fields `n`, `a` and `d` and when I change `n`, then the size of `a` changes accordingly. And then, if I pass a pointer to this object to the decoder, it would be able to parse it correctly, as if the memory representation of the object came from some external buffer.

wwader 3y ago

Hi, sorry for the delay, on vacation. There is no process really more then convincing me :) and i think i will accept any decoder that is for a format used in public, standardized or proprietary. I do want to add some kind of runtime format support and i'm working adding kaitai support but it's not ready yet, it's not an easy thing to do :) but i've made very good progress. ideally it will be something like: fq -d format.ksy <query> file Subscribe or keep an eye on this issue for updates https://github.com/wader/fq/issues/627 https://github.com/wader/fq/issues/627 And feel free to ask any questions!