6 ms·

GNU poke: The extensible editor for structured binary data

282 points by marcodiego 4y ago

guidoism 4y ago
What a great name!
babas 4y ago
It may not be ideal to praise commercial software in a GNU software thread, but I really enjoy using 010 Editor. It's one of the few programs I pay for. Its C-like template syntax (looks similar to GNU Poke syntax) makes reverse engineering binary formats easy and fun!
- nequo 4y ago
  What are the pros and cons of 010 Editor compared with GNU poke?
- colejohnson66 4y ago
  Check out ImHex! It's a FOSS alternative to 010 Editor.
elevation 4y ago
I always liked Winhex for this type of thing. It could dissect a structure with a c-like syntax. I found it as shareware as a kid and have never used a hex editor I liked more. https://www.x-ways.net/winhex/ https://www.x-ways.net/winhex/
- sparkie 4y ago
  wxHexEditor has a similar look and feel to WinHex, and similarly supports editing of extremely large files or raw devices. It's nowhere near as featureful but still very useful, and doesn't have the proprietary license. https://github.com/EUA/wxHexEditor https://github.com/EUA/wxHexEditor / http://www.wxhexeditor.org/ http://www.wxhexeditor.org/
- Andoryuuta 4y ago
  Interesting! I've heard of winhex before, but I never knew it had a templating language. I've been using 010 Editor, which similarly lets you define templates with c-like syntax, run it against a file, and edit the fields from there. (Cross platform too, which is nice. It is a paid product though) https://www.sweetscape.com/010editor/ https://www.sweetscape.com/010editor/
  - themerone 4y ago
    It's nice to see it's cross platform now. It was windows only back when I needed it.
- xvilka 4y ago
  C-like syntax is a poor choice for data description. It's not flexible enough. This is why many newer DDLs were created.
klyrs 4y ago
I love/hate that "pickle" is defined through usage. Curious to know if that name has any meaning (e.g. python pickles are for long-term storage of ephemeral data), but not curious enough to watch a video just in case that's covered.
- upwardbound 4y ago
  A pickle is a cucumber preserved in preservatives for long term storage, hence that name I think (and it start with P like python).
  - Timon3 4y ago
    Just to expand on this: > Pickling is the process of preserving or extending the shelf life of food by either anaerobic fermentation in brine or immersion in vinegar. https://en.wikipedia.org/wiki/Pickling https://en.wikipedia.org/wiki/Pickling
- hummus_bae 4y ago
  [dead]
bogwog 4y ago
Haven't watched the full video, but this looks similar ImHex (https://imhex.werwolv.net/ https://imhex.werwolv.net/), which also includes a pattern language thing to describe the structure of data. I used it once for a project, and it was useful when it worked, although I ran into some limitations when trying to model container formats. Maybe it could do that and I just couldn't figure it out at the time, but if you have say a zip file with different file formats, you couldn't tell the language to switch between different structures based on like an index or a header that tells you the format of a subsection. It was a limitation of the pattern language. I wonder if GNU poke is more advanced in that regard? A tool like this would be super useful for debugging custom binary formats, but some formats can get pretty complex.
AceJohnny2 4y ago
Can anyone comment on how this compares to Hachoire? https://hachoir.readthedocs.io/en/latest/ https://hachoir.readthedocs.io/en/latest/
still_grokking 4y ago
In my opinion the core on which poke was made is actually much more interesting than the tool as such. Poke is actually not only an editor, it's a domain specific language. This language is built on top of something called GNU Jitter. https://www.gnu.org/software/jitter/ https://www.gnu.org/software/jitter/ Jitter is a very interesting project! It's a kind of generic interpreter that is—as the name suggests—almost a JIT compiler. It's something more or less "in between". Technically it's more an interpreter + VM. But it uses all kinds of (I think novel) tricks to remove the overhead of interpretation, which makes the code at runtime almost look like what a JIT compiler would have produced. Some more internal info than on the GNU page on how this works can be found in the following slides: http://ageinghacker.net/talks/jitter-slides--saiu--ghm2017--2017-08-25.pdf http://ageinghacker.net/talks/jitter-slides--saiu--ghm2017--...
- sparkie 4y ago
  I'm curious as to whether the techniques used here are safe against Spectre-like attacks. There appears to be a heavy reliance on indirect branching. The slides here are dated just before Spectre was published, so it would not have been a concern at the time.
  - coldtea 4y ago
    >I'm curious as to whether the techniques used here are safe against Spectre-like attacks Because people will be compromising your GNU poke or Jitter code?
    - sparkie 4y ago
      Not poke per se, but Jitter is intended to be a retargetable interpreter for other programming languages, so maybe.
tdiff 4y ago
Can it be taught to read serialized protobufs?
- hgsgm 4y ago
  Yes. Go for it.
marssaxman 4y ago
How fun! Reminds me of ResEdit, from classic Mac OS, and its template resources.
DannyB2 4y ago
Not the same thing, but I am reminded of Rez, Derez and the Resource Editor of ancient Mac daze.
- salgernon 4y ago
  Rez and DeRez still ship with xcode (or did the last time I checked)
VLM 4y ago
That would be a fun CLI for retro/tiny computers. Imagine something like that for a Z80 or 68hc11 vs the typical boot rom.
mdp2021 4y ago
It was about time! The capabilities of hex editors on Linux is strongly underwhelming when compared to, say, WinHex almost 30 years ago!
DvdGiessen 4y ago
See also the recent talk the author gave at FOSDEM last month[1] for an introduction to the tool and its use cases. [1]: https://fosdem.org/2023/schedule/event/bintools_poke/ https://fosdem.org/2023/schedule/event/bintools_poke/
- kreetx 4y ago
  IMO one of the most humorous talks at the event! :)
turtledragonfly 4y ago
Oh, this is one I hadn't heard of. There are several different takes on this idea, and I've had the notion to make my own for some years, too (not enough free time...) I feel like a good "how powerful is this tool, really?" test would be to model the Adobe PSD format with it. That format is bananas, and is what got me into this topic in the first place. But for those interested, here are some other tools in this vein: * fq - like jq for binary data: https://github.com/wader/fq https://github.com/wader/fq * Kaitai Struct - https://kaitai.io/ https://kaitai.io/ ** visualizer, for the above: https://github.com/kaitai-io/kaitai_struct_visualizer/ https://github.com/kaitai-io/kaitai_struct_visualizer/ * HexFiend - a hex editor, but with "binary templates" feature : https://github.com/HexFiend/HexFiend https://github.com/HexFiend/HexFiend ** binary templates, for the above: https://github.com/HexFiend/HexFiend/blob/master/templates/Tutorial.md https://github.com/HexFiend/HexFiend/blob/master/templates/T... * binspector - https://github.com/binspector/binspector https://github.com/binspector/binspector * binary-parsing - a collection of links to similar such tools : https://github.com/dloss/binary-parsing https://github.com/dloss/binary-parsing * unblob - https://github.com/onekey-sec/unblob https://github.com/onekey-sec/unblob ** previous HN thread, for the above: https://news.ycombinator.com/item?id=34434249 https://news.ycombinator.com/item?id=34434249
- SloopJon 4y ago
  This is why I keep reading these threads, looking for something I can wrap my head around. I started looking at Kaitai Struct not too long ago. I purposely started with a pretty simple format, but I think I'm not smart enough to figure out the input format. Also, I'm pretty sure I hate YAML. I tried building Poke just last week, having resolved to try it since 1.0, but hit problems with the Boehm GC on my old Mac, and then what looks like Readline problems on my new Mac. I haven't given up, but I'm off to a slow start. Hachoir, mentioned in another comment, looks promising, although the input looks suspiciously similar to a program I'd write myself.
  - fcambus 4y ago
    It's a bit overkill for only building Poke, but you can use Pkgsrc [1] to do so on macOS: Bootstrap Pkgsrc: git clone https://github.com/NetBSD/pkgsrc.git https://github.com/NetBSD/pkgsrc.git cd pkgsrc/bootstrap ./bootstrap --unprivileged And then build and install Poke: cd ~/pkgsrc/editors/poke bmake install package clean clean-depends [1] https://pkgsrc.org https://pkgsrc.org
armitron 4y ago
Yet another half-assed custom language that nobody wants to learn? What happened to Guile being THE GNU extension language?
wiseowise 4y ago
Love the website design!

282 points by marcodiego 4y ago

What a great name!

It may not be ideal to praise commercial software in a GNU software thread, but I really enjoy using 010 Editor. It's one of the few programs I pay for. Its C-like template syntax (looks similar to GNU Poke syntax) makes reverse engineering binary formats easy and fun!

nequo 4y ago
What are the pros and cons of 010 Editor compared with GNU poke?
colejohnson66 4y ago
Check out ImHex! It's a FOSS alternative to 010 Editor.

elevation 4y ago

I always liked Winhex for this type of thing. It could dissect a structure with a c-like syntax. I found it as shareware as a kid and have never used a hex editor I liked more. https://www.x-ways.net/winhex/ https://www.x-ways.net/winhex/

sparkie 4y ago
wxHexEditor has a similar look and feel to WinHex, and similarly supports editing of extremely large files or raw devices. It's nowhere near as featureful but still very useful, and doesn't have the proprietary license. https://github.com/EUA/wxHexEditor https://github.com/EUA/wxHexEditor / http://www.wxhexeditor.org/ http://www.wxhexeditor.org/
Andoryuuta 4y ago
Interesting! I've heard of winhex before, but I never knew it had a templating language. I've been using 010 Editor, which similarly lets you define templates with c-like syntax, run it against a file, and edit the fields from there. (Cross platform too, which is nice. It is a paid product though) https://www.sweetscape.com/010editor/ https://www.sweetscape.com/010editor/
- themerone 4y ago
  It's nice to see it's cross platform now. It was windows only back when I needed it.
xvilka 4y ago
C-like syntax is a poor choice for data description. It's not flexible enough. This is why many newer DDLs were created.

klyrs 4y ago

I love/hate that "pickle" is defined through usage. Curious to know if that name has any meaning (e.g. python pickles are for long-term storage of ephemeral data), but not curious enough to watch a video just in case that's covered.

upwardbound 4y ago
A pickle is a cucumber preserved in preservatives for long term storage, hence that name I think (and it start with P like python).
- Timon3 4y ago
  Just to expand on this: > Pickling is the process of preserving or extending the shelf life of food by either anaerobic fermentation in brine or immersion in vinegar. https://en.wikipedia.org/wiki/Pickling https://en.wikipedia.org/wiki/Pickling
hummus_bae 4y ago
[dead]

bogwog 4y ago

Haven't watched the full video, but this looks similar ImHex (https://imhex.werwolv.net/ https://imhex.werwolv.net/), which also includes a pattern language thing to describe the structure of data. I used it once for a project, and it was useful when it worked, although I ran into some limitations when trying to model container formats. Maybe it could do that and I just couldn't figure it out at the time, but if you have say a zip file with different file formats, you couldn't tell the language to switch between different structures based on like an index or a header that tells you the format of a subsection. It was a limitation of the pattern language. I wonder if GNU poke is more advanced in that regard? A tool like this would be super useful for debugging custom binary formats, but some formats can get pretty complex.

AceJohnny2 4y ago

Can anyone comment on how this compares to Hachoire? https://hachoir.readthedocs.io/en/latest/ https://hachoir.readthedocs.io/en/latest/

still_grokking 4y ago

In my opinion the core on which poke was made is actually much more interesting than the tool as such. Poke is actually not only an editor, it's a domain specific language. This language is built on top of something called GNU Jitter. https://www.gnu.org/software/jitter/ https://www.gnu.org/software/jitter/ Jitter is a very interesting project! It's a kind of generic interpreter that is—as the name suggests—almost a JIT compiler. It's something more or less "in between". Technically it's more an interpreter + VM. But it uses all kinds of (I think novel) tricks to remove the overhead of interpretation, which makes the code at runtime almost look like what a JIT compiler would have produced. Some more internal info than on the GNU page on how this works can be found in the following slides: http://ageinghacker.net/talks/jitter-slides--saiu--ghm2017--2017-08-25.pdf http://ageinghacker.net/talks/jitter-slides--saiu--ghm2017--...

sparkie 4y ago
I'm curious as to whether the techniques used here are safe against Spectre-like attacks. There appears to be a heavy reliance on indirect branching. The slides here are dated just before Spectre was published, so it would not have been a concern at the time.
- coldtea 4y ago
  >I'm curious as to whether the techniques used here are safe against Spectre-like attacks Because people will be compromising your GNU poke or Jitter code?
  - sparkie 4y ago
    Not poke per se, but Jitter is intended to be a retargetable interpreter for other programming languages, so maybe.

tdiff 4y ago

Can it be taught to read serialized protobufs?

hgsgm 4y ago
Yes. Go for it.

marssaxman 4y ago

How fun! Reminds me of ResEdit, from classic Mac OS, and its template resources.

DannyB2 4y ago

Not the same thing, but I am reminded of Rez, Derez and the Resource Editor of ancient Mac daze.

salgernon 4y ago
Rez and DeRez still ship with xcode (or did the last time I checked)

VLM 4y ago

That would be a fun CLI for retro/tiny computers. Imagine something like that for a Z80 or 68hc11 vs the typical boot rom.

mdp2021 4y ago

It was about time! The capabilities of hex editors on Linux is strongly underwhelming when compared to, say, WinHex almost 30 years ago!

DvdGiessen 4y ago

See also the recent talk the author gave at FOSDEM last month[1] for an introduction to the tool and its use cases. [1]: https://fosdem.org/2023/schedule/event/bintools_poke/ https://fosdem.org/2023/schedule/event/bintools_poke/

kreetx 4y ago
IMO one of the most humorous talks at the event! :)

turtledragonfly 4y ago

Oh, this is one I hadn't heard of. There are several different takes on this idea, and I've had the notion to make my own for some years, too (not enough free time...) I feel like a good "how powerful is this tool, really?" test would be to model the Adobe PSD format with it. That format is bananas, and is what got me into this topic in the first place. But for those interested, here are some other tools in this vein: * fq - like jq for binary data: https://github.com/wader/fq https://github.com/wader/fq * Kaitai Struct - https://kaitai.io/ https://kaitai.io/ ** visualizer, for the above: https://github.com/kaitai-io/kaitai_struct_visualizer/ https://github.com/kaitai-io/kaitai_struct_visualizer/ * HexFiend - a hex editor, but with "binary templates" feature : https://github.com/HexFiend/HexFiend https://github.com/HexFiend/HexFiend ** binary templates, for the above: https://github.com/HexFiend/HexFiend/blob/master/templates/Tutorial.md https://github.com/HexFiend/HexFiend/blob/master/templates/T... * binspector - https://github.com/binspector/binspector https://github.com/binspector/binspector * binary-parsing - a collection of links to similar such tools : https://github.com/dloss/binary-parsing https://github.com/dloss/binary-parsing * unblob - https://github.com/onekey-sec/unblob https://github.com/onekey-sec/unblob ** previous HN thread, for the above: https://news.ycombinator.com/item?id=34434249 https://news.ycombinator.com/item?id=34434249

SloopJon 4y ago
This is why I keep reading these threads, looking for something I can wrap my head around. I started looking at Kaitai Struct not too long ago. I purposely started with a pretty simple format, but I think I'm not smart enough to figure out the input format. Also, I'm pretty sure I hate YAML. I tried building Poke just last week, having resolved to try it since 1.0, but hit problems with the Boehm GC on my old Mac, and then what looks like Readline problems on my new Mac. I haven't given up, but I'm off to a slow start. Hachoir, mentioned in another comment, looks promising, although the input looks suspiciously similar to a program I'd write myself.
- fcambus 4y ago
  It's a bit overkill for only building Poke, but you can use Pkgsrc [1] to do so on macOS: Bootstrap Pkgsrc: git clone https://github.com/NetBSD/pkgsrc.git https://github.com/NetBSD/pkgsrc.git cd pkgsrc/bootstrap ./bootstrap --unprivileged And then build and install Poke: cd ~/pkgsrc/editors/poke bmake install package clean clean-depends [1] https://pkgsrc.org https://pkgsrc.org

armitron 4y ago

Yet another half-assed custom language that nobody wants to learn? What happened to Guile being THE GNU extension language?

wiseowise 4y ago

Love the website design!