4 ms·

Visualizing Entropy in Binary Files

47 points by jgrodziski 12y ago

fenollp 12y ago
Have a look at Cantor Dust [1], a visual reverse-engineering program that is promising. The author says in a presentation [2] that CD is based on Cortesi's work. [1] https://sites.google.com/site/xxcantorxdustxx/visual-re https://sites.google.com/site/xxcantorxdustxx/visual-re [2] https://www.youtube.com/watch?v=4bM3Gut1hIk https://www.youtube.com/watch?v=4bM3Gut1hIk Here is a demo: https://media.blackhat.com/bh-us-12/Arsenal/Domas/_cantor.dust_.7z.zip https://media.blackhat.com/bh-us-12/Arsenal/Domas/_cantor.du...
cortesi 12y ago
Author of the post here. I'm working on a browser-based interactive analysis tool based on these visualizations. It's nowhere near ready for release, but those who are interested can find a demo here: http://binvis.io http://binvis.io This is not even an alpha yet - I plan to announce the initial release in the next month or so. Eventually, I would like to add the ability to upload, share and annotate files for analysis. Comments and criticisms welcome!
- 13 12y ago
  The level of detail and structure you can pick out with this tool is amazing. https://i.imgur.com/lsRA1RN.png https://i.imgur.com/lsRA1RN.png
  - muyuu 12y ago
    Removed?
- scentoni 12y ago
  That's excellent! Would be very cool to also have frequency digraphs for the current segment.
  - cortesi 12y ago
    Frequency analysis and byte dot plots are definitely on the way. Do you have a concrete example of the digraph frequency feature you can link me to? Any other feature suggestions are welcome too.
    - scentoni 12y ago
      The link below to the Cantor Dust presentation shows 2d and 3d examples. I wonder if there are advantages of other space-filling curves, like http://en.wikipedia.org/wiki/Sierpi%C5%84ski_curve http://en.wikipedia.org/wiki/Sierpi%C5%84ski_curve and http://en.wikipedia.org/wiki/Peano_curve http://en.wikipedia.org/wiki/Peano_curve
      - cortesi 12y ago
        Right - these are the frequency dot plots I first saw in Greg Conti's work (some examples here: http://sebug.net/paper/Meeting-Documents/BlackHat-USA2008/BH_US_08_Conti_Dean_Visual_Forensic_Analysis.pdf http://sebug.net/paper/Meeting-Documents/BlackHat-USA2008/BH...). They're definitely on the way. Regarding other space-filling curves - I'm in two minds about this. I've implemented z-order and the H-curve for binvis, but for different reasons neither of these are as useful as the Hilbert curve, so they're not exposed in the interface. My feeling now is that I should pick one locality preserving layout and one intuitive layout to keep things simple and usable. I want binvis to be a workaday tool, so I'm keeping my space-filling curve geekery to another project (https://github.com/cortesi/scurve https://github.com/cortesi/scurve).
wxs 12y ago
These images remind me of Piet[1], an esoteric programming language where the programs are images. There you see the structure of the algorithm, rather than the binary, visually presented. [1] http://www.dangermouse.net/esoteric/piet/samples.html http://www.dangermouse.net/esoteric/piet/samples.html
seivadmas 12y ago
Perhaps a naive question but... what is this useful for?
- cyphunk 12y ago
  reverse engineering binaries. as patterns are visualised you can also skip reverse engineering for some goals. for example general entropy levels of bytes (in relation to their siblings) may allow one to quickly pull out encryption keys from binaries or memory images.
pmoriarty 12y ago
This reminds me of a tool that came out 10 or 15 years ago, which allowed you to feed it arbitrary binary data that it could display in 3D. I remember reading a Slashdot article on it in which one of the examples was a 3D, freely-rotating view of the Linux kernel binary. Does anyone know what I'm talking about? I can't seem to recall the name of the project or find it. Also related: http://vimeo.com/110257380 http://vimeo.com/110257380 http://gynvael.coldwind.pl/?id=199 http://gynvael.coldwind.pl/?id=199 https://www.youtube.com/watch?v=5f7hZBoEwV0 https://www.youtube.com/watch?v=5f7hZBoEwV0 https://github.com/pwaller/binview https://github.com/pwaller/binview

47 points by jgrodziski 12y ago

Have a look at Cantor Dust [1], a visual reverse-engineering program that is promising. The author says in a presentation [2] that CD is based on Cortesi's work. [1] https://sites.google.com/site/xxcantorxdustxx/visual-re https://sites.google.com/site/xxcantorxdustxx/visual-re [2] https://www.youtube.com/watch?v=4bM3Gut1hIk https://www.youtube.com/watch?v=4bM3Gut1hIk Here is a demo: https://media.blackhat.com/bh-us-12/Arsenal/Domas/_cantor.dust_.7z.zip https://media.blackhat.com/bh-us-12/Arsenal/Domas/_cantor.du...

cortesi 12y ago

Author of the post here. I'm working on a browser-based interactive analysis tool based on these visualizations. It's nowhere near ready for release, but those who are interested can find a demo here: http://binvis.io http://binvis.io This is not even an alpha yet - I plan to announce the initial release in the next month or so. Eventually, I would like to add the ability to upload, share and annotate files for analysis. Comments and criticisms welcome!

13 12y ago
The level of detail and structure you can pick out with this tool is amazing. https://i.imgur.com/lsRA1RN.png https://i.imgur.com/lsRA1RN.png
- muyuu 12y ago
  Removed?
scentoni 12y ago
That's excellent! Would be very cool to also have frequency digraphs for the current segment.
- cortesi 12y ago
  Frequency analysis and byte dot plots are definitely on the way. Do you have a concrete example of the digraph frequency feature you can link me to? Any other feature suggestions are welcome too.
  - scentoni 12y ago
    The link below to the Cantor Dust presentation shows 2d and 3d examples. I wonder if there are advantages of other space-filling curves, like http://en.wikipedia.org/wiki/Sierpi%C5%84ski_curve http://en.wikipedia.org/wiki/Sierpi%C5%84ski_curve and http://en.wikipedia.org/wiki/Peano_curve http://en.wikipedia.org/wiki/Peano_curve
    - cortesi 12y ago
      Right - these are the frequency dot plots I first saw in Greg Conti's work (some examples here: http://sebug.net/paper/Meeting-Documents/BlackHat-USA2008/BH_US_08_Conti_Dean_Visual_Forensic_Analysis.pdf http://sebug.net/paper/Meeting-Documents/BlackHat-USA2008/BH...). They're definitely on the way. Regarding other space-filling curves - I'm in two minds about this. I've implemented z-order and the H-curve for binvis, but for different reasons neither of these are as useful as the Hilbert curve, so they're not exposed in the interface. My feeling now is that I should pick one locality preserving layout and one intuitive layout to keep things simple and usable. I want binvis to be a workaday tool, so I'm keeping my space-filling curve geekery to another project (https://github.com/cortesi/scurve https://github.com/cortesi/scurve).

wxs 12y ago

These images remind me of Piet[1], an esoteric programming language where the programs are images. There you see the structure of the algorithm, rather than the binary, visually presented. [1] http://www.dangermouse.net/esoteric/piet/samples.html http://www.dangermouse.net/esoteric/piet/samples.html

seivadmas 12y ago

Perhaps a naive question but... what is this useful for?

cyphunk 12y ago
reverse engineering binaries. as patterns are visualised you can also skip reverse engineering for some goals. for example general entropy levels of bytes (in relation to their siblings) may allow one to quickly pull out encryption keys from binaries or memory images.

pmoriarty 12y ago

This reminds me of a tool that came out 10 or 15 years ago, which allowed you to feed it arbitrary binary data that it could display in 3D. I remember reading a Slashdot article on it in which one of the examples was a 3D, freely-rotating view of the Linux kernel binary. Does anyone know what I'm talking about? I can't seem to recall the name of the project or find it. Also related: http://vimeo.com/110257380 http://vimeo.com/110257380 http://gynvael.coldwind.pl/?id=199 http://gynvael.coldwind.pl/?id=199 https://www.youtube.com/watch?v=5f7hZBoEwV0 https://www.youtube.com/watch?v=5f7hZBoEwV0 https://github.com/pwaller/binview https://github.com/pwaller/binview