6 ms·
Visual Analysis of Binary Files
- deleted 3y ago[deleted]
- yhvh 3y agoCantor dust https://youtu.be/C8--cXwuuFQ?si=xamOxGk2hpD5Mlmq https://youtu.be/C8--cXwuuFQ?si=xamOxGk2hpD5Mlmq https://inside.battelle.org/blog-details/battelle-publishes-open-source-binary-visualization-tool https://inside.battelle.org/blog-details/battelle-publishes-...
- torarnv 3y agoI remember seeing a similar layout used for an article or blog post about reverse engineering a binary format. I’ve tried finding it later, but couldn’t. Anyone know of such a tool that lets you describe the various headers, fields, etc of a binary format with a similar visualization?
- cortesi 3y agoAh, this is an old side-project of mine. Something I should probably make clearer is that files are not uploaded anywhere - the app is completely local, and all analysis is done in the browser. This version is written in React but when time permits I plan to release an updated version written in Rust, along with a library of fast implementations of space-filling curves and related utilities.
- dahart 3y agoHey what’s the clustering algorithm for the “cluster” vis mode?
- ericpauley 3y agoLooks like a Hilbert Curve: https://en.wikipedia.org/wiki/Hilbert_curve https://en.wikipedia.org/wiki/Hilbert_curve
- Waterluvian 3y agoI’m fascinated by this but on mobile at least I can’t quite figure out what the colours mean. Is there a legend I can peek at?
- cortesi 3y agoThere is, but I see it's not visible on smaller resolutions - I should fix that. The default color scheme just classifies bytes into black (0x00), white (0xff), blue (ascii), and low (green) and high (red).
- deleted 3y ago[deleted]
- iamcreasy 3y agoLook cool. What can you get out of this kind of visual analysis?
- Solvency 3y agoIt'd be cooler if these were interpreted as sound waves. I'd love to goto sleep listening to notepad.exe.
- jdironman 3y agoDid you happen to catch the post regarding ROM dumping the Nintendo Gameboy (DS?) by allowing the crashed game play through eventually playing ROM contents? just wondering if that's what gave you the idea here. But I agree, ASMr would be interesting!
- Cthulhu_ 3y agoI dunno, depends on whether it's raw sound or something gentle. Imagine listening to the modem connecting sound.
- TillE 3y agoGetting a really quick, coarse view of what a file format looks like, easily picking out different types of data. Try uploading a SNES ROM or similar and you'll probably see a lot of distinct squares because of how the ROM banks work. Super Metroid has a surprising amount of filler scattered all over.
- NobodyNada 3y ago> Super Metroid has a surprising amount of filler scattered all over. It only really looks scattered because the visualizer uses a space-filling curve instead of laying out the bytes of the file linearly. If you change the curve to "scan" instead of "cluster", it becomes a little easier to see what the filler is: it's the unused space at the end of each ROM bank. (As someone who's spent a lot of time in that particular binary, I think the use of the space-filling curve obscures a lot of details that would be a lot easier to spot otherwise -- such as the structure of the ROM banks, the location and layout of various pieces of compressed and uncompressed data, and repeated structures like a couple KB of common routines that are duplicated at the beginning of every bank containing enemy code.)
- Retr0id 3y agoI just wanted to say thanks, this is something I use regularly when looking at unknown firmware blobs or file formats, to look for compressed/encrypted data or other structures
- barlog 3y agoLooks fantastic cortesi! and binvis is the coolest name!!
- TheCoreh 3y agoThis tool has been incredibly useful to me for getting a high level overview of compiled binaries and JS bundles: you can typically tell if something fishy is getting included by the unexpected changes in entropy/categories. Thanks!
- fortran77 3y agoI've been trying to figure out the file format used for "POV" fan displays I've bought from AliExpress. This visualizer was a big help....
- eps 3y agoDoes it render data line by line or with something like Hilbert's curve to preserve locality of distinct features?
- kencausey 3y agoPreviously https://news.ycombinator.com/item?id=11077222 https://news.ycombinator.com/item?id=11077222 https://news.ycombinator.com/item?id=9140249 https://news.ycombinator.com/item?id=9140249 and other instances without comments.
- dang 3y agoThanks! Macroexpanded: Visualizing binaries with space-filling curves (2011) - https://news.ycombinator.com/item?id=24813704 https://news.ycombinator.com/item?id=24813704 - Oct 2020 (15 comments) Visualizing binaries with space-filling curves (2011) - https://news.ycombinator.com/item?id=14544191 https://news.ycombinator.com/item?id=14544191 - June 2017 (8 comments) Binvis.io – Visual Analysis of Binary Files - https://news.ycombinator.com/item?id=11077222 https://news.ycombinator.com/item?id=11077222 - Feb 2016 (9 comments) Binvis.io: visual analysis of binary files - https://news.ycombinator.com/item?id=9140249 https://news.ycombinator.com/item?id=9140249 - March 2015 (1 comment) Visualizing binaries with space-filling curves - https://news.ycombinator.com/item?id=3449743 https://news.ycombinator.com/item?id=3449743 - Jan 2012 (20 comments) Others?
- defrost 3y agoThis is where I miss | still pull out the original ERMapper geospatial viewer with the .ERS ASCII file "header" for viewing. https://gdal.org/drivers/raster/ers.html https://gdal.org/drivers/raster/ers.html does not in any way do it justice. What it allows you to do is create a text header description for a binary file .. pretty much any binary image (and not intended to be image) "raw" (ish) format that existed then and now. Band interleaved, band seperated, row interleaved, column orientated, etc. 8 bit, 12, 24, 32 bit, big or little endian, with or without header "junk" to skip at start of file and|or at the start of each "virtual row" The handy hack was the ability to view as image and file that had time series aquisition by sketching out how to block it, how to understand the binary, insert optional colour maps .. and then being able to rapidly visually scan 2GB or 16GB of binary blob with real time fly about zoom in zoom out capability. Intended for multichannel satellite | aircraft | other instrument data with a variety of metadata (timing, position, instrument orientation, etc) - useful far beyond its core target. Addendum: for the curious, somewhat better description of the .ERS layout from PDF page 33 onwards of https://www.aseg.org.au/sites/default/files/ER%20Mapper%20Customisation.pdf https://www.aseg.org.au/sites/default/files/ER%20Mapper%20Cu...
- neandrake 3y agoA video of someone implementing a project of a related topic, for identifying binary patterns based on visualization. Interesting stuff. https://www.youtube.com/watch?v=AUWxl0WdiNI https://www.youtube.com/watch?v=AUWxl0WdiNI
- fragmede 3y agoI would love to take an binary I worked on and feed it to this and get it printed and framed and hang it up somewhere.
- doubloon 3y agothere was a time on linux when you could cat /proc/kcore > /dev/video or something like that, some kind of old framebuffer device.
- makmanalp 3y agoWait this is really cool!! Reminds me of a thing I did a few years ago that I never completed (sigh) which was about visualizing access patterns in database storage engines. Here's the report with pics and videos if that sounds interesting! http://akmanalp.com/static/memory_trace.pdf http://akmanalp.com/static/memory_trace.pdf
- anonymousDan 3y agoThis is awesome. I'm actually working on a project at the moment that uses DynamoRio to capture write instructions. Would be super interesting to use your tool to visualise the access patterns. Is your code available somewhere?
- makmanalp 3y agoGah, it's probably in some old laptop somewhere and I can see if I can dig it up this weekend. What does your project do?
- anonymousDan 3y agoWe are looking at how you can take an existing in-memory datastore (e.g. Redis) and migrate it to use non-volatile memory (NVM) such as Intel Optane NVDIMMs for fast persistence. Similar pitch to this paper: https://www.usenix.org/conference/osdi20/presentation/zhang-wen https://www.usenix.org/conference/osdi20/presentation/zhang-... But better obviously :)
- makmanalp 3y agoThat sounds awesome! I can't seem to find it right now but I'm pretty sure I started out with one of the samples, I think memtrace_x86.c and that required only minimal modification to get it to do most of what I needed, minus eliminating addresses I don't care about, plotting and stuff which is not extra hard. If you get stuck and / or end up doing something cool, shoot me an email at hn at my website on my profile!
- nayuki 3y agoHow does the mouseover animation of that topmost image work? It seems random yet deterministic.
- cortesi 3y agoThe mousover always shows a contiguous section of the underlying file. So you're seeing a constant-length section of the space-filling curve centered on the cursor as you move the mouse about. The hex to the right is that same contiguous piece of the file. Sort of neat, actually.
- rendaw 3y agoI think maybe they're referring to the logo on the main page, not on a binary file? But that's an answer to another question I definitely had.
- cortesi 3y agoOh, right! That's just a bit of fun - a random pattern (picked to be vaguely visually pleasing), which is laid out on the Hilbert curve, and then offset by a calculated amount depending on the mouseover. So I'm just shifting the same pattern forwards and backwards in a ring buffer, basically.
- sad-lemon 3y agoI remember someone made scarves/blankets with certain binary visualizations. Very cool stuff.
- sgtnoodle 3y agoReminds me of a cool tool I made at a previous company. I combined a software validation ticket database with version control blame in order to paint the entire source code repo line-by-line to represent how "validated" it was. I generated a HTML listing with png thumbnails next to every source file, allowing you to quickly find areas of interest. Maybe that sounds boring, but it was flight code for a space ship.
- dr_kiszonka 3y agoSounds like a fun project!
- h4x0rr 3y agohttps://codisec.com/veles/ https://codisec.com/veles/ This is also really cool, similar idea but in 3d space
- fabian2k 3y agoThe default view reorders the bytes as far as I can tell. It's mentioned in the help, but I don't really get how to make use of this feature. Maybe I tried it on the wrong kinds of files, but I found the default view confusing. But it is a really cool tool, if you have to figure out a binary file format looking for patterns is certainly very useful.
- throw_pm23 3y agoThis reminds of old ZX Spectrum computers that had 48K RAM of which ~4K was the buffer containing what was actually shown on the screen. Some software was so space-constrained that when loading, it would temporarily have to write into the screen buffer, so when you loaded something it may have filled the screen with images similar to those of this tool. Then as loading completed, it would shuffle things around in memory to be able to use the screen for its intended purpose.
- mschaef 3y agoThis is a screenshot from a COSMAC ELF program that displayed a picture of Enterprise. https://en.m.wikipedia.org/wiki/File:Iconic_COSMAC_Elf_spaceship_scene.png https://en.m.wikipedia.org/wiki/File:Iconic_COSMAC_Elf_space... The machine was short enough on RAM by default that even at runtime, the screen buffer was shared. The random noise at the top of the screen is the program used to display the image on the bottom.
- giamma 3y agoOn the C64 some packers used screen memory to host the code. The first was The Screen Cruncher by 1001 Crew, a Dutch group. It's mentioned in this interview by its original author. https://web.archive.org/web/20040902191610/http://www.c64hq.com/index1152.php https://web.archive.org/web/20040902191610/http://www.c64hq....
- zX41ZdbW 3y agoI recently did a very similar thing, but for visualizing DNS records on IPv4: https://reversedns.space/ https://reversedns.space/
- zX41ZdbW 3y agoI also recommend checking this: https://codisec.com/binary-data-visualization/ https://codisec.com/binary-data-visualization/ - visualization of binaries by mapping n-grams into n-dimensional space.
- Piraty 3y agohttps://github.com/letoram/senseye/tree/1bab4b9c60ad43302e460a24de14a0ac136bea7f https://github.com/letoram/senseye/tree/1bab4b9c60ad43302e46... https://youtu.be/WBsv9IJpkDw https://youtu.be/WBsv9IJpkDw
- Cockbrand 3y agoThis reminds me of the old Amiga tool where one could visualize the RAM contents. As RAM wasn't cleared after a CTRL+A+A reset, one could find the graphics of the game one played before rebooting. Good times, but I can't seem to remember the tool's name.