4 ms·
Why, hello there Hacker News. Just a note that I've written two follow-on blog posts after this one, developing the idea a bit further. First, using a color fu
by cortesi 15y ago
Why, hello there Hacker News. Just a note that I've written two follow-on blog posts after this one, developing the idea a bit further.
First, using a color function that encodes local entropy to show how crypto keys and other high-entropy data can be picked straight out of a visualization:
http://corte.si/posts/visualisation/entropy/index.html http://corte.si/posts/visualisation/entropy/index.html
And next, using this in bulk on samples from a malware database, with what I think are beautiful and interesting results:
http://corte.si/posts/visualisation/malware/index.html http://corte.si/posts/visualisation/malware/index.html
- breadbox 15y agoIndeed; I highly recommend everyone read those posts. The original link is an interesting idea; the two follow-on posts are a surprisingly effective practical demonstration.
- morganpyne 15y agoBeautiful work as usual. The entropy visualisations in particular stuck out for me, and I can imagine how useful it is to be able to see this kind of information at a glance while dissecting malware.
- cortesi 15y agoCheers. :) I'm trying very, very hard to resist the urge to start writing a binary dissection tool based on this. I'm picturing a hex viewer with a space-filling curve navigation pane, with options to switch between different pixel layouts, and color maps. Then again, the last thing I need is another side-project.
- gbhn 15y agoDo most binaries have this high ratio of encrypted/obfuscated content? That is, would a checker which simply looked at high-entropy fraction of a binary be able to detect malware not in its database? Obviously this is trivial to defeat, but it might be the case that a broad defense such as this would force malware authors to let their code grow much bigger, which might in turn lead to other generic signatures. Would it be worthwhile?
- cortesi 15y agoUnfortunately perfectly legitimate compressed sections are both very entropic and very common, so just an entropy measurement won't be very useful for malware detection. I think there might be some promise in looking at patterns of entropy, though. What you can't see in that malware post is that very many pieces of malware show very similar layouts of entropy, even if the malware itself is very different. I weeded out similar images so that people could get an idea of the range of layouts, but the 50 visualizations on the blog are a "condensed" version of about 400 highly redundant images. Part of this is because different types of malware can still use the same "packer" - tools that pack a binary to obfuscate malware and make it hard to detect and reverse engineer. Part of it is just because there are certain techniques that are commonly used. All of this requires more study, but it's interesting.
- wazoox 15y agoThis would be not only a wonderful toy, but a really fantastic tool. Of course I'm not trying to influence you in the slightest way... :)
- jeremysalwen 15y agoI'm curious as to how it would look if you used some other measure of entropy besides local symbol frequency. Probably a general purpose compression algorithm should give you a good idea, like gzipping the blocks. Hmm, I'm not sure if this could work, but perhaps if you use a stream based compression algorithm you could relatively precisely see how much compressed data it takes to represent up to a certain point in the file, with only a single pass (rather than having to compress a huge number of local windows). Of course this is probably going to weight earlier parts of the file heavier, simply because the compression won't be calibrated yet to efficiently encode. So you could also run it on a byte-reversed version of the file, and a "rotated" version of the file (i.e. file[n/2:n]+file[0:n/2]), and a rotated-byte reversed version, and combine all those metrics together in some way (maybe min(entropy1,entropy2,entropy3,entropy4)). That way you could get an entropy measure which compensates for the sort of alphabet runs that fooled shannon entropy.
- cortesi 15y agoI'm working on something like this for the next evolution of the entropy visualizations. I've toyed with compression, but have had nicer results so far with other randomness estimators. I'll do a writeup once I have something to show. For a continuous compression function your idea of running on a rotated or reversed version of the file and then taking a minimum is a cunning one! Right now, I'm still working with sliding windows, but I'll keep it in mind if I turn back to compression.
- lsb 15y agoVery cool! I'm trying to visualize entropy based on Google Books' n-gram models of text.
- jeremysalwen 15y agoI'm curious what other randomness estimators you'll be using... (or maybe I'll just have to wait for another blog post).