7 ms·

Your hex editor should color-code bytes

614 points by tobr 6mo ago

bandrami 6mo ago
Emacs's hexl-mode does this, incidentally, though annoyingly by default it makes all faces the same color. I never understood why it defines the faces but then doesn't customize them.
- TeMPOraL 6mo ago
  What exactly does it do? I'm looking at hexl-mode sources in my Emacs, and I see it defining only two faces - hexl-address-region and hexl-ascii-region.
  - kleiba2 6mo ago
    That's correct, as far as I can tell: the first one is used for all hex values in the "main" are of the buffer, and the second one for the character representation of each byte in the right-hand side column.
- smithza 6mo ago
  A gist I (claude) threw together that maps this color scheme on hexl-mode. https://gist.github.com/zacharyasmith/bcfd91c9685c6cec30385a687a92477b https://gist.github.com/zacharyasmith/bcfd91c9685c6cec30385a...
js8 6mo ago
I think semantic coloring (based on structure) is more useful. Also (can't help as someone working with z/OS), if you really want to make hex output readable, I recommend using big-endian machine.
a_t48 6mo ago
I've started doing this with hashes in a CLI I'm working on. For slow prints, it's somewhat helpful https://asciinema.org/a/aD38Pk88CZgSZqtq https://asciinema.org/a/aD38Pk88CZgSZqtq but for debug dumps with many many hashes it really helps readability and tracking hashes across lines.
asibahi 6mo ago
When I read this article a few days ago it inspired me to create my own hex viewer : https://ar-ms.me/thoughts/3sl-a-sweet-hex-utility/ https://ar-ms.me/thoughts/3sl-a-sweet-hex-utility/ The cool thing about it imo (outside of colors) is a `--windows` flag. Which separates the hex view into partitions: so `-w 2:-3:5` shows the first two bytes on a line, then skips three bytes, then shows the next 5 bytes on a line, then the rest of the file. Easy to use combined with a terminal's up arrow.
samzong_ 6mo ago
[dead]
psychoslave 6mo ago
That said, even colored these dumps still feels unappealing to me — so yes this is admittedly subjective gut jumping in the conversation. I get that occult form can also be an attractive force. The post put on the table an interesting point about how to improve the presentation layer to fit what’s human cognition is good at spotting (in general, or at least for the expected audience with some training). And it does start proposing something with these color schemes. But isn’t it kind of missing the forest for the tree? Actually why do we even have rendering with [012345678ABCDEF], when a specific set of (colored/imaged?) glyphs would be able to make more obvious what’s on the table? Or even beyond the hexadecimal grouping, wouldn’t be more relevant to render something "intuitively" far more easy to grap without several layer of internalized interpretation through acculturation?
- GuB-42 6mo ago
  I can't think of anything better than a hex dump for representing raw binary data. I don't mean that there are no others, equally good representations, but hex dumps win because of familiarity. Of course, if you know about the format, there are better ways, but it goes beyond the scope of a hex editor, though the most advanced ones support things like template files and can display structured data, disassembly, etc...
- q3k 6mo ago
  > Actually why do we even have rendering with [012345678ABCDEF], when a specific set of (colored/imaged?) glyphs would be able to make more obvious what’s on the table? Most of us have internalized the relationship between digits in [0-9] for a very long time. Adding 6 more glyphs after that is quite easy (and they're also somewhat well known in the world), and after a while you stop even thinking about the glyphs consciously anyway. A hex 'C' intuitively means to me '4 from the end'. A hex 'F' intuitively means to me 'all 4 bits are set to 1'. I don't see any advantage to switching to a different glyph set for this base, other than disruption for disruption's sake. > Or even beyond the hexadecimal grouping, wouldn’t be more relevant to render something "intuitively" far more easy to grap without several layer of internalized interpretation through acculturation? Modern computers deal with 8-bit bytes, and their word sizes are a multiple of bytes - unless you're dealing with bit-packed data, which is comparatively rare (closest is bit twiddling of MMIO registers, which is when you sometimes switch to binary; although for a 4-bit hex nibble you can still learn arbitrary combinations of bits on/off into its value). This means you can group 8 bits into 1 digits of 8 bits as one glyph (alphabet too large to be useful), 2 digits of 4 (hex), 4 digits of 2 (alphabet too small to give a benefit over binary) and 8 digits of 1 (binary). Hex just works really well as a practical middle ground. Back when computers used 12 bit words (PDP-8 and friends) octal (4 digits of 3 bits represented in the 0-7 alphabet) was more popular.
azalemeth 6mo ago
I really like hexyl [1], which does this by default. https://github.com/sharkdp/hexyl https://github.com/sharkdp/hexyl
- kqr 6mo ago
  The author uses hexyl as an example of trying, but not doing it right.
Archelaos 6mo ago
This article made me think how I could use similar techinques to colour code the data in database tables. Has anyone here tried that and has some recommendations where to start, etc.?
- duckmysick 6mo ago
  DataGrip (and Pycharm by extension) lets you apply heatmap colors to dataframes and database tables: https://www.jetbrains.com/help/datagrip/tables-view-data.html#apply_a_heatmap https://www.jetbrains.com/help/datagrip/tables-view-data.htm... Another option would be to load data in pandas and display it in a Jupyter notebook with style.background_gradient() Polars delegate styling to Great Tables, but it's also doable there: https://posit-dev.github.io/great-tables/get-started/colorizing-with-data.html https://posit-dev.github.io/great-tables/get-started/coloriz...
xyx0826 6mo ago
If you analyze binary files often, I highly recommend binvis - http://binvis.io/ http://binvis.io/. It creates a colored minimap for files it loads and has two available arrangements. Pixel color is based on range of bytes, eg ASCII/null bytes/FF bytes. Besides, it’s a pretty basic hex viewer that runs in your browser. The minimap is extremely powerful for identifying interesting areas and patterns in unknown data.
- pratyahava 6mo ago
  > it’s a pretty basic hex viewer that runs in your browser excuse me? "basic" and "runs in your browser" together sound very contradictory to me. while doing things i actually feel (yes, emotionally) much better when there is no browser open on my machine, but only text editors, vcs gui and file managers, and terminals of course. and sometimes i reject an idea to start a browser just thinking how much ram it will take (ha, what a progress we have done - one github issue tab, with text only and no images, takes 180mb of ram).
  - franga2000 6mo ago
    It's basic bause it does like two things. It's not advanced or complex. HN is also a basic forum, even though it runs in a browser.
dspillett 6mo ago
Everything should try do some basic syntax highlighting IMO. Not too much, or it just becomes a sea of formatting that doesn't help at all. It is surprising how much difference just a little splash of colour can make if it isn't overdone. If possible, always include configuration options for the user though, so those with colour-blindness issues can tweak things to their needs, those who are just fussy can make the output fit with their finely adjusted system-wide colour schemes¹, and even better, where you can, allow bold/italic/other as well as colours so that those who barely see colour at all can play too. Of course none of this helps those using screen-readers and other tech, so make sure that all your fancy colouring & such is additive so if it is all “lost” no meaning is absolutely lost with it. -------- [1] Some people can be very vocal about this, more so than if highlighting isn't possible at all. If you give any output formatting they'll expect you to match, or be able to be made to match, their preferred style.
- finaard 6mo ago
  I'd recommend for every developer to get one or more colourblind friends. I have some, and regularly send them screenshots of what I'm working on to get feedback what they can see and what they can read/can't read. They've been absolutely invaluable for making sure their kind of people can't use my apps properly.
  - gortok 6mo ago
    8% of the male population has some form of colorblindness (for women it’s around 0.5%). I have deuteranomaly colorblindness. If you search for images on the internet related to that type of colorblindness you’ll find representations of how we see color and how we see the world. It is not a fun condition to have, and leads to lots of problems in my everyday life. This blog post accidentally accentuated that issue, since the colors are (to what I can understand) very similar looking to me as a colorblind person. 1 in 12 men and 1 in 200 women go through the same sorts of experiences, and it’s worth it, if you aren’t color deficient, to try out some of the colorblindness sites and see the world as we do. https://www.colourblindawareness.org/colour-blindness/colour-blindness-experience-it/ https://www.colourblindawareness.org/colour-blindness/colour...
greatgib 6mo ago
To me the random colors at each byte is messing up with my brain making it hard to fast identify C0 or any other value that I could more easily identify in all black. But color would be nice more based on the bytes logic. Eventually the 00 in a shaded grey instead of black, and in best case scenario by logic unit based on your protocol. And worst case scenario by groups of words or so.
7bit 6mo ago
> it’s much easier to pick out the unique byte when it’s a different color! human brains are really good at spotting visual patterns—given the right format Don't really see the advantage. Unique bytes have no unique meaning across data types. The only good syntax highlight to me is 00 and perhaps FF. But that's my opinion of course. Anything else that has no direct relation to what you're looking at is meaningless.
- gblargg 6mo ago
  It would be interesting to do a heat map coloring based on frequency of that value.
- masklinn 6mo ago
  > The only good syntax highlight to me is 00 and perhaps FF. But that's my opinion of course. Would probably make the most sense to have various ranges you can enable depending on what you’re looking for (or to look for patterns) e.g. for single byte coloration I could see - nul - printable / non-printable ascii - non-ascii - UTF8 leading / continuation - separators - start/end pairs (both printable and non printable)
adv_zxy 6mo ago
radare2 also has excellent hex viewing/editing support, if one manages to grok the usage of it.
- snvzz 6mo ago
  The iaito GUI suddenly got good recently.
delta_p_delta_x 6mo ago
> Your hex editor should colour-code bytes so it is easier for users to distinguish patterns > Article is fully in lowercase, which makes it harder for readers to make out sentences and the flow of the article > mfw the irony
- gblargg 6mo ago
  The text smashed up to the left border doesn't help either.
NooneAtAll3 6mo ago
Why did author decide that best way to demonstrate his idea would be by cutting contrast in half? color-coding might be a great solution, but you don't really know beforehand which byte values are important. Manually selecting C0 to make it stand out it just ctrl+f with extra steps. (But I wouldn't mind something like "color 00 separate from ascii separate from the rest)
- seszett 6mo ago
  > Manually selecting C0 to make it stand out That's not what they did, actually. C0 is the only byte in there that's above 3F or so, and it's far from it. Hence the very different colour, and the lack of contrast between the colours of the other bytes.
- oblio 6mo ago
  OP is automatically selecting similar colors across the spectrum. C0 is just very far from the rest so the color is very different to the other bytes.
cuechan 6mo ago
For anyone who regularly has to look at/analyze binary files, i highly recommend ImHex [1]. Its a hex editor built with imgui and has a lot of built in tools. Imo the best feature is the data structure editor. You can write a data type definition similar to C and it overlays it on the hexdump and parses it in a structured way while you type. It also has a node based editor. 1: https://github.com/WerWolv/ImHex https://github.com/WerWolv/ImHex
- altairprime 6mo ago
  Does it do color highlighting by value of hex bytes, though?
  - criddell 6mo ago
    Yes. https://docs.werwolv.net/imhex#screenshots https://docs.werwolv.net/imhex#screenshots
    - seritools 6mo ago
      as far as i can tell, no it does not. it only desaturates 00 in particular. the other colors you see in the screenshots come from matched formats/patterns. it does not do direct coloring based on byte value.
      - criddell 6mo ago
        Under the Edit menu select "Highlighting Rules..." and you can define or load any set of rules you can imagine.
        altairprime 6mo ago
        Thanks! How many rules would it take to create 256 different colors for 00..FF?
        deleted 6mo ago
kokakiwi 6mo ago
ImHex (https://imhex.werwolv.net/ https://imhex.werwolv.net/) is also a really nice Hex editor with tons of plugins (patterns, file support, etc.) and even an embedded language for adding more patterns easily
- aa-jv 6mo ago
  +1 for ImHex - I've had a lot of fun with Pattern Language lately .. its quite fruitful to, for instance, have an AI take a few headers worth of input to produce some PL code which highlights all the things very well. For a while I almost forget I'm in a hex editor and not in some custom data wrangling environment ..
deleted 6mo ago
[deleted]
ralferoo 6mo ago
I actually stopped reading after the intro because I fundamentally disagreed with its premise. The "find the C0" took me about 1/4 second with uncoloured. Looking at the coloured took my eyes about 3 seconds to recover from the colour overload, then I was scanning down and found the colours so distracting with the constant switching between orange, pink and yellows than it took me a total of about 5 seconds to scan down as far as the blue C0. Maybe if it was all uncoloured and blue just for that, I might have actually noticed it looking different earlier. It's been a while since I used hexedit on Linux, but I think that highlighted search results in reverse colours, just like less does for text search. Personally, I'd prefer that to colours.
- assimpleaspossi 6mo ago
  Agree. This is what indentation and brackets, etc. are for. So much color makes it a clown car image to wade through.
  - teddyh 6mo ago
    The old disparaging term was “angry fruit salad”: <http://www.catb.org/~esr/jargon/html/A/angry-fruit-salad.html http://www.catb.org/~esr/jargon/html/A/angry-fruit-salad.htm...>
red_admiral 6mo ago
My hex editor should let me turn syntax highlighting on and off; follow my personal color theme (and not produce light gray on white in the terminal); and let me highlight specific things I'm searching for like OD OA or FF FE.
roelschroeven 6mo ago
When you're going to color-code bytes in a hex dump, I would expect each ASCII character in the right column to have the same color as the hex byte in the left column, making it easier to pair them. I wonder why that wasn't done here.
- pragma_x 6mo ago
  I came in here to comment the same. Our brains are wonderful pattern recognition engines and the reader would absolutely be able to more readily see the correlation between hex and character representations this way. It might even accelerate learning hex values in the process.
- simonomi 6mo ago
  i tried it out, but ended up preferring fewer categories. the colors already exist in the main panel and i still find non-graphic ASCII/graphic ASCII/non-ASCII to be useful categories to have
nticompass 6mo ago
I used to use wxHexEditor and that had a feature where I could select a section of the file and highlight it in a color. When I was working to decode a certain file format, I used that to color-code different sections of the file and it was super useful. Those color-codes were stored in a separate file so you could load them back in.
ChrisRR 6mo ago
What a bad way to illustrate your point by using such similar looking pastel colours
- altairprime 6mo ago
  Ironically, my complaint about the article was that the author apparently only uses typical human vision ranges here, rather than mapping 00..FFh onto an OKlab gradient of hue 0..359° that rewards those few of us with impeccable color fidelity with even better highlighting than most can see :) No doubt there’s value in contentful highlighting but I’d rather just have a straight hue translation on the circle at a fixed luminosity. There’s only 256 hues to discern, after all! And what a pleasure it would be to learn to read hex code by hue alone :D
  - simonomi 6mo ago
    for the record, i did use OKlab to pick my colors ! (see the CSS for my site https://github.com/simonomi/simonomi.github.io/blob/aa55ea855c17d5253d85d0440f41871aadc27b83/_includes/code.css#L51-L68 https://github.com/simonomi/simonomi.github.io/blob/aa55ea85...) i'm not opposed to using a unique color per byte, but i didn't want 256 different CSS classes for byte coloring, so i manually picked the 16 most distinct hues i could
    - altairprime 6mo ago
      You did great! I could tell the colors were equidistant from how they resembled Penumbra’s schemes. I just have really abnormal vision and have spent several years reading raw hex codes for a project. So I crave more color, heh :D
PunchyHamster 6mo ago
I wonder how hard it would be to color code repeating sequences
orphea 6mo ago
I get the idea but those specific examples are awful - not enough contrast.
Someone 6mo ago
The first example is “go ahead, try to find the single C0 in these bytes” and then argues one should highlight C0 bytes. If that’s true, how does the tool know I will be looking for C0 bytes and not for 03, D3, etc? The logical conclusion of that would be that the hex editor should uniquely color code every byte. And following the other examples even that’s not enough. The proposed solution is to create groups of byte values that each get their unique color. I think that helps, but we can do better: add a search feature. That tells your editor what you are looking for. Once you enter a search string, it can highlight all hits. Yes, “colorful output in a hexdump is useful for the same reason that syntax highlighting for code is useful”, but do you know what syntax highlighting needs? Knowledge of the expected content of a file. Without that, a hex editor at best can guess at how to color-code stuff. IMO, if you want to add syntax coloring to a hex editor, give it pluggable syntax coloring and heuristics for deciding which one to use when. While at it, also let those plugins control where to break lines, whether to show hex at all (why show it at all if a file has a few paragraphs of English text or an array of IEEE doubles?), etc. Those plug-ins will make errors and sometimes, users will want to see all byte values, so you’ll need a way for the user to override them.
- Antibabelic 6mo ago
  I don't think that's quite the point that example was intended to illustrate. The idea is not the you're looking for C0 bytes or any other kinds of bytes in particular, but rather that it's easier to fish out unique and interesting information in a sea of noise when you have color-coded bytes: like the fact that there's a conspicuous lonely C0 or some other value or series of values that stand out.
  - bombcar 6mo ago
    Exactly - though the writing style buries the lede until the end; the recommendation is to have at least a different color for each of the first nibbles (e.g, 0x is one color, 1x another). As a minimum that makes some outliers pop out. He then has other recommendations like highlighting the ASCII range, etc. As a note, the some up there is load-bearing - color may lull you into complacency where the difference between 01 and 0F is major and important but not highlighted. More complicated regex built color tools designed to highlight "anomalies" could be developed but then you need to define what anomalies are (patters, places where a pattern changes, etc).
fleebee 6mo ago
> having more colors makes it possible to recognize more complex patterns The implicit cost here is that the simple patterns become harder to recognize when every byte is only subtly differently colored. Rather than give everything a different color, I'd rather have the important stuff highlighted. In the comparisons given, I think hexyl's highlighting scheme is significantly more useful.
whizzter 6mo ago
Anyone tried using Kaitai descriptions? It seems like a fairly flexible system that would be an excellent starting point for a hex-editor that wants to add good higher level coloring (and perhaps even editing).
randusername 6mo ago
I think this is a cool idea. I'd want to take it further by using full RGB and cycling through some colormaps with different properties. Sequential, diverging, cyclic like in matplotlib. https://matplotlib.org/stable/users/explain/colors/colormaps.html https://matplotlib.org/stable/users/explain/colors/colormaps... Can't think of a specific use-case off the top of my head, but sometimes I just want the "feel" of the data when I'm plotting something, and maybe the same scattershot approach would pay off at some point on unknown hex data if it was an option.
myfonj 6mo ago
When (rarely) using hex editors, one thing constantly comes to my mind: isn't base 16 arabic-roman numerals a bit awkward for "skimmable" overview? Color-coding indeed helps immensely there, but wouldn't simply letting bits and bops shine in eight bit clusters, resembling the "physical" shape of the eight-bit byte, be somewhat more readable? We even have characters in the Unicode for representing 0..255 variations, actually two distinct groups: Braille (arguably a bit misuse for binary) and octants (accompanied by older predecessors). So what would be |65|97|66|98|67|99|32|126|32|72|101|108|108|111|44|32|109|111|109|33|32|240|159|166|132| in base-10 or |41|61|42|62|43|63|20|7e|20|48|65|6c|6c|6f|2c|20|6d|6f|6d|21|20|f0|9f|a6|84| in base-16, could be |⢈|⢊|⡈|⡊|⣈|⣊|⠂|⡾|⠂|⠌|⢪|⠮|⠮|⣮|⠦|⠂|⢮|⣮|⢮|⢂|⠂|⠛|⣵|⡣|⠡| in Braille, or |𜵲|𜵶|𜴷|𜴻|𜶭|𜶱|𜴀|𜵯|𜴀|𜴋|𜶔|𜴭|𜴭|𜷟|𜴫|𜴀|𜶢|𜷟|𜶢|𜵴|𜴀|(⁕)|𜷢|𜵖|𜴙| using octants. Most significant bit is at the top left here, the least one is bottom right -- it felt somewhat intuitive to me this way, your intuition may differ, obviously. Or, naturally, "AaBbCc ~ Hello, mom! <Unicorn Emoji>" as a "UTF-8" text. Try: http://myfonj.github.io/tst/byte-dec-hex-braille-octant.html http://myfonj.github.io/tst/byte-dec-hex-braille-octant.html) Test (with added "CSS" variant and "highlight" of empty dots): http://myfonj.github.io/tst/byte-visualisation-exploration.html http://myfonj.github.io/tst/byte-visualisation-exploration.h... (⁕) HN apparently eats upper-half block. Amusing that only this particular ("old", as referred earlier) one got filtered out… Also caveat: Android phones have messed-up Braille block due outdated broken embedded font, so all patterns with dots in the left half appear in the right instead. Long reported, not fixed, IIRC.
- aa-jv 6mo ago
  This is a great idea and I concur with your line of thought - there is room for expression in the realm of number representations .. have you considered building an ImHex plugin that would illustrate your point? The use of octants is particularly intriguing .. One thing I often ponder on, along similar lines, is whether I can write some clever plugin that would put FF ChartWell - a font which uses ligatures to render useful graphs out of boring numerical data - into use, within ImHex. Seen how ligatures can be used this way? https://typographica.org/typeface-reviews/chartwell/ https://typographica.org/typeface-reviews/chartwell/ Your idea of discovering new means of representing numeral data put me in mind of ligature hacking, in any case ...
bwiggs 6mo ago
DEFCON30, Mayhem CTF. We were given a file full of random bytes. The flag was in there somewhere. It was too random to be encrypted, there wasn't any structure. `file` didn't return anything, truly just a bag of bytes. I had decided to install `hexyl` as an alternative option to some of the other hex editors installed o my linux machine. All the bytes were colored grey. I scrolled the file and noticed a blip of yellow. A random golden `{` amongst all the noise. Weird. The next colored byte was a `C`, then `T`, `F`. --- At that time, I was mostly using HexFiend to look at raw files, which didn't have byte coloring. For DEFCON I had decided to drive my linux machine. I had ghex installed, but i had also decided to install and try `hexyl` via cli. So seeing bytes in color was purely by chance that I had installed it. I eventually posted an issue to ghex to add color support. https://gitlab.gnome.org/GNOME/ghex/-/issues/60 https://gitlab.gnome.org/GNOME/ghex/-/issues/60 I need to see if I can find the file and post it on that blog post. https://bwiggs.com/posts/2023-08-31-hacking-in-color/ https://bwiggs.com/posts/2023-08-31-hacking-in-color/
- abcd_f 6mo ago
  > It was too random to be encrypted That's a rather odd remark.
  - bwiggs 6mo ago
    You would still expect some amount of file structure, ex byte headers or something at the beginning/end of file. no?
    - Crestwave 6mo ago
      That would be expected for encodings or container file formats. Straight-up encryption like AES produces results that are visually indistinguishable from random data.
    - throwawaysoxjje 6mo ago
      I’d expect the greater length of the encrypted data (which should look random) vs the structured header/footer to rapidly push the Shannon entropy to the maximum
mplanchard 6mo ago
Anyone know of a good emacs package for this?
- smithza 6mo ago
  A gist I (claude) threw together that maps this color scheme on hexl-mode. https://gist.github.com/zacharyasmith/bcfd91c9685c6cec30385a687a92477b https://gist.github.com/zacharyasmith/bcfd91c9685c6cec30385a...
soegaard 6mo ago
This was a great article and inspired me to add support for binary files in `peek`. https://soegaard.github.io/peek/#%28part._binary-files%29 https://soegaard.github.io/peek/#%28part._binary-files%29 For me the key insight is that similar values should get similar colors. And since Fx and 0x are "similar" the color palette should be cyclic.
sidewndr46 6mo ago
compare a simple high contrast display to one of that makes it difficult to read and hurts my eyes? Sure! absolutely! I'll pass thank you
MisterTea 6mo ago
> compare that to one with colors: The colors make it worse as I'm red-green colorblind. Looking at that mess is eye strain. Honestly I mostly prefer syntax highlighting turned off as it causes eye strain. I have found the black on light yellow theme of the Acme editor to be a very comfortable monochrome color scheme.
stronglikedan 6mo ago
> go ahead, try to find the single C0 in these bytes: Ctrl+C, Ctrl+F, Ctrl_V... Easy!
- oblio 6mo ago
  That part isn't literal. It's meant to be: "find the outlier in this sea of hex-coded bytes".
taeric 6mo ago
I find it funny that I found the "single C0" pretty much instantly. I grant that the post largely has a point, mind you. But scanning for a needle in a haystack is something that you just don't often do? I am, of course, now very curious how often folks are using hex editors. And itching for an excuse to open a file that way. :D
- traderj0e 6mo ago
  The first time I used one was in high school when I was playing some video game in an emulator and wanted to convert the game save to another emulator's format, cause halfway through I realized the first emu had issues with the game. There was no conversion software, but with the hex editor I figured out that you needed to remove some header and change the endianness.
shmerl 6mo ago
Any good neovim plugin recommendation for hex editing?
- contingencies 6mo ago
  I came to scan for the same thing, basically. Is there a vi(m) hex editor? Hexmode https://github.com/fidian/hexmode https://github.com/fidian/hexmode or vim-deadc0de https://www.vim.org/scripts/script.php?script_id=6033 https://www.vim.org/scripts/script.php?script_id=6033 With decent color support? No.
  - traderj0e 6mo ago
    I was kinda expecting vim with whatever common plugins I installed to do this already, but no. It does nice highlighting for all sorts of text formats, even non-code things like CSV, but not hex.
0xfalafel 6mo ago
Shameless plug: I have written an hexadecmial editor (in Rust) that colors bytes. It looks similar to hexyl. https://github.com/0xfalafel/hextazy https://github.com/0xfalafel/hextazy Some nice features are: - robust undo - insertions (not just overwriting) - inspect the value of selected bytes - search I have also recently discover this other project that is pretty cool: https://github.com/mentebinaria/dz6 https://github.com/mentebinaria/dz6
wang_li 6mo ago
Color coding is a simplistic, half-assed way of doing semantic analysis. Go full assed and make an /etc/magic aware file decoder that decodes and highlights anomalies.
dhosek 6mo ago
I found the coloring in most of the examples to be more distracting than helpful. I can see cases where it could be helpful (e.g., highlighting bytes in the 0x20–0x7E range for spotting ASCII strings, or a fancier one that can identify UTF-8 strings, or better still, invalid sequences in what might otherwise be UTF-8), but most of the cases here didn’t really help all that much for me.
TheRealPomax 6mo ago
It'd be nicer if the hex coloring actually matched the "ascii" coloring as well. Orange on the left but green on the right does not help find things.
nickwanninger 6mo ago
I added type-based color printing to my hexdump in my kernel [1] if anyone wants to have that code. It was instrumental in finding bugs quickly in wee hours of the night sometimes, especially if you have heap corruption. ---- [1] https://github.com/ChariotOS/chariot/blob/e046849c668458d25eee4268776eb2ee85d88a62/kernel/lib/util.cpp#L33-L100 https://github.com/ChariotOS/chariot/blob/e046849c668458d25e...
xvilka 6mo ago
Rizin[1][2] does exactly this, also there is a compact hex-II[3] mode. [1] https://rizin.re https://rizin.re [2] https://github.com/rizinorg/rizin https://github.com/rizinorg/rizin [3] https://speakerdeck.com/ange/no-more-dumb-hex https://speakerdeck.com/ange/no-more-dumb-hex
jcalvinowens 6mo ago
If you just want to see patterns, and don't actually need to see the values, you can go a step further and simply visualize the data as a bitmap, e.g. dd if=/dev/urandom bs=$[256*256] count=1 | display -size 256x256 -depth 8 GRAY:- You can do the same thing with audio, which makes different sorts of patterns obvious, e.g. dd if=/dev/urandom bs=$[256*256] count=1 | aplay -c 1
- felooboolooomba 6mo ago
  > dd if=/dev/urandom bs=$[256*256] count=1 | aplay -c 1 This plays sounds, sounds like "Liberate Mae?"
- EvanAnderson 6mo ago
  I made little CLI tool back in the MS-DOS days to dump binaries into VGA mode 0x13. It allowed me to vary the width of "line" wrapping. It was a killer tool for seeing data in binaries.
ape4 6mo ago
As a next level beyond coloring... how about adding some interactivity? How about a slider to control the brightness of each type of byte. Turn everything but text to 10% when when you're looking for some words in a binary file.
Findecanor 6mo ago
If you're making a hex editor and going to have colour coding, I'd think you expend some effort to make the colouring schemes configurable — and easy to configure and change. Maybe load and save as separate files. Different colouring schemes for different types of data.
leetrout 6mo ago
ipython saved me this week when it color coded b'\x100' Which was not obvious to me in the print output that it is \x10 and a literal 0.
vesui 6mo ago
Fantastic writeup, and it's so cool to see you at the front of HN! This article compels me to add colored syntax highlighting for hex code blocks on my own blog...
amelius 6mo ago
Or maybe use an LLM to auto-format your hex dumps based on structure?
Xophmeister 6mo ago
If you have a Kaitai spec for the format, then its IDE[0] will do structural highlighting for you. This is web-based, but it doesn’t seem like too much of a stretch to implement a TTY version. [1]: ide.kaitai.io
skalidindi3 6mo ago
https://github.com/skalidindi3/cxxd https://github.com/skalidindi3/cxxd I wrote this a LONG while back. At the time, I was fresh out of college and my first job included a lot of reverse engineering communication protocols to make machines work together for automation purposes. I will personally testify to how useful it was to see visual patterns to aid in this. The single biggest benefit was seeing that one particular protocol switched endianness WITHIN a specific packet.
deepsun 6mo ago
Rob Pike says colors add cognitive load. He calls syntax highlighting "juvenile".
- fc417fc802 6mo ago
  I think that depends on quantity. There are certainly some juvenile examples that take things too far.
eviks 6mo ago
And the lon leading zero should be a space instead, will cut down on visual noise in the examples in the post a lot to let your color codes do the rest
deleted 6mo ago
[deleted]
barbs 6mo ago
Reminds me of this blog post where someone was trying to reverse-engineer a hand-rolled encryption protocol in a crossword app by colorizing the output and looking for patterns. https://www.muppetlabs.com/~breadbox/txt/acre.html https://www.muppetlabs.com/~breadbox/txt/acre.html
account42 6mo ago
Those examples look horrible. Rainbow barf with low contrast is not an improvement to anything.

614 points by tobr 6mo ago

Emacs's hexl-mode does this, incidentally, though annoyingly by default it makes all faces the same color. I never understood why it defines the faces but then doesn't customize them.

TeMPOraL 6mo ago
What exactly does it do? I'm looking at hexl-mode sources in my Emacs, and I see it defining only two faces - hexl-address-region and hexl-ascii-region.
- kleiba2 6mo ago
  That's correct, as far as I can tell: the first one is used for all hex values in the "main" are of the buffer, and the second one for the character representation of each byte in the right-hand side column.
smithza 6mo ago
A gist I (claude) threw together that maps this color scheme on hexl-mode. https://gist.github.com/zacharyasmith/bcfd91c9685c6cec30385a687a92477b https://gist.github.com/zacharyasmith/bcfd91c9685c6cec30385a...

js8 6mo ago

I think semantic coloring (based on structure) is more useful. Also (can't help as someone working with z/OS), if you really want to make hex output readable, I recommend using big-endian machine.

a_t48 6mo ago

I've started doing this with hashes in a CLI I'm working on. For slow prints, it's somewhat helpful https://asciinema.org/a/aD38Pk88CZgSZqtq https://asciinema.org/a/aD38Pk88CZgSZqtq but for debug dumps with many many hashes it really helps readability and tracking hashes across lines.

asibahi 6mo ago

When I read this article a few days ago it inspired me to create my own hex viewer : https://ar-ms.me/thoughts/3sl-a-sweet-hex-utility/ https://ar-ms.me/thoughts/3sl-a-sweet-hex-utility/ The cool thing about it imo (outside of colors) is a `--windows` flag. Which separates the hex view into partitions: so `-w 2:-3:5` shows the first two bytes on a line, then skips three bytes, then shows the next 5 bytes on a line, then the rest of the file. Easy to use combined with a terminal's up arrow.

samzong_ 6mo ago

[dead]

psychoslave 6mo ago

That said, even colored these dumps still feels unappealing to me — so yes this is admittedly subjective gut jumping in the conversation. I get that occult form can also be an attractive force. The post put on the table an interesting point about how to improve the presentation layer to fit what’s human cognition is good at spotting (in general, or at least for the expected audience with some training). And it does start proposing something with these color schemes. But isn’t it kind of missing the forest for the tree? Actually why do we even have rendering with [012345678ABCDEF], when a specific set of (colored/imaged?) glyphs would be able to make more obvious what’s on the table? Or even beyond the hexadecimal grouping, wouldn’t be more relevant to render something "intuitively" far more easy to grap without several layer of internalized interpretation through acculturation?

GuB-42 6mo ago
I can't think of anything better than a hex dump for representing raw binary data. I don't mean that there are no others, equally good representations, but hex dumps win because of familiarity. Of course, if you know about the format, there are better ways, but it goes beyond the scope of a hex editor, though the most advanced ones support things like template files and can display structured data, disassembly, etc...
q3k 6mo ago
> Actually why do we even have rendering with [012345678ABCDEF], when a specific set of (colored/imaged?) glyphs would be able to make more obvious what’s on the table? Most of us have internalized the relationship between digits in [0-9] for a very long time. Adding 6 more glyphs after that is quite easy (and they're also somewhat well known in the world), and after a while you stop even thinking about the glyphs consciously anyway. A hex 'C' intuitively means to me '4 from the end'. A hex 'F' intuitively means to me 'all 4 bits are set to 1'. I don't see any advantage to switching to a different glyph set for this base, other than disruption for disruption's sake. > Or even beyond the hexadecimal grouping, wouldn’t be more relevant to render something "intuitively" far more easy to grap without several layer of internalized interpretation through acculturation? Modern computers deal with 8-bit bytes, and their word sizes are a multiple of bytes - unless you're dealing with bit-packed data, which is comparatively rare (closest is bit twiddling of MMIO registers, which is when you sometimes switch to binary; although for a 4-bit hex nibble you can still learn arbitrary combinations of bits on/off into its value). This means you can group 8 bits into 1 digits of 8 bits as one glyph (alphabet too large to be useful), 2 digits of 4 (hex), 4 digits of 2 (alphabet too small to give a benefit over binary) and 8 digits of 1 (binary). Hex just works really well as a practical middle ground. Back when computers used 12 bit words (PDP-8 and friends) octal (4 digits of 3 bits represented in the 0-7 alphabet) was more popular.

azalemeth 6mo ago

I really like hexyl [1], which does this by default. https://github.com/sharkdp/hexyl https://github.com/sharkdp/hexyl

kqr 6mo ago
The author uses hexyl as an example of trying, but not doing it right.

Archelaos 6mo ago

This article made me think how I could use similar techinques to colour code the data in database tables. Has anyone here tried that and has some recommendations where to start, etc.?

duckmysick 6mo ago
DataGrip (and Pycharm by extension) lets you apply heatmap colors to dataframes and database tables: https://www.jetbrains.com/help/datagrip/tables-view-data.html#apply_a_heatmap https://www.jetbrains.com/help/datagrip/tables-view-data.htm... Another option would be to load data in pandas and display it in a Jupyter notebook with style.background_gradient() Polars delegate styling to Great Tables, but it's also doable there: https://posit-dev.github.io/great-tables/get-started/colorizing-with-data.html https://posit-dev.github.io/great-tables/get-started/coloriz...

xyx0826 6mo ago

If you analyze binary files often, I highly recommend binvis - http://binvis.io/ http://binvis.io/. It creates a colored minimap for files it loads and has two available arrangements. Pixel color is based on range of bytes, eg ASCII/null bytes/FF bytes. Besides, it’s a pretty basic hex viewer that runs in your browser. The minimap is extremely powerful for identifying interesting areas and patterns in unknown data.

pratyahava 6mo ago
> it’s a pretty basic hex viewer that runs in your browser excuse me? "basic" and "runs in your browser" together sound very contradictory to me. while doing things i actually feel (yes, emotionally) much better when there is no browser open on my machine, but only text editors, vcs gui and file managers, and terminals of course. and sometimes i reject an idea to start a browser just thinking how much ram it will take (ha, what a progress we have done - one github issue tab, with text only and no images, takes 180mb of ram).
- franga2000 6mo ago
  It's basic bause it does like two things. It's not advanced or complex. HN is also a basic forum, even though it runs in a browser.

dspillett 6mo ago

Everything should try do some basic syntax highlighting IMO. Not too much, or it just becomes a sea of formatting that doesn't help at all. It is surprising how much difference just a little splash of colour can make if it isn't overdone. If possible, always include configuration options for the user though, so those with colour-blindness issues can tweak things to their needs, those who are just fussy can make the output fit with their finely adjusted system-wide colour schemes¹, and even better, where you can, allow bold/italic/other as well as colours so that those who barely see colour at all can play too. Of course none of this helps those using screen-readers and other tech, so make sure that all your fancy colouring & such is additive so if it is all “lost” no meaning is absolutely lost with it. -------- [1] Some people can be very vocal about this, more so than if highlighting isn't possible at all. If you give any output formatting they'll expect you to match, or be able to be made to match, their preferred style.

finaard 6mo ago
I'd recommend for every developer to get one or more colourblind friends. I have some, and regularly send them screenshots of what I'm working on to get feedback what they can see and what they can read/can't read. They've been absolutely invaluable for making sure their kind of people can't use my apps properly.
- gortok 6mo ago
  8% of the male population has some form of colorblindness (for women it’s around 0.5%). I have deuteranomaly colorblindness. If you search for images on the internet related to that type of colorblindness you’ll find representations of how we see color and how we see the world. It is not a fun condition to have, and leads to lots of problems in my everyday life. This blog post accidentally accentuated that issue, since the colors are (to what I can understand) very similar looking to me as a colorblind person. 1 in 12 men and 1 in 200 women go through the same sorts of experiences, and it’s worth it, if you aren’t color deficient, to try out some of the colorblindness sites and see the world as we do. https://www.colourblindawareness.org/colour-blindness/colour-blindness-experience-it/ https://www.colourblindawareness.org/colour-blindness/colour...

greatgib 6mo ago

To me the random colors at each byte is messing up with my brain making it hard to fast identify C0 or any other value that I could more easily identify in all black. But color would be nice more based on the bytes logic. Eventually the 00 in a shaded grey instead of black, and in best case scenario by logic unit based on your protocol. And worst case scenario by groups of words or so.

7bit 6mo ago

> it’s much easier to pick out the unique byte when it’s a different color! human brains are really good at spotting visual patterns—given the right format Don't really see the advantage. Unique bytes have no unique meaning across data types. The only good syntax highlight to me is 00 and perhaps FF. But that's my opinion of course. Anything else that has no direct relation to what you're looking at is meaningless.

gblargg 6mo ago
It would be interesting to do a heat map coloring based on frequency of that value.
masklinn 6mo ago
> The only good syntax highlight to me is 00 and perhaps FF. But that's my opinion of course. Would probably make the most sense to have various ranges you can enable depending on what you’re looking for (or to look for patterns) e.g. for single byte coloration I could see - nul - printable / non-printable ascii - non-ascii - UTF8 leading / continuation - separators - start/end pairs (both printable and non printable)

adv_zxy 6mo ago

radare2 also has excellent hex viewing/editing support, if one manages to grok the usage of it.

snvzz 6mo ago
The iaito GUI suddenly got good recently.

delta_p_delta_x 6mo ago

> Your hex editor should colour-code bytes so it is easier for users to distinguish patterns > Article is fully in lowercase, which makes it harder for readers to make out sentences and the flow of the article > mfw the irony

gblargg 6mo ago
The text smashed up to the left border doesn't help either.

NooneAtAll3 6mo ago

Why did author decide that best way to demonstrate his idea would be by cutting contrast in half? color-coding might be a great solution, but you don't really know beforehand which byte values are important. Manually selecting C0 to make it stand out it just ctrl+f with extra steps. (But I wouldn't mind something like "color 00 separate from ascii separate from the rest)

seszett 6mo ago
> Manually selecting C0 to make it stand out That's not what they did, actually. C0 is the only byte in there that's above 3F or so, and it's far from it. Hence the very different colour, and the lack of contrast between the colours of the other bytes.
oblio 6mo ago
OP is automatically selecting similar colors across the spectrum. C0 is just very far from the rest so the color is very different to the other bytes.

cuechan 6mo ago

For anyone who regularly has to look at/analyze binary files, i highly recommend ImHex [1]. Its a hex editor built with imgui and has a lot of built in tools. Imo the best feature is the data structure editor. You can write a data type definition similar to C and it overlays it on the hexdump and parses it in a structured way while you type. It also has a node based editor. 1: https://github.com/WerWolv/ImHex https://github.com/WerWolv/ImHex

altairprime 6mo ago
Does it do color highlighting by value of hex bytes, though?
- criddell 6mo ago
  Yes. https://docs.werwolv.net/imhex#screenshots https://docs.werwolv.net/imhex#screenshots
  - seritools 6mo ago
    as far as i can tell, no it does not. it only desaturates 00 in particular. the other colors you see in the screenshots come from matched formats/patterns. it does not do direct coloring based on byte value.
    - criddell 6mo ago
      Under the Edit menu select "Highlighting Rules..." and you can define or load any set of rules you can imagine.
      - altairprime 6mo ago
        Thanks! How many rules would it take to create 256 different colors for 00..FF?
        deleted 6mo ago

kokakiwi 6mo ago

ImHex (https://imhex.werwolv.net/ https://imhex.werwolv.net/) is also a really nice Hex editor with tons of plugins (patterns, file support, etc.) and even an embedded language for adding more patterns easily

aa-jv 6mo ago
+1 for ImHex - I've had a lot of fun with Pattern Language lately .. its quite fruitful to, for instance, have an AI take a few headers worth of input to produce some PL code which highlights all the things very well. For a while I almost forget I'm in a hex editor and not in some custom data wrangling environment ..

deleted 6mo ago

[deleted]

ralferoo 6mo ago

I actually stopped reading after the intro because I fundamentally disagreed with its premise. The "find the C0" took me about 1/4 second with uncoloured. Looking at the coloured took my eyes about 3 seconds to recover from the colour overload, then I was scanning down and found the colours so distracting with the constant switching between orange, pink and yellows than it took me a total of about 5 seconds to scan down as far as the blue C0. Maybe if it was all uncoloured and blue just for that, I might have actually noticed it looking different earlier. It's been a while since I used hexedit on Linux, but I think that highlighted search results in reverse colours, just like less does for text search. Personally, I'd prefer that to colours.

assimpleaspossi 6mo ago
Agree. This is what indentation and brackets, etc. are for. So much color makes it a clown car image to wade through.
- teddyh 6mo ago
  The old disparaging term was “angry fruit salad”: <http://www.catb.org/~esr/jargon/html/A/angry-fruit-salad.html http://www.catb.org/~esr/jargon/html/A/angry-fruit-salad.htm...>

red_admiral 6mo ago

My hex editor should let me turn syntax highlighting on and off; follow my personal color theme (and not produce light gray on white in the terminal); and let me highlight specific things I'm searching for like OD OA or FF FE.

roelschroeven 6mo ago

When you're going to color-code bytes in a hex dump, I would expect each ASCII character in the right column to have the same color as the hex byte in the left column, making it easier to pair them. I wonder why that wasn't done here.

pragma_x 6mo ago
I came in here to comment the same. Our brains are wonderful pattern recognition engines and the reader would absolutely be able to more readily see the correlation between hex and character representations this way. It might even accelerate learning hex values in the process.
simonomi 6mo ago
i tried it out, but ended up preferring fewer categories. the colors already exist in the main panel and i still find non-graphic ASCII/graphic ASCII/non-ASCII to be useful categories to have

nticompass 6mo ago

I used to use wxHexEditor and that had a feature where I could select a section of the file and highlight it in a color. When I was working to decode a certain file format, I used that to color-code different sections of the file and it was super useful. Those color-codes were stored in a separate file so you could load them back in.

ChrisRR 6mo ago

What a bad way to illustrate your point by using such similar looking pastel colours

altairprime 6mo ago
Ironically, my complaint about the article was that the author apparently only uses typical human vision ranges here, rather than mapping 00..FFh onto an OKlab gradient of hue 0..359° that rewards those few of us with impeccable color fidelity with even better highlighting than most can see :) No doubt there’s value in contentful highlighting but I’d rather just have a straight hue translation on the circle at a fixed luminosity. There’s only 256 hues to discern, after all! And what a pleasure it would be to learn to read hex code by hue alone :D
- simonomi 6mo ago
  for the record, i did use OKlab to pick my colors ! (see the CSS for my site https://github.com/simonomi/simonomi.github.io/blob/aa55ea855c17d5253d85d0440f41871aadc27b83/_includes/code.css#L51-L68 https://github.com/simonomi/simonomi.github.io/blob/aa55ea85...) i'm not opposed to using a unique color per byte, but i didn't want 256 different CSS classes for byte coloring, so i manually picked the 16 most distinct hues i could
  - altairprime 6mo ago
    You did great! I could tell the colors were equidistant from how they resembled Penumbra’s schemes. I just have really abnormal vision and have spent several years reading raw hex codes for a project. So I crave more color, heh :D

PunchyHamster 6mo ago

I wonder how hard it would be to color code repeating sequences

orphea 6mo ago

I get the idea but those specific examples are awful - not enough contrast.

Someone 6mo ago

The first example is “go ahead, try to find the single C0 in these bytes” and then argues one should highlight C0 bytes. If that’s true, how does the tool know I will be looking for C0 bytes and not for 03, D3, etc? The logical conclusion of that would be that the hex editor should uniquely color code every byte. And following the other examples even that’s not enough. The proposed solution is to create groups of byte values that each get their unique color. I think that helps, but we can do better: add a search feature. That tells your editor what you are looking for. Once you enter a search string, it can highlight all hits. Yes, “colorful output in a hexdump is useful for the same reason that syntax highlighting for code is useful”, but do you know what syntax highlighting needs? Knowledge of the expected content of a file. Without that, a hex editor at best can guess at how to color-code stuff. IMO, if you want to add syntax coloring to a hex editor, give it pluggable syntax coloring and heuristics for deciding which one to use when. While at it, also let those plugins control where to break lines, whether to show hex at all (why show it at all if a file has a few paragraphs of English text or an array of IEEE doubles?), etc. Those plug-ins will make errors and sometimes, users will want to see all byte values, so you’ll need a way for the user to override them.

Antibabelic 6mo ago
I don't think that's quite the point that example was intended to illustrate. The idea is not the you're looking for C0 bytes or any other kinds of bytes in particular, but rather that it's easier to fish out unique and interesting information in a sea of noise when you have color-coded bytes: like the fact that there's a conspicuous lonely C0 or some other value or series of values that stand out.
- bombcar 6mo ago
  Exactly - though the writing style buries the lede until the end; the recommendation is to have at least a different color for each of the first nibbles (e.g, 0x is one color, 1x another). As a minimum that makes some outliers pop out. He then has other recommendations like highlighting the ASCII range, etc. As a note, the some up there is load-bearing - color may lull you into complacency where the difference between 01 and 0F is major and important but not highlighted. More complicated regex built color tools designed to highlight "anomalies" could be developed but then you need to define what anomalies are (patters, places where a pattern changes, etc).

fleebee 6mo ago

> having more colors makes it possible to recognize more complex patterns The implicit cost here is that the simple patterns become harder to recognize when every byte is only subtly differently colored. Rather than give everything a different color, I'd rather have the important stuff highlighted. In the comparisons given, I think hexyl's highlighting scheme is significantly more useful.

whizzter 6mo ago

Anyone tried using Kaitai descriptions? It seems like a fairly flexible system that would be an excellent starting point for a hex-editor that wants to add good higher level coloring (and perhaps even editing).

randusername 6mo ago

I think this is a cool idea. I'd want to take it further by using full RGB and cycling through some colormaps with different properties. Sequential, diverging, cyclic like in matplotlib. https://matplotlib.org/stable/users/explain/colors/colormaps.html https://matplotlib.org/stable/users/explain/colors/colormaps... Can't think of a specific use-case off the top of my head, but sometimes I just want the "feel" of the data when I'm plotting something, and maybe the same scattershot approach would pay off at some point on unknown hex data if it was an option.

myfonj 6mo ago

When (rarely) using hex editors, one thing constantly comes to my mind: isn't base 16 arabic-roman numerals a bit awkward for "skimmable" overview? Color-coding indeed helps immensely there, but wouldn't simply letting bits and bops shine in eight bit clusters, resembling the "physical" shape of the eight-bit byte, be somewhat more readable? We even have characters in the Unicode for representing 0..255 variations, actually two distinct groups: Braille (arguably a bit misuse for binary) and octants (accompanied by older predecessors). So what would be |65|97|66|98|67|99|32|126|32|72|101|108|108|111|44|32|109|111|109|33|32|240|159|166|132| in base-10 or |41|61|42|62|43|63|20|7e|20|48|65|6c|6c|6f|2c|20|6d|6f|6d|21|20|f0|9f|a6|84| in base-16, could be |⢈|⢊|⡈|⡊|⣈|⣊|⠂|⡾|⠂|⠌|⢪|⠮|⠮|⣮|⠦|⠂|⢮|⣮|⢮|⢂|⠂|⠛|⣵|⡣|⠡| in Braille, or |𜵲|𜵶|𜴷|𜴻|𜶭|𜶱|𜴀|𜵯|𜴀|𜴋|𜶔|𜴭|𜴭|𜷟|𜴫|𜴀|𜶢|𜷟|𜶢|𜵴|𜴀|(⁕)|𜷢|𜵖|𜴙| using octants. Most significant bit is at the top left here, the least one is bottom right -- it felt somewhat intuitive to me this way, your intuition may differ, obviously. Or, naturally, "AaBbCc ~ Hello, mom! <Unicorn Emoji>" as a "UTF-8" text. Try: http://myfonj.github.io/tst/byte-dec-hex-braille-octant.html http://myfonj.github.io/tst/byte-dec-hex-braille-octant.html) Test (with added "CSS" variant and "highlight" of empty dots): http://myfonj.github.io/tst/byte-visualisation-exploration.html http://myfonj.github.io/tst/byte-visualisation-exploration.h... (⁕) HN apparently eats upper-half block. Amusing that only this particular ("old", as referred earlier) one got filtered out… Also caveat: Android phones have messed-up Braille block due outdated broken embedded font, so all patterns with dots in the left half appear in the right instead. Long reported, not fixed, IIRC.

aa-jv 6mo ago
This is a great idea and I concur with your line of thought - there is room for expression in the realm of number representations .. have you considered building an ImHex plugin that would illustrate your point? The use of octants is particularly intriguing .. One thing I often ponder on, along similar lines, is whether I can write some clever plugin that would put FF ChartWell - a font which uses ligatures to render useful graphs out of boring numerical data - into use, within ImHex. Seen how ligatures can be used this way? https://typographica.org/typeface-reviews/chartwell/ https://typographica.org/typeface-reviews/chartwell/ Your idea of discovering new means of representing numeral data put me in mind of ligature hacking, in any case ...

bwiggs 6mo ago

DEFCON30, Mayhem CTF. We were given a file full of random bytes. The flag was in there somewhere. It was too random to be encrypted, there wasn't any structure. `file` didn't return anything, truly just a bag of bytes. I had decided to install `hexyl` as an alternative option to some of the other hex editors installed o my linux machine. All the bytes were colored grey. I scrolled the file and noticed a blip of yellow. A random golden `{` amongst all the noise. Weird. The next colored byte was a `C`, then `T`, `F`. --- At that time, I was mostly using HexFiend to look at raw files, which didn't have byte coloring. For DEFCON I had decided to drive my linux machine. I had ghex installed, but i had also decided to install and try `hexyl` via cli. So seeing bytes in color was purely by chance that I had installed it. I eventually posted an issue to ghex to add color support. https://gitlab.gnome.org/GNOME/ghex/-/issues/60 https://gitlab.gnome.org/GNOME/ghex/-/issues/60 I need to see if I can find the file and post it on that blog post. https://bwiggs.com/posts/2023-08-31-hacking-in-color/ https://bwiggs.com/posts/2023-08-31-hacking-in-color/

abcd_f 6mo ago
> It was too random to be encrypted That's a rather odd remark.
- bwiggs 6mo ago
  You would still expect some amount of file structure, ex byte headers or something at the beginning/end of file. no?
  - Crestwave 6mo ago
    That would be expected for encodings or container file formats. Straight-up encryption like AES produces results that are visually indistinguishable from random data.
  - throwawaysoxjje 6mo ago
    I’d expect the greater length of the encrypted data (which should look random) vs the structured header/footer to rapidly push the Shannon entropy to the maximum

mplanchard 6mo ago

Anyone know of a good emacs package for this?

smithza 6mo ago
A gist I (claude) threw together that maps this color scheme on hexl-mode. https://gist.github.com/zacharyasmith/bcfd91c9685c6cec30385a687a92477b https://gist.github.com/zacharyasmith/bcfd91c9685c6cec30385a...

soegaard 6mo ago

This was a great article and inspired me to add support for binary files in `peek`. https://soegaard.github.io/peek/#%28part._binary-files%29 https://soegaard.github.io/peek/#%28part._binary-files%29 For me the key insight is that similar values should get similar colors. And since Fx and 0x are "similar" the color palette should be cyclic.

sidewndr46 6mo ago

compare a simple high contrast display to one of that makes it difficult to read and hurts my eyes? Sure! absolutely! I'll pass thank you

MisterTea 6mo ago

> compare that to one with colors: The colors make it worse as I'm red-green colorblind. Looking at that mess is eye strain. Honestly I mostly prefer syntax highlighting turned off as it causes eye strain. I have found the black on light yellow theme of the Acme editor to be a very comfortable monochrome color scheme.

stronglikedan 6mo ago

> go ahead, try to find the single C0 in these bytes: Ctrl+C, Ctrl+F, Ctrl_V... Easy!

oblio 6mo ago
That part isn't literal. It's meant to be: "find the outlier in this sea of hex-coded bytes".

taeric 6mo ago

I find it funny that I found the "single C0" pretty much instantly. I grant that the post largely has a point, mind you. But scanning for a needle in a haystack is something that you just don't often do? I am, of course, now very curious how often folks are using hex editors. And itching for an excuse to open a file that way. :D

traderj0e 6mo ago
The first time I used one was in high school when I was playing some video game in an emulator and wanted to convert the game save to another emulator's format, cause halfway through I realized the first emu had issues with the game. There was no conversion software, but with the hex editor I figured out that you needed to remove some header and change the endianness.

shmerl 6mo ago

Any good neovim plugin recommendation for hex editing?

contingencies 6mo ago
I came to scan for the same thing, basically. Is there a vi(m) hex editor? Hexmode https://github.com/fidian/hexmode https://github.com/fidian/hexmode or vim-deadc0de https://www.vim.org/scripts/script.php?script_id=6033 https://www.vim.org/scripts/script.php?script_id=6033 With decent color support? No.
- traderj0e 6mo ago
  I was kinda expecting vim with whatever common plugins I installed to do this already, but no. It does nice highlighting for all sorts of text formats, even non-code things like CSV, but not hex.

0xfalafel 6mo ago

Shameless plug: I have written an hexadecmial editor (in Rust) that colors bytes. It looks similar to hexyl. https://github.com/0xfalafel/hextazy https://github.com/0xfalafel/hextazy Some nice features are: - robust undo - insertions (not just overwriting) - inspect the value of selected bytes - search I have also recently discover this other project that is pretty cool: https://github.com/mentebinaria/dz6 https://github.com/mentebinaria/dz6

wang_li 6mo ago

Color coding is a simplistic, half-assed way of doing semantic analysis. Go full assed and make an /etc/magic aware file decoder that decodes and highlights anomalies.

dhosek 6mo ago

I found the coloring in most of the examples to be more distracting than helpful. I can see cases where it could be helpful (e.g., highlighting bytes in the 0x20–0x7E range for spotting ASCII strings, or a fancier one that can identify UTF-8 strings, or better still, invalid sequences in what might otherwise be UTF-8), but most of the cases here didn’t really help all that much for me.

TheRealPomax 6mo ago

It'd be nicer if the hex coloring actually matched the "ascii" coloring as well. Orange on the left but green on the right does not help find things.

nickwanninger 6mo ago

I added type-based color printing to my hexdump in my kernel [1] if anyone wants to have that code. It was instrumental in finding bugs quickly in wee hours of the night sometimes, especially if you have heap corruption. ---- [1] https://github.com/ChariotOS/chariot/blob/e046849c668458d25eee4268776eb2ee85d88a62/kernel/lib/util.cpp#L33-L100 https://github.com/ChariotOS/chariot/blob/e046849c668458d25e...

xvilka 6mo ago

Rizin[1][2] does exactly this, also there is a compact hex-II[3] mode. [1] https://rizin.re https://rizin.re [2] https://github.com/rizinorg/rizin https://github.com/rizinorg/rizin [3] https://speakerdeck.com/ange/no-more-dumb-hex https://speakerdeck.com/ange/no-more-dumb-hex

jcalvinowens 6mo ago

If you just want to see patterns, and don't actually need to see the values, you can go a step further and simply visualize the data as a bitmap, e.g. dd if=/dev/urandom bs=$[256*256] count=1 | display -size 256x256 -depth 8 GRAY:- You can do the same thing with audio, which makes different sorts of patterns obvious, e.g. dd if=/dev/urandom bs=$[256*256] count=1 | aplay -c 1

felooboolooomba 6mo ago
> dd if=/dev/urandom bs=$[256*256] count=1 | aplay -c 1 This plays sounds, sounds like "Liberate Mae?"
EvanAnderson 6mo ago
I made little CLI tool back in the MS-DOS days to dump binaries into VGA mode 0x13. It allowed me to vary the width of "line" wrapping. It was a killer tool for seeing data in binaries.

ape4 6mo ago

As a next level beyond coloring... how about adding some interactivity? How about a slider to control the brightness of each type of byte. Turn everything but text to 10% when when you're looking for some words in a binary file.

Findecanor 6mo ago

If you're making a hex editor and going to have colour coding, I'd think you expend some effort to make the colouring schemes configurable — and easy to configure and change. Maybe load and save as separate files. Different colouring schemes for different types of data.

leetrout 6mo ago

ipython saved me this week when it color coded b'\x100' Which was not obvious to me in the print output that it is \x10 and a literal 0.

vesui 6mo ago

Fantastic writeup, and it's so cool to see you at the front of HN! This article compels me to add colored syntax highlighting for hex code blocks on my own blog...

amelius 6mo ago

Or maybe use an LLM to auto-format your hex dumps based on structure?

Xophmeister 6mo ago

If you have a Kaitai spec for the format, then its IDE[0] will do structural highlighting for you. This is web-based, but it doesn’t seem like too much of a stretch to implement a TTY version. [1]: ide.kaitai.io

skalidindi3 6mo ago

https://github.com/skalidindi3/cxxd https://github.com/skalidindi3/cxxd I wrote this a LONG while back. At the time, I was fresh out of college and my first job included a lot of reverse engineering communication protocols to make machines work together for automation purposes. I will personally testify to how useful it was to see visual patterns to aid in this. The single biggest benefit was seeing that one particular protocol switched endianness WITHIN a specific packet.

deepsun 6mo ago

Rob Pike says colors add cognitive load. He calls syntax highlighting "juvenile".

fc417fc802 6mo ago
I think that depends on quantity. There are certainly some juvenile examples that take things too far.

eviks 6mo ago

And the lon leading zero should be a space instead, will cut down on visual noise in the examples in the post a lot to let your color codes do the rest

deleted 6mo ago

[deleted]

barbs 6mo ago

Reminds me of this blog post where someone was trying to reverse-engineer a hand-rolled encryption protocol in a crossword app by colorizing the output and looking for patterns. https://www.muppetlabs.com/~breadbox/txt/acre.html https://www.muppetlabs.com/~breadbox/txt/acre.html

account42 6mo ago

Those examples look horrible. Rainbow barf with low contrast is not an improvement to anything.