9 ms·
What Is Huffman Coding?
- ggghhhfff 6y agoI am curious as to how this works for filetypes other than text files- what are the contents of each node in the tree for, say, a PNG file?
- kumarvvr 6y agoOne way could be to split the image file into RGB or CMYK channels, and then compress the relative brightness of pixels per channel.
- st_goliath 6y agoIt works the *exact* same way. You process the input one byte at a time, build a histogram, construct the Huffman Tree and encode the input. Why should it work different? Text files are just regular files where the bytes only use a specific subset of the possible values they could have. If you do have some knowledge about the data you are encoding, you can be a little bit smarter about it: e.g. for the text section of an executable, you might work on individual instructions instead of bytes, maybe use common, prepared Huffman Trees, so you don't have to encode the tree itself. On a side note: IIRC the Intel Management Engine does that using a proprietary Huffman Tree, backed into the hardware itself, as an obfuscation technique[1]. To circle back to your question: PNG simply feeds the pixel data into the zlib deflate() function as-is. [1] https://en.wikipedia.org/wiki/Intel_Management_Engine#Design https://en.wikipedia.org/wiki/Intel_Management_Engine#Design
- Dylan16807 6y agoYou ask why it should work differently but then give a good reason why it should work differently: sometimes splitting by bytes is not the best unit. And it actually does often work differently for PNG! PNG has a handful of preprocessing options for the pixels. So in filter mode 2, for example, deflate is encoding the difference between each pixel and the pixel above it. More or less.
- twic 6y agoBut you still apply Huffman coding to the bytes after filtering. I think this is true even when using 16 bits per sample, but i'm not sure. From what i remember, in gzip, Huffman coding is applied to the offsets of repeated blocks, which are not single bytes.
- cdrini 6y agoThe main key is that you can look at any file as a string of bits! And apply Huffman coding at whatever granularity you like. For text files, you're essentially applying it at the byte level (since each character is a byte (in ASCII, anyways)). For images, you might have one byte per colour channel, RGB. Then you can apply Huffman coding at the byte level, or even at the 3 byte level to operate on entire colours as opposed to channels.
- andreareina 6y agoHuffman coding generally works on symbols, where a symbol can be represented by any (possibly variable-length!) string of bits. I recall reading about a compression scheme (lzw?) where the symbol table had an entry for each raw byte value, plus the encoding instructions (e.g. lookback x bytes for y bytes).
- Jasper_ 6y agoThat's pretty much all of them except LZW (which is an LZ78-alike, rather than LZ77). gzip has a number of Huffman tables for literals, distance, and for building Huffman tables at runtime... Yes, the instructions for building the Huffman tables used for decompression are themselves compressed using... more Huffman tables (HCLEN)! Anything to save a few bits...
- andreareina 6y agoOh yeah, might have been remembering gzip. All I remember is that it was one of the old/common ones. Don't know why my mind jumped to lzw instead of the obvious gzip haha.
- Rendello 6y agoIt's generally used on bytes, but as others have said, it can be any symbol. Even on complete words! https://www.nayuki.io/page/huffman-coding-english-words https://www.nayuki.io/page/huffman-coding-english-words
- user-the-name 6y agoIt's the opposite: It's almost never used on bytes, because that just doesn't give you a lot of compression. It is generally used as the final stage of some other compression algorithm, and operates on symbols generated by that algorithm. Often, this is some variation on LZ77, and the symbols are something like "bytes 0-255" in addition to various symbols that denote a match in previous data of some length and at some offset.
- nayuki 6y agoIndeed, this is an example where each English word in a book gets a unique symbol for the purposes of Huffman coding. Note that the Huffman output is in base 52 (abc...xyzABC...XYZ) alphabet instead of the usual binary.
- user-the-name 6y agoHuffman coding is not used directly by PNG. PNG instead uses the regular DEFLATE algorithm from Zip (and Gzip), which uses Huffman coding as its last stage, to output the symbols created by its LZ77-based compression algorithm.
- magicalhippo 6y agoJPEG uses Huffman as the compression stage. Here's[1] a brief but nice overview, here's[2] a more in-depth one. Some encoders just use a precomputed Huffman table, but you can make an optimized one as discussed here[3]. Huffman is not the most optimal compression that can be used, for example Dropbox's Lepton[4] saves an additional 20% by replacing the Huffman stage with something better. However, since it's a lossless stage this can be done transparently, which is nice. [1]: http://www.robertstocker.co.uk/jpeg/jpeg_new_11.htm http://www.robertstocker.co.uk/jpeg/jpeg_new_11.htm [2]: https://www.impulseadventure.com/photo/jpeg-huffman-coding.html https://www.impulseadventure.com/photo/jpeg-huffman-coding.h... [3]: https://www.impulseadventure.com/photo/optimized-jpeg.html https://www.impulseadventure.com/photo/optimized-jpeg.html [4]: https://dropbox.tech/infrastructure/lepton-image-compression-saving-22-losslessly-from-images-at-15mbs https://dropbox.tech/infrastructure/lepton-image-compression...
- jmspring 6y agoHad classes from David Huffman while at UC Santa Cruz. One of my favorite professors and he did not like when people constantly brought up Huffman coding. Spent several hours talking various topics, but one of his favorite areas of exploration when I was around campus was paper folding. A couple of links with examples: - https://www.cise.ufl.edu/~manuel/huffman/index.html https://www.cise.ufl.edu/~manuel/huffman/index.html - https://collections.mitmuseum.org/collection/david-a-huffman-collection/ https://collections.mitmuseum.org/collection/david-a-huffman...
- akamoonknight 6y agoDo you know how he makes such precise curved folds? I'm no expert, but love me some origami and I can't believe that I'd be able to get folds to turn out like that. Anything I would attempt would invariably have little sub-creases instead of a smooth fold.
- bitslayer 6y agoNormally you would use a creaser, which is like a dull knife that is pressed down into the paper, perhaps over a slightly giving surface.
- unwind 6y agoImages don't load from the second link for me. Sad museum? :/ Chrome on Android btw.
- kuu 6y agofrom Firefox PC works fine :)
- rndgermandude 6y agoMy Firefox says for media-internal.mitmuseum.org where the images are hosted on The certificate is only valid for the following names: *.s3.amazonaws.com, s3.amazonaws.com Let's hope you happened to get served a valid cert, and not that the cert validation is disabled or broken in your Firefox.
- cdrini 6y agoWonderful summary and very well explained! Those graphics could be in a CS textbook :) A bit more (or perhaps another article) on how the tree can be encoded would be useful, since the total size of your compressed example would need to include the size of the tree. Also: JPEGs also have a Huffman tables region in their binary; haven't dug too deeply, but seems like they also use Huffman coding!
- terrelln 6y agoI recently reworked how zstd builds its Huffman decoding tables (not decoding itself) to avoid unpredictable branches and speed the table building up by about ~2x [0]. This is insignificant for large decompressions, but if you're decompressing only a few KB, the table building time can dominate the actual decompression. It sort of goes to show that while Huffman codes have been around for ages, implementations can still improve, especially as the hardware we use changes. [0] https://github.com/facebook/zstd/pull/2271 https://github.com/facebook/zstd/pull/2271
- utopcell 6y agoNice.
- soheil 6y agoIt's a compression technique used in stuff like PNG and gzip, saved you a click.
- Jasper_ 6y agoGah! Another explanation that uses "Huffman Trees"! Nobody uses that, we all use Canonical Huffman, where all you have are the number of symbols per code length, letting you efficiently build tables. Yes, tables are used instead of trees. The trees are a distraction. https://en.wikipedia.org/wiki/Canonical_Huffman_code https://en.wikipedia.org/wiki/Canonical_Huffman_code
- Rendello 6y agoI'm glad the resources on the basic trees exist, since that's what got me interested. But I would love to see more resources on canonical Huffman and especially the package-merge algorithm! The former was confusing at first but mind-blowing when it clicked for me, and the latter looks awesome but the implementation is incomprehensible to me —I guess I'm stuck allocating trees until I can figure it out!
- Jasper_ 6y agoCanonical Huffman can be thought of as a pre-determined tree shape that's easy to re-construct. First, decide your bit lengths from your probabilities (something the most probable should get the shortest code-length, choose wisely), and then add 1s to the start of each longer to make it unambiguous. If we have three symbols, A, B, and C, and let's say we assign bit lengths of A=1, B=2, C=2, meaning A is the most probable then we count: A = 0 B = 10 C = 11 For code lengths A=1, B=2, C=4, D=4, E=4, then we have: A = 0 B = 10 C = 1100 D = 1101 E = 1110 Note that all we need to send is the bit lengths (1, 2, 4, 4, 4) to the other side, and we have an algorithm to assign the bits. Though, even (1, 2, 4, 4, 4) is actually too much information, we just need to send the number of symbols for each given length: (1, 1, 0, 3) Much faster, smaller, and generally better than sending over a whole tree.
- Rendello 6y agoThanks! I've got the canonical stuff down pat already in my project, but I am still allocating the tree to get the bit-lengths in the first place. Once I have the package-merge algorithm up and running, the code will never allocate a tree at all for construction, plus I'll have the added benefit of length-limited codes.
- Rendello 6y agoMy current project is a Huffman coder built in Zig, it's been a lot of fun to build. The Huffman tree generation is simple, but there's a lot of different nuances and variations to account for. For example, a "canonical Huffman code" can save you space encoding the table through some clever trickery. In short, you can encode the table by counting the number of bytes that use each bit-count, and the bytes used in the file. You don't need to store the patterns at all, since in the canonical algorithm the patterns are regenerated from the bit-lengths. [1] Right now I'm trying to implement the package-merge algorithm,[2] which will allow me to create the table without building an inefficient fully-allocated tree, and more importantly will allow me to limit the lengths of my codes (the maximum code length is n-1, where `n` is the length of your alphabet. Working with bytes and using 255 bit codes is obnoxious). Unfortunately all explanations of the algorithm I've found are very academic and mathematical, so I'm having trouble working it out. Some of you might be interested in the short video Tom Scott made about Huffman coding.[3] 1. https://en.wikipedia.org/wiki/Canonical_Huffman_code#Encoding_the_codebook https://en.wikipedia.org/wiki/Canonical_Huffman_code#Encodin... 2. https://en.wikipedia.org/wiki/Package-merge_algorithm https://en.wikipedia.org/wiki/Package-merge_algorithm 3. https://www.youtube.com/watch?v=JsTptu56GM8 https://www.youtube.com/watch?v=JsTptu56GM8
- NieDzejkob 6y agoAren't 255 bit codes the optimal choice for an input distribution skewed enough?
- nayuki 6y agoYes, but you have to consider how to describe code lengths from 1 to 255. For example, DEFLATE only allows Huffman codes to be 0 to 15 bits long, which simplifies the number of possibilities that the table encoder needs to handle. http://www.zlib.org/rfc-deflate.html#dyn http://www.zlib.org/rfc-deflate.html#dyn
- tcgv 6y agoShameless plug: A couple of years ago I wrote an implementation of the huffman coding algorithm as well while studying it, along with unit tests. I was also interested in practicing OOP. You can find the result in the link below. - https://github.com/TCGV/HuffmanCoding https://github.com/TCGV/HuffmanCoding
- huzaif 6y agoGood stuff. Thanks for sharing.
- nooyurrsdey 6y agoLearning about Huffman Encoding in school was my first exposure to these sort of algorithms. I was an electrical engineering major and had little exposure to computer science at the time. It captivated my interest immediately - it was such a simple and effective approach, and it demystified how compression algorithms worked. I really like this overview. It's not meant to be a full discourse, more just an intro for newcomers. And I think it gets the basic idea across very effectively.
- doc_gunthrop 6y agoTo grok a concept it helps to actually do it. You can take on a coding challenge for Huffman Encoding at Codewars: https://www.codewars.com/kata/54cf7f926b85dcc4e2000d9d https://www.codewars.com/kata/54cf7f926b85dcc4e2000d9d
- swframe2 6y agoAs long as we're on this topic, might as well pivot to information theory. https://www.youtube.com/playlist?list=PLruBu5BI5n4aFpG32iMbdWoRVAA-Vcso6 https://www.youtube.com/playlist?list=PLruBu5BI5n4aFpG32iMbd...
- zmodem 6y agoThanks for the link! I started watching and this looks like an interesting lecture series. (The book looks interesting too.)
- scoloo 6y agogo code style is ugly.
- algorithm314 6y agoOne of the fastest Huffman compression libraries is FPC https://github.com/algorithm314/FPC/ https://github.com/algorithm314/FPC/ It even has compression ratio better than some AC implementations. Better than ZSTD's
- gorgoiler 6y agoHuffman coding is the bomb with the kids. Any kind of encoding / cipher stuff is well received but the application of a binary tree makes it HC more cool. I love teaching this part of the syllabus so much.
- numlock86 6y ago> This is 29 bits instead of 96, with no data loss. Great! What's the reasoning of leaving out the tree structure needed for decoding in this argument?
- davethedevguy 6y agoIt's discussed later. It talks about the fact that both sides need to have the same tree, and ways to accomplish that.
- utopcell 6y agoSilly example. It compresses the phrase "do or do not" that has 6 symbols (d, o, r, n, t, space), builds a huffman tree just for these and then assumes significant compression by using 8 bits per character for the uncompressed case.
- bntyhntr 6y agoWhat's silly about that? If I were just getting started with this kind of thing, I think this would've been a great post for me. As I haven't done this since an intro class in college, it was actually a nice quick refresher. 6 symbols is easy to keep in the head all at once.
- SkyBelow 6y agoWouldn't any compression algorithm be a silly example when using such a small amount of data as the overheard to communicate information about the compression would take more data than was saved and potentially more data than the original message? I think it still suffices as an example because it would be easy to infer how this scales up to an entire book. Finer details are left out, but is that the sort of detail that should be present in a very short introductory article?
- utopcell 6y agoNot communicating enough info to rebuild the tree is a separate issue, but it is mentioned in the article so the reader is not left guessing. In the example however, we have a set of 6 symbols that would require 3 bits each to represent uncompressed but instead assume 8 bits. It invites fairness questions, which would distract a reader that has not been exposed to the concept before. This is an otherwise nice intro to the topic though.
- davethedevguy 6y agoI take your point. My intention was to pick an example that produced a small tree with only a few leaf nodes (so that the diagram was easy to follow), but still contained some duplication. My hope was that somebody new to the concept could then infer the results for larger inputs. I did not intend to imply that this would be a valid use case for building a Huffman tree in practice.
- benibela 6y agoI implemented Huffman Coding when I was 14 years old: http://benibela.de/sources_en.html#huffman.zip http://benibela.de/sources_en.html#huffman.zip Probably the first non-trivial data structure/algorithm I had implemented (previously I was making games, where you need no algorithm more complex than rectangle intersection)
- wyclif 6y agoAm I the only one who clicked on this thinking it was about Steve 'Spez' Huffman's current side project?