6 ms·
Ts_zip: Text Compression Using Large Language Models
- Twirrim 2y ago"(and hopefully decompress)" is a horrifying descriptor.
- perching_aix 2y agoThey're clearly just poking fun at it.
- hansvm 2y agoIt adds levity to the article and also introduces the reader to the sorts of things that can go wrong if they try it at home. The last paragraph highlights how they fixed one of the main pitfalls I normally see in this sort of thing, where floating-point operations are mangled in myriad ways in the name of efficiency (almost always correct for physics or whatever, but a single bit being incorrect will occasionally mangle this compression scheme). Mind you, actually doing what they claimed in that last paragraph is usually painful. The easiest approaches re-implement floating-point operations in software using integer instructions, and the complexity increases from there.
- orbital-decay 2y agoNot just efficiency, if you have e.g. floating point values arriving asynchronously to be accumulated, you'll always have a slightly unpredictable result. Fun fact: Gemini 2.0 Flash is 100% deterministic with temp 0, unlike most models. This must be related to TPUs somehow, not sure why all previous Gemini versions are not like that, though.
- rahimnathwani 2y agoPrior discussion: https://news.ycombinator.com/item?id=37152978 https://news.ycombinator.com/item?id=37152978
- jodrellblank 2y agoLooks like it’s been updated since then; commenters in that thread are saying the decompressor needs to run on the same hardware as the compressor; now the link says: > “The model is evaluated in a deterministic and reproducible way. Hence the result does not depend on the exact GPU or CPU model nor on the number of configured threads. This key point ensures that a compressed file can be decompressed using a different hardware or software configuration.”
- vessenes 2y agoFabrice has recently extended this work into audio encoding, an area which to me seems more useful than shaving a bit more off wik8 compression rates. Demo and code? Available at bellard.org as well.
- zamadatix 2y agoLink for the curious https://bellard.org/tsac/ https://bellard.org/tsac/ Has anyone done the work of comparing this to other similar extreme audio compression solutions?
- deleted 2y ago[deleted]
- droideqa 2y agoI have always thought compression to be an analog to intelligence. The smarter you are, the better at summarization you are.
- mikevin 2y agoI'm curious what the compressed text looks like. Anyone have an example?
- munch117 2y agoBinary goo, barely distinguishable from random data, if at all. The arithmetic coder will make sure of that. It's the nature of compression: Any discernible pattern could have been exploited for further compression.
- Lerc 2y agoIf it is within cooee of state of the art the compressed text should look like a pile of random bits. If it looks like anything at all other than randomness then you can describe whatever it is that it looks like to get more compression.
- 0x0 2y agoThis is particularly interesting as there seems to be, for decades, a general consensus that the problem of text compression is the same as the problem of artificial intelligence, for example https://en.wikipedia.org/wiki/Hutter_Prize https://en.wikipedia.org/wiki/Hutter_Prize
- WhitneyLand 2y agoI’m not sure this is strictly true. It seems more accurate to say there are deep connections between the two rather than they are theoretically equivalent problems. His work is really cool though no doubt.
- retrac 2y agoThere's a general consensus that entropy is deeply spooky. It pops up in physics in black holes and the heat death of the universe. The physicist Erwin Schrodinger suggested that life itself consumes negative entropy, and others have proposed other definitions of life that are entropic. Some definitions of intelligence also centre on entropy. What to make of all that however, has anything but consensus.
- Y_Y 2y ago[flagged]
- retrac 2y agoI was trying to convey a subjective and emotional experience. Obviously I failed. I hope that when you try to express awe it isn't dismissed as weasel words. I give up. Delete my account please dang. This site isn't good for my mental health.
- Y_Y 2y agoI didn't like your comment, that's all. I'm just one anonymous asshole, I can't invalidate your sense of awe. FWIW, I didn't want or expect to harm your mental health.
- 0-_-0 2y ago1 MBps is insanely fast for a method like this, it must be in the 100k tokens per second range. Probably with large batches.
- justmarc 2y agoThis man is an absolute wizard, and a legend who hasn't stopped since the fantastic LZEXE days.
- remram 2y agoIf I read this correctly, the largest test reported on this page is the "enwik9" dataset, which compresses to 213 MB with xz and only 135 MB with this method, a 78 MB difference... using a model that is 340 MB (and was probably trained on the test data). No one would be impressed with saving 78 MB on compression using a 340 MB dictionary so I am not sure why this is good? Please let me know if I misunderstand.
- binary132 2y agoIf you’re compressing 100 or 100k such datasets, presuming that it is not custom tuned for this corpus, then wouldn’t you still save much more than you spend?
- remram 2y agoI'm not saying the result is completely useless, I am comparing it to the age-old technique of using a dictionary. Does this new LLM-powered technique improve upon the old dictionary technique? Dictionaries also don't require a GPU or this amount of RAM. Where I assume LLMs would shine is lossy compression.
- binary132 2y agoAh ok, I think we made different assumptions about whether the model was specific to the particular dataset so each one would need a new model — a dictionary is specific to the particular dataset being compressed, right? I was thinking the LLM would be a general-purpose text compression model.
- remram 2y agoNot particularly. You could make a dictionary from "the English web", with common character sequences found on those sites you use as input.
- ksec 2y agoI have the same question, what is the different between LLM and Dictionary in the context of compression. Can I not "train" a dictionary?
- j_juggernaut 2y agoMade a quick and dirt streamlit app to play around encrypt decrypt https://llmencryptdecrypt-euyfofcjh8bf2utuha2zox.streamlit.app/ https://llmencryptdecrypt-euyfofcjh8bf2utuha2zox.streamlit.a...
- meindnoch 2y agoIt is very good at decrypting the string "Error".
- deleted 2y ago[deleted]
- zamadatix 2y agoAlso worth checking out some of the author's other compressors e.g. another one of their neural network solutions using a transformer https://bellard.org/nncp/ https://bellard.org/nncp/ holds the top spot in the Large Text Compression Benchmark. It's ~3 orders of magnitude slower though.
- bhouston 2y agoI believe almost all LLMs are trained using wikpedia these days. So compressing wikipedia well without including the size of the LLM in the compression result is a bit of a cheat. I guess one would argue it is a universal dataset representing understanding the English language and real-world relationships at this point but it is still a bit of a cheat.
- atiedebee 2y agoThere's a reason compression benchmarks often times include the size of the executable when benchmarking compression ratios. Although Matt Mahoney's large text compression benchmark[0] does currently have a transformer model at number 1. [0] http://www.mattmahoney.net/dc/text.html http://www.mattmahoney.net/dc/text.html
- cat5e 2y agoHas this been attempted for raw binary? Using an NN to predict the most likely next binary string?
- deleted 2y ago[deleted]
- jll29 2y agoSpeed and compression are are one thing, but I wonder how much energy Ts_zip consumes compared to gzip?