6 ms·
Using a LLM to compress text
- pronoiac 2y agoI expected general-purpose compressors to do better, but I was wrong. For the whole document: 174355 pg11.txt 60907 pg11.txt.gz-9 58590 pg11.txt.zstd-9 54164 pg11.txt.xz-9 25360 [from blog post]
- EgoIncarnate 2y agoWhich model did you use?
- Intralexical 2y agoIDK why, but BZIP2 seems to do somewhat better that other compression algorithms for natural language text: $ curl https://www.gutenberg.org/cache/epub/11/pg11.txt | bzip2 --best | wc 246 1183 48925 Also, ZSTD goes all the way up to `--ultra -22` plus `--long=31` (4GB window— Irrelevant here since the file fits in the default 8MB anyway).
- pastage 2y agoYou can use a preshared dictionary with bzip2 and zstd so you can get that down alot by using different dictionaries depending on certain rules. I dont know if it helps with literature but I had great success in sending databases with free text like that. In the end it was easier to just use one dictionary for everything and just skip the rules.
- immibis 2y agoAnd here's the opposite - using gzip as a (merely adequate) "large" language model: https://aclanthology.org/2023.findings-acl.426.pdf https://aclanthology.org/2023.findings-acl.426.pdf By the way, if you want to see how well gzip actually models language, take any gzipped file, flip a few bits, and unzip it. If it gives you a checksum error, ignore that. You might have to unzip in a streaming way so that it can't tell the checksum is wrong until it's already printed the wrong data that you want to see.
- anonu 2y agoShouldn't you factor in the size of the compression tools needed?
- Legend2440 2y agoLLMs blow away traditional compressors because they are very good predictors, and prediction and compression are the same operation. You can convert any predictor into a lossless compressor by feeding the output probabilities into an entropy coding algorithm. LLMs can get compression ratios as high as 95% (0.4 bits per character) on english text. https://arxiv.org/html/2404.09937v1 https://arxiv.org/html/2404.09937v1
- Der_Einzige 2y agoThe 1-1 correspondence between prediction and compression is one of the most counterintuitive and fascinating things in all of AI to me.
- mjburgess 2y agoIts only equivalent for a very narrow sense of 'prediction' , namely modelling conditional probability distributions over known data. There's no sense, for example, in which deriving a prediction about the nature of reality from a novel scientific theory is 'compression' eg., suppose we didn't know a planet existed, and we looked at orbital data. There's no sense in which compressing that data would indicate another planet existed. It's a great source of confusion that people think AI/ML systems are 'predicting' novel distributions of observations (science), vs., novel observations of the same distribution (statistics). It should be more obvious that the latter is just compression, since it's just taking a known distribution of data and replacing it with a derivative optimal value. Science predicts novel distributions based on theories, ie., it says the world is other than we previously supposed.
- palmtree3000 2y agoSure it is! If we were trying to compress an archive of orbital data, one way to do it would be "initial positions + periodic error correction". If you have the new planet, your errors will be smaller and can be represented in less space at the same precision.
- 2y ago
- Intralexical 2y agoThe practical way to do this is to train a custom dictionary with a normal compression algorithm.
- infamouscow 2y agoYes, but that also spends an order of magnitude less on money and energy, so it's not going to get you on HN. Neat parlor trick, though.
- clay_the_ripper 2y agoIt does seem like this is a possible method to test if an LLM has your data in it. People have found other ways to do that of course, but this is pretty clever.
- mvkel 2y agoNot necessarily. This also uncovers the weakness of the NYT lawsuit. Imagine in your corpus of training data is the following: - bloga.com: "I read in the NYT that 'it rains cats and dogs twice per year'" - blogb.com: "according to the NYT, 'cats and dogs level rain occurs 2 times per year." - newssite.com: "cats and dogs rain events happen twice per year, according to the New York Times" Now, you chat with an LLM trained on this data, asking it "how many times per year does it rain cats and dogs?" "According to the New York Times, it rains cats and dogs twice per year." NYT content was never in the training data, however it -is- mentioned a lot on various sources throughout commoncrawl-approved sources, therefore gets a higher probability association with next token. Zoom that out to full articles quoted throughout the web, and you get false positives.
- deleted 2y ago[deleted]
- refulgentis 2y agoThey were getting huge chunks, verbatim of NYT articles out. I remember being stunned. Then I remember finding out there was some sort of trick to it that made it seem sillier.
- KaoruAoiShiho 2y agoWas it that NYT articles are routinely pirated on reddit comments and the like?
- viraptor 2y agoDoes it matter? What's the legal view on "I downloaded some data which turns out to be copied from a copyrighted source and it was probably trivial to figure it out, then trained the LLM on it"? I mean, they work on data processing - of course they would expect that if someone responds with 10 paragraphs in reporting style, under a link to NYT... that's just the article.
- blackle 2y agoThere is a way to do this same compression by utilizing the raw probability distributions that the LLM produces as output. Fabrice Bellard has some experiments with doing this with transformers: https://bellard.org/nncp/ https://bellard.org/nncp/ The idea is that if you can produce an accurate probably distribution over the next bit/byte/token, then you can compress things with an entropy compressor (huffman encoding, range encoding, asymmetric numeral systems, etc). This comment is too small of a space to explain fully how they work, but it suffices to say that pretty much every good compression algorithm models probability distributions in some way.
- FooBarBizBazz 2y agoExactly this. The way to do this is to use the LLM as the statistical model inside an arithmetic coder. You use its next-token probabilities to encode the next token. The KL divergence between the LLM's probabilities, and the empirical probabilities in the corpus that you actually compress, is the compression inefficiency.
- saeranv 2y ago> The idea is that if you can produce an accurate probably distribution over the next bit/byte/token... But how can you get credible probability distributions from the LLMs? My understanding is that the outputs specifically can't be interpreted as a probability distribution, even though superficially they resemble a PMF, due to the way the softmax function tends to predict close to 100% for the predicted token. You can still get an ordered list of most probable tokens (which I think beam search exploits), but they specifically aren't good representations of the output probability distribution since they don't model the variance well.
- blackle 2y agoMy understanding is that minimizing perplexity (what LLMs are generally optimized for) is equivalent to finding a good probably distribution over the next token.
- ilaksh 2y agoDid I miss it, or did he completely leave out the name of the LLM model he used?
- KTibow 2y agoOn the topic of very strong compression: http://prize.hutter1.net/ http://prize.hutter1.net/
- mmoskal 2y agoA problem with this is that the inference has to be 100% deterministic. For example of you change tailing order of matmul you may get slightly different results. It's easy to imagine people changing tailing in llama.cpp or other libraries. You are very unlikely to get bit exact results from different libraries. Interestingly (though maybe not relevant here) you can also get different results from multi-user inference systems depending on what other requests are in the batch. It's possible to avoid this but I'm pretty sure most systems don't. The "slightly different" bit of course makes it worse - it will work 99% of the time.
- david_draco 2y agoProbably could set the seed and temperature and store these. Run with a few variations and keeping the best would probably work even better, at the expense of more computation.
- jxy 2y agoThere was this: Compression Represents Intelligence Linearly, https://arxiv.org/html/2404.09937v1 https://arxiv.org/html/2404.09937v1
- personjerry 2y agoI mean, sure this is compression in the sense that I can send you a tiny "compressed text" and all you need is this multi-terabyte model to decompress it!
- markisus 2y agoBut we don’t usually count the decompression program size when evaluating compression ratio. Eg 7-Zip is about 1 MB, but you don’t think about that when evaluating particular 7z files.
- Filligree 2y agoWe would if it’s a multi-gigabyte program the receiver doesn’t have installed.
- viraptor 2y agoMaybe not multi-gigabyte, but in a new system/phone in a year, you're basically guaranteed to find at least one tiny model. We may even get some "standard" model everyone can reliably use as a reference.
- Filligree 2y agoAt that point it would be useful. Although, I wonder if it wouldn’t make more sense to train one specifically for the job. Current LLMs can predict HTML, sure, but they’re large and slow for the task.
- markisus 2y agoYeah sometimes compression ratio is not the right question when there are other practical concerns like disk space or user experience. But I do want to point out that almost everyone installs at least one multigigabyte file to decompress other files, and that is the OS.
- Filligree 2y ago
- anonu 2y agoURL to the text is basically compression.
- recursive 2y agoIn some contexts, although it has an extra dependency, which is the web server.
- Intralexical 2y agoYou can already pre-train compression on text without using an LLM: $ curl https://www.gutenberg.org/cache/epub/11/pg11.txt > text.txt $ split -n 500 text.txt trainpart. Using a normal compression algorithm: $ zstd --train trainpart.* -o dictionary Save dictionary of size 112640 into file dictionary $ zstd -vD dictionary text.txt *** Zstandard CLI (64-bit) v1.5.5, by Yann Collet *** text.txt : 15.41% ( 170 KiB => 26.2 KiB, text.txt.zst) For this example, ZSTD warns that the dictionary training set is 10X-100X too small to be efficient. Realistically, I guess you'd train it over E.G. the entire Gutenberg library. Then you can distribute specific books to people who already have the dictionary. Or: $ curl -L https://archive.org/download/completeworksofl1920carr/completeworksofl1920carr_hocr_searchtext.txt.gz | gzip -d | sed -E 's/\s+/ /g' > FullTextsSample.txt $ zstd -v -19 --patch-from FullTextsSample.txt text.txt text.txt : 16.50% ( 170 KiB => 28.1 KiB, text.txt.zst) Not sure how much performance would drop for realistic use. But there are also some knobs you can tune. Refer to: https://github.com/facebook/zstd/#dictionary-compression-how-to https://github.com/facebook/zstd/#dictionary-compression-how... https://github.com/facebook/zstd/wiki/Zstandard-as-a-patching-engine https://github.com/facebook/zstd/wiki/Zstandard-as-a-patchin... $ man zstd - Dictionary occupies only kilobytes or megabytes of storage, instead of gigabytes or terabytes. - Dictionary can be re-trained for specific data at negligble cost. - Compression and decompression are deterministic by default. - Doesn't take large amount of GPU resources to compress/decompression. - This is actually designed to do this.
- Alifatisk 2y agoI am so glad I read your comment, so faschinating!
- Terretta 2y agoInteresting you're showing 15% - 16%, and the LLM technique showed 15%.* (To your point, one of those measures isn't including gigabytes of LLM in its size savings, as if it's part of the .exe size instead.) * EDIT to link to discussion further down: https://news.ycombinator.com/item?id=40245530 https://news.ycombinator.com/item?id=40245530
- tednoob 2y agoIs this method used during training? Seems to me there could be a point to only backpropagate when the model is wrong?
- zwaps 2y agoI mean this is implicit in back propagation, say, you need to store gradients anyway but if you get to a zero loss than you are just done.
- leodriesch 2y agoThe model is always wrong, since it predicts a propability distribution over all possible tokens, but the target has 100% possibility for one token and 0 for all others.
- Tiberium 2y agoAm I missing something, or is there no repo to try it out?
- deleted 2y ago[deleted]
- throwaway81523 2y agoA compression method something like this figured into Vernor Vinge's 1988-era short story "The Blabber".
- KaoruAoiShiho 2y agoCan this be used to save on token cost in LLM inferencing?
- piloto_ciego 2y agoPotentially hot take, but compression is what I think language is for. It’s easier to transmit a story than the entire sensory experience of something happening, so it saves us time and attention. Being able to convert complex tasks and ideas into short stories has likely served us well since the brain burns a lot of calories.
- 65a 2y agoI don't think this is a hot take at all, it's matches my understanding. One of the reasons language itself is so difficult (miscommunication, etc) is we have a mostly similar but not identical "compression table" of ideas that words map to, and why we spend so much time aligning on terms, to ensure correct "decompression". We need compression because internally cognition has a very "wide" set of inputs/outputs (basically meshed), but we only have a serial audio link with relatively narrow bandwidths, so the whole thing that allows us to even discuss this is basically a neat evolutionary hack.
- kmeisthax 2y agoThere's a Google paper from a year ago that proposes using this as a statistical test to determine if a model was trained on a particular text or not.