3 ms·
Maybe this was the inspiration for your project but I just saw a paper that applied compressors as part of a text classification system getting accuracy competi
by hedgehog 3y ago
Maybe this was the inspiration for your project but I just saw a paper that applied compressors as part of a text classification system getting accuracy competitive with BERT. 14 lines of Python. https://aclanthology.org/2023.findings-acl.426/ https://aclanthology.org/2023.findings-acl.426/
- canjobear 3y agoYeah, that paper made me wonder how well gzip would work as a language model so I typed up this thing in a coffee shop
- sam_lowry_ 3y agoReminds immediately this famous article by Ted Chiang "ChatGPT Is a Blurry JPEG of the Web " https://www.newyorker.com/tech/annals-of-technology/chatgpt-is-a-blurry-jpeg-of-the-web https://www.newyorker.com/tech/annals-of-technology/chatgpt-...
- DonHopkins 3y agoI'm looking forward to Progressive ChatGPT, that immediately answers your question with an thoughtfully chosen emoji, that expands into an irresistible clickbait title, that expands into a viral tweet, that expands into a fascinating forum comment, that expands into a ranting and raving blog post, that expands into a ponderous wikipedia page, that expands into a voluminous e-book, that expands into ...
- dleeftink 3y agoThank you for posting, I enjoyed the article. In true cybernetics fashion, the maximum compression ratio of information seems to be the current grail of our machine learning efforts, whether that is through inferring heuristics or gradually incorporating knowledge about the world. I'm interested to see where on the information density gradient various organisms exists in their ability to do so..DNA seems awfully wordy, but has proven to be a very dense storage medium.
- visarga 3y agoThere is also an approach to use a LLM and only compress the errors from LLM to the target text. You'd have to add the LLM to the final size, but you could have it already downloaded at the target, in which case I expect nothing will beat it for text compression. Of course it would be very slow.
- visarga 3y agoYes, but this paper was about classification, not language generation.