6 ms·
That's an interesting idea, and it's cool that he took it as far as he did, but I'm surprised by the last paragraph: We're sensitive to data loss in text form:
by ZeroGravitas 13y ago
That's an interesting idea, and it's cool that he took it as far as he did, but I'm surprised by the last paragraph:
We're sensitive to data loss in text form: we can only consume a few dozens of bytes per second, and so any error is obvious. Conversely, we're almost blind to it in pictures and images: and so losing quality doesn't bother us all that much.
I'm not aware of any "lossy" encodings for text, but I assume they'd do pretty well on shakespeare and pretty terrible on images encoded as text. It's just that the techniques are tuned for the standard data, so most things you smuggle through are going to be "weird" and suffer more than usual.
- kevingadd 13y agoYou could probably quite effectively do 'lossy' compression on text given enough knowledge of the language. Compress words with similar meanings into each other, automatically select variants of words based on context (plurality, etc), automatically insert punctuation, and so on. The result would probably look like it was written by someone with an incomplete knowledge of English (or by a machine) but might end up being quite readable. Of course, on the other hand, you can just compress the text with LZMA :)
- drewcrawford 13y agoit's comomn kdownlege in inoitrmfaon thoery ciecrls taht you can reoerdr lttres in wrods and the relsut is pecfretly undeastnrdlabe as lnog as you keep the fsirt and lsat letrtes the smae. http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=4126417 http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=4126...
- VMG 13y agoThat is more about built-in redundancy and error correction than lossy compression.
- zokier 13y agoBut such features could be exploited to make lossy compression. If humans can recognize words with some of the letters randomized, then there is no need to store those letters individually. Instead just insert random letters at decomperssion phase.
- tbrownaw 13y agohttp://www.mrc-cbu.cam.ac.uk/people/matt.davis/cmabridge/ http://www.mrc-cbu.cam.ac.uk/people/matt.davis/cmabridge/ I've written this page, to try to explain the science behind this meme. There are elements of truth in this, but also some things which scientists studying the psychology of language (psycholinguists) know to be incorrect. The page also contains a number of counter-examples, show that the result often is not "pflteecry uaaedlbdrsntne".
- reeses 13y agoThis is in fact not proven for the entire population.
- abruzzi 13y agoThe easiest/quickest lossy text compressions that could theoretically almost halve the size of your data is to drop capitalization (and also assuming you're using some minimal encoding that can only encode the characters you're using.) there is some semantic info that is imparted by capitals, but not much.
- mistercow 13y agoNo, that would not cut the length in half, or anything close to it. One form of "minimal encoding" you're describing would be to use arithmetic coding where the predictor simply assigns equal probability to each symbol actually used in the input. For example, if you encode a 1000 letter message which includes at least one instance of every letter (both upper and lower case), then you'd have 52 symbols, and your simple prediction step would simply assign 1/52 as the probability for each symbol. This will result in a constant 5.7 bits of output per symbol. So your 1000 letter message would encode at 5700 bits. Now, suppose you discard capitalization. Now there are 26 symbols, each assigned a probability of 1/26, resulting in a constant 4.7 bits per symbol. Your 1000 letter message will now encode at 4700 bits. So you only save about 17% in this very generously constructed case. The mistake you've made is to assume that cutting the number of symbols in half will cut the symbol length in half. In fact, reducing the number of symbols by half merely reduces the symbol length by a single bit. For example, you need 8 bits to specify between 256 possibilities, but if you cut the possibilities down to 128, you still need 7 bits. In addition, lossy compression is usually combined with lossless compression, and a smart predictor could use context to reduce the cost of capital letters to much less than one bit per symbol (assuming that the text has a predictable structure, like human language).
- minikites 13y agoPost the text to Tumblr and JPGs to Flickr and then sit back and reflect on lossy text compression for website titles.
- tg3 13y agoExactly - the whole point of lossy compression is to compress it in a way that has a minimal effect on the type of data you're trying to compress while maximizing the compression. Using a compression method on a different data set is unsurprisingly ineffective. We're not "blind" to the changes in images, but the algorithm is designed that the changes are minimal when in image form.
- repiret 13y agoIn a certain sense, thats what Reader's Digest Condensed Books are all about - lossy text compression. But we let machines do it all the time too. Go from UTF-8 to ASCII-7. You won't loose much in English - orientation of quotation marks, distinction among various types of dashes. Naïve and Café and El Niño and other words borrowed from other languages and commonly spelt with their native character set would also need to be spelt differently. You could drop down to 5 bits per character, giving you only upper case letters and room for 6 punctuations. Numerals would need to be spelt out, which means you could no longer distinguish among these: 1, One, ONE, one. You can also find lots of lossy text compression, focused on minimizing characters, on Twitter or in text messages.
- VMG 13y agoIn some sense TL;DR is a form of lossy compression. You don't express your complete thoughts but a shortened version of it. IMHO.
- csense 13y ago> orientation of quotation marks I hate it when I open a document and there's non-ASCII characters cluttering it up just because someone wanted magic quote marks. If you must have oriented quotes, you should do them the ``LaTeX way,'' so you don't have to get into Unicode, which sucks [1]. [1] https://news.ycombinator.com/item?id=4835507 https://news.ycombinator.com/item?id=4835507
- CJefferson 13y agoYes, I'm sure I could come up with a lossy encryption system for text. There was a popular meme going around for a while of sorting all but the first and last characters of a word.
- TheCraiggers 13y agoInteresting idea. So basically any word with greater than 3 characters could be converted into beginning / end char and a number representing the number of chars in between. I wonder how much of our ability to read text like that is based on context though. Also, I'm going to assume that this only works with fairly common words (which is unique for each person). For example, I'm guessing running said compression on the works of Shakespeare wouldn't work as well, since most people struggle to make sense of them to begin with.
- hobs 13y agoi18n
- deleted 13y ago[deleted]
- growse 13y agoLossy encodings for text are easy. I can compress the entirety of 'Romeo And Juliet' to this: "Boy and girl fall in love, they both die." Yes, it's lost a bit of detail and resolution, but it's significantly more compact than the original, while retaining somewhat the same recognizable message. All we need next is a computer algorithm that generates TLDR.
- czr80 13y agoI hear Yahoo might have one.