4 ms·

If you limit yourself to the first 64k words, you have a much larger vocabulary, still limited, and still fitting within two bytes per word. Maybe I'm misunder

by glurgh 13y ago

If you limit yourself to the first 64k words, you have a much larger vocabulary, still limited, and still fitting within two bytes per word. Maybe I'm misunderstanding what you're saying but if you're compressing adaptively by building a dictionary of the target material, 64k words would give you lossless compression of Shakespeare - the unique words in the collected works (with some wiggle room for 'word forms') are in the 30-40k range.

pixelcort 13y ago
Yes, but at that point the dictionary is fixed and could be pre-shared or computed; the file size could be smaller than gzip, for example.
barakm 13y ago
True! You could do that too. Sending the codebook is a cromulent strategy. It'd be lossless too. That's (to a first order) how LZ works. I was arguing you can use a standard codebook (some agreed-upon top-N list) instead. I imagine if you did you'd lose a couple words ("wherefore" springs to mind, as it's not in common English usage) but by and large you'd have just about everything. Technically, you can also have a lossless JPEG (or at least, within epsilon), but that's not why it's done. More importantly, the entropy of English text isn't really that high -- ~ 1 bit per letter -- which means lossless compression works pretty well on it. Images less so. The lesson here being that compressing English text as suggested in the article is rather meaningless, and can be done much, much better. When the article asks at the end: > We're sensitive to data loss in text form: we can only consume a few dozens of bytes per second, and so any error is obvious. > Conversely, we're almost blind to it in pictures and images: and so losing quality doesn't bother us all that much. > Should it? My answer is "mu". I wanted to answer "no", but that would mean I even accept the question as valid. EDIT: s/in this way/as suggested in the article/ for clarity.
- glurgh 13y ago
  I suppose my confusion arises from the fact that pretty much any practical compression scheme is essentially 'content adaptive' rather than 'standard codebooky'. In the lossy cases, add whatever magical filtering to toss out the perceptually irrelevant frequencies. I agree with you that the article misunderstands compression and coding in general I just couldn't quite figure out what one of your counter-examples was about.

by glurgh 13y ago

pixelcort 13y ago

Yes, but at that point the dictionary is fixed and could be pre-shared or computed; the file size could be smaller than gzip, for example.

barakm 13y ago

True! You could do that too. Sending the codebook is a cromulent strategy. It'd be lossless too. That's (to a first order) how LZ works. I was arguing you can use a standard codebook (some agreed-upon top-N list) instead. I imagine if you did you'd lose a couple words ("wherefore" springs to mind, as it's not in common English usage) but by and large you'd have just about everything. Technically, you can also have a lossless JPEG (or at least, within epsilon), but that's not why it's done. More importantly, the entropy of English text isn't really that high -- ~ 1 bit per letter -- which means lossless compression works pretty well on it. Images less so. The lesson here being that compressing English text as suggested in the article is rather meaningless, and can be done much, much better. When the article asks at the end: > We're sensitive to data loss in text form: we can only consume a few dozens of bytes per second, and so any error is obvious. > Conversely, we're almost blind to it in pictures and images: and so losing quality doesn't bother us all that much. > Should it? My answer is "mu". I wanted to answer "no", but that would mean I even accept the question as valid. EDIT: s/in this way/as suggested in the article/ for clarity.

glurgh 13y ago
I suppose my confusion arises from the fact that pretty much any practical compression scheme is essentially 'content adaptive' rather than 'standard codebooky'. In the lossy cases, add whatever magical filtering to toss out the perceptually irrelevant frequencies. I agree with you that the article misunderstands compression and coding in general I just couldn't quite figure out what one of your counter-examples was about.