2 ms·
SCSU and BOCU-1 can both help a lot with that last point. They're two standard methods of compressing unicode strings that are primarily in a single language—ob
by PeCaN 10y ago
SCSU and BOCU-1 can both help a lot with that last point. They're two standard methods of compressing unicode strings that are primarily in a single language—obviously it's not as good as lz4 or gzip or other general-purpose compression algorithms, but it also doesn't increase the size so is ideal for up to ~a few kb.
- mjevans 10y agoI think I'd prefer transparent stream/file-system compression of text documents for this type of issue.
- rspeer 10y agoSCSU and BOCU-1 were never seriously used, as far as I can tell, except to prove a point back when developers were uncertain about implementing Unicode. Unicode standardizer: "Your software needs to support more than one language at a time. Please use Unicode." Developer: "I don't want to. I've already got a codepage that's designed for the language I care about, and Unicode will make everything take up too much space." Unicode standardizer: "Here's BOCU-1, an encoding of Unicode that compresses monolingual text into nearly the same amount of space as your favorite codepage." Developer: "Uh, thanks, but that's weird." Unicode standardizer: "Yeah, never mind. How about you try this new encoding called UTF-8?" Developer: "Oh, this works really well and I guess it's small enough. I'll use it."