3 ms·
> I've never understood why we can't just use a really inefficient-but-practical string encoding, e.g. UTF-32, with a very low-cost stream compression algorithm
by vardump 11y ago
> I've never understood why we can't just use a really inefficient-but-practical string encoding, e.g. UTF-32, with a very low-cost stream compression algorithm on top.
That's right. We could figure out how to represent common characters with less bits. For low-cost character stream compression, byte granularity should work nicely with computers, so we should it. A byte could have a bit that indicates the range is extended, like highest bit, because it's easy to process. Same idea could be chained to represent full range of unicode characters. That way we'd have compressed UTF-32 characters!
- derefr 11y agoI meant more something like RLE or simple Huffman encoding. A "string" type like Redis's dict type or ObjC's array type, where the internal representation changes as the allocated object grows in size and therefore is worth making different trade-offs on. (So, a five-byte string, maybe UTF-8 is a fine encoding. A 3KB string? Worth prepending a compression dictionary to.)