4 ms·
The main problem is that a lot of characters in real world are from ASCII part. So for 24-bit 66% of memory space will be wasted. And it's bad not only because
by vbezhenar 11y ago
The main problem is that a lot of characters in real world are from ASCII part. So for 24-bit 66% of memory space will be wasted. And it's bad not only because it uses RAM, HDD, bandwidth, but generally because it uses precious processor cache storage.
- derefr 11y agoGiven an efficient string processing methodology mostly consisting of 1. holding onto opaque string binaries until you need to compose them all together at the end (i.e. the way Erlang's iolists work) and 2. applying encapsulated operations over a given string (e.g. "split", "scan with regular expression") rather than trying to index into it; I've never understood why we can't just use a really inefficient-but-practical string encoding, e.g. UTF-32, with a very low-cost stream compression algorithm on top.
- vardump 11y ago> I've never understood why we can't just use a really inefficient-but-practical string encoding, e.g. UTF-32, with a very low-cost stream compression algorithm on top. That's right. We could figure out how to represent common characters with less bits. For low-cost character stream compression, byte granularity should work nicely with computers, so we should it. A byte could have a bit that indicates the range is extended, like highest bit, because it's easy to process. Same idea could be chained to represent full range of unicode characters. That way we'd have compressed UTF-32 characters!
- derefr 11y agoI meant more something like RLE or simple Huffman encoding. A "string" type like Redis's dict type or ObjC's array type, where the internal representation changes as the allocated object grows in size and therefore is worth making different trade-offs on. (So, a five-byte string, maybe UTF-8 is a fine encoding. A 3KB string? Worth prepending a compression dictionary to.)