3 ms·

> Encodings are fundamentally hard in our current code environment. If your language doesn't make you explicitly think about encodings, you are writing bad code

> Encodings are fundamentally hard in our current code environment. If your language doesn't make you explicitly think about encodings, you are writing bad code. Period. Full stop. If your language does make you think about encodings and you just make it go away with compiler incantations or just bumbling about until the problems seems to go away, sort of, as long as you don't poke them too hard, you are writing bad code. Period. Full stop. If your language has no support at all for encodings, may God have mercy on your soul. This, to me, is the fundamental flaw. By now we should be able to have a single encoding that takes up as little space as possible while supporting every known character, and leaving room for more. Most machines now are at least 32-bit... that's almost 4.3 billion characters. Surely there are less than that in the world? One of my "Grand Lifetime Projects" is to build a new programming language and a new OS built with it. Part of that will certainly be handling strings in an efficient way, both in terms of computer and programmer time. I have some ideas swirling around for creating One True Encoding that allows for extensibility.

jerf 16y ago
"This, to me, is the fundamental flaw. By now we should be able to have a single encoding that takes up as little space as possible while supporting every known character, and leaving room for more." That's pretty much UTF-8. If you're going to stuff everything into one encoding, you are going to have to make tradeoffs. See also UCS-4, which is simply "throw 32 bits at every character". Nobody uses it because it makes everything pretty big. (At least a 3-byte CJK character tends to mean more on average than a single English character.) If you haven't already, take some time to read over the Unicode standards. It is very enlightening. This is especially true if you want to make the "one true encoding"; gotta know what the bar is to beat, right? There's way more to Unicode than just "Here's a catalog of all possible characters and here's numbers for them all", and there are reasons why there's way more than that.

jerf 16y ago

"This, to me, is the fundamental flaw. By now we should be able to have a single encoding that takes up as little space as possible while supporting every known character, and leaving room for more." That's pretty much UTF-8. If you're going to stuff everything into one encoding, you are going to have to make tradeoffs. See also UCS-4, which is simply "throw 32 bits at every character". Nobody uses it because it makes everything pretty big. (At least a 3-byte CJK character tends to mean more on average than a single English character.) If you haven't already, take some time to read over the Unicode standards. It is very enlightening. This is especially true if you want to make the "one true encoding"; gotta know what the bar is to beat, right? There's way more to Unicode than just "Here's a catalog of all possible characters and here's numbers for them all", and there are reasons why there's way more than that.