3 ms·

This is roughly true. Obviously with UTF-8 or UTF-16 you'll have O(n) access times for an arbitrary code point, but you can always convert to UTF-32 and that en

This is roughly true. Obviously with UTF-8 or UTF-16 you'll have O(n) access times for an arbitrary code point, but you can always convert to UTF-32 and that encoding is just an array of code points so you get O(1) random access back.

remram 4y ago
But random access to codepoints is still useless for all purposes.
WorldMaker 4y ago
UTF-32 !== UCS-2 and makes no guarantees about fixed code point size like UCS-2 did (and is considered deprecated because of that). There's an encoding issue with UTF-16 extending past the current "Astral Plane", but neither UTF-8 nor UTF-32 have that problem. We have no current idea at all what we might ever think to encode past the "Astral Plane", but that doesn't mean you should ever assume UTF-32 is UCS-2 and free of surrogate code points.
- tinalumfoil 4y ago
  You gave me enough doubt that I went and looked this up. > UTF-32 is a fixed-length encoding used to encode Unicode code points that uses exactly 32 bits per code point... The main advantage of UTF-32 is that the Unicode code points are directly indexed. Finding the Nth code point in a sequence of code points is a constant-time operation. https://en.wikipedia.org/wiki/UTF-32 https://en.wikipedia.org/wiki/UTF-32 > The [UTF-32] Unicode encoding form that assigns each Unicode scalar value to a single unsigned 32-bit code unit with the same numeric value as the Unicode scalar value. > Because surrogate code points are not included in the set of Unicode scalar values, UTF-32 code units in the range 0000D80016..0000DFFF16 are ill-formed. https://www.unicode.org/versions/Unicode5.0.0/ch03.pdf https://www.unicode.org/versions/Unicode5.0.0/ch03.pdf (page 40) > As a consequence, UCS-4 can now be taken effectively as an alias for the Unicode encoding form UTF-32, except that UTF-32 has the extra requirement that additional Unicode semantics be observed for all characters. https://www.unicode.org/versions/Unicode5.0.0/appC.pdf https://www.unicode.org/versions/Unicode5.0.0/appC.pdf (page 7)
  - WorldMaker 4y ago
    Yeah, I did briefly confuse UCS-2 (16-bit) and UCS-4 (32-bit) in my terminology, but the point still stands: lots of people expected the optimism in Unicode 1.0 that Unicode stopped at UCS-2 (16-bits) and never expected the "Astral Plane" to open up (and UCS-4 added). There was lots of software (including Windows) that had to do a lot of work to fix assumptions that UCS-2 was "the last word" on Unicode encoding and switch to UTF-16 or UTF-8. That "except" there is still doing a lot of work pointing out that UTF-32 is not exactly UCS-4, so that point still stands. There's no surrogate code points allowed in UTF-32 today, but again if you want to assume "future compatibility" you shouldn't assume that UTF-32 is UCS-4 and will never have surrogate codepoints. As I said, we don't know if that next "plane" will ever open up into UCS-8 (64-bit) encodings, but if you are assuming it will never open up you are making the exact same sorts of assumptions that Unicode 1.0 users made with UCS-2 and had to consequently fix a decade later. (And some of those same applications would have to fix again if 64-bit encodings opened up, because like I said UTF-16 has a math bug that its surrogates can't encode there, though UTF-8 is fine, and UTF-16 may be the worst of all Unicode encodings in the long run, but its legacy lives on so much because of all the early bets on UCS-2 being the last word and "easiest" Unicode that turned out to be bad assumptions when the "astral plane" opened up.)

remram 4y ago

But random access to codepoints is still useless for all purposes.

WorldMaker 4y ago

UTF-32 !== UCS-2 and makes no guarantees about fixed code point size like UCS-2 did (and is considered deprecated because of that). There's an encoding issue with UTF-16 extending past the current "Astral Plane", but neither UTF-8 nor UTF-32 have that problem. We have no current idea at all what we might ever think to encode past the "Astral Plane", but that doesn't mean you should ever assume UTF-32 is UCS-2 and free of surrogate code points.

tinalumfoil 4y ago
You gave me enough doubt that I went and looked this up. > UTF-32 is a fixed-length encoding used to encode Unicode code points that uses exactly 32 bits per code point... The main advantage of UTF-32 is that the Unicode code points are directly indexed. Finding the Nth code point in a sequence of code points is a constant-time operation. https://en.wikipedia.org/wiki/UTF-32 https://en.wikipedia.org/wiki/UTF-32 > The [UTF-32] Unicode encoding form that assigns each Unicode scalar value to a single unsigned 32-bit code unit with the same numeric value as the Unicode scalar value. > Because surrogate code points are not included in the set of Unicode scalar values, UTF-32 code units in the range 0000D80016..0000DFFF16 are ill-formed. https://www.unicode.org/versions/Unicode5.0.0/ch03.pdf https://www.unicode.org/versions/Unicode5.0.0/ch03.pdf (page 40) > As a consequence, UCS-4 can now be taken effectively as an alias for the Unicode encoding form UTF-32, except that UTF-32 has the extra requirement that additional Unicode semantics be observed for all characters. https://www.unicode.org/versions/Unicode5.0.0/appC.pdf https://www.unicode.org/versions/Unicode5.0.0/appC.pdf (page 7)
- WorldMaker 4y ago
  Yeah, I did briefly confuse UCS-2 (16-bit) and UCS-4 (32-bit) in my terminology, but the point still stands: lots of people expected the optimism in Unicode 1.0 that Unicode stopped at UCS-2 (16-bits) and never expected the "Astral Plane" to open up (and UCS-4 added). There was lots of software (including Windows) that had to do a lot of work to fix assumptions that UCS-2 was "the last word" on Unicode encoding and switch to UTF-16 or UTF-8. That "except" there is still doing a lot of work pointing out that UTF-32 is not exactly UCS-4, so that point still stands. There's no surrogate code points allowed in UTF-32 today, but again if you want to assume "future compatibility" you shouldn't assume that UTF-32 is UCS-4 and will never have surrogate codepoints. As I said, we don't know if that next "plane" will ever open up into UCS-8 (64-bit) encodings, but if you are assuming it will never open up you are making the exact same sorts of assumptions that Unicode 1.0 users made with UCS-2 and had to consequently fix a decade later. (And some of those same applications would have to fix again if 64-bit encodings opened up, because like I said UTF-16 has a math bug that its surrogates can't encode there, though UTF-8 is fine, and UTF-16 may be the worst of all Unicode encodings in the long run, but its legacy lives on so much because of all the early bets on UCS-2 being the last word and "easiest" Unicode that turned out to be bad assumptions when the "astral plane" opened up.)