5 ms·

I don't understand how the algorithm on page 21 works. Aren't many Unicode characters formed with multiple code points, like <modifying-mark><basic-character>?

I don't understand how the algorithm on page 21 works. Aren't many Unicode characters formed with multiple code points, like <modifying-mark><basic-character>? If these are reversed to be <basic-character><modifying-mark>, then the textual output would actually be different, wouldn't it? Shouldn't rev(1) reverse graphemes instead of code points?

slededit 10y ago
It looks like the talk is limited to instructions on how to produce well formed utf-8 string transformations. Once you go farther than that your into the nitty gritty of internationalization which is itself a moving target as cultures change. For example how should strrev handle BIDI control characters? When is the unicode BIDI algorithm appropriate and when is it not? (hint: formulas look pretty messed up with it). At some point these become application specific.
kccqzy 10y ago
I would say, it should instead reverse extended grapheme clusters. The advent of emojis and emoji ZWJ sequences make those kind of "characters" especially common. For example the family emoji ‍‍‍‍‍‍ is thought of as a single "character" from the user's perspective but it actually has seven code points. EDIT: HN ate my emoji so there's a link to its emojipedia page: http://emojipedia.org/family-man-woman-girl-boy/ http://emojipedia.org/family-man-woman-girl-boy/
- Animats 10y ago
  I had no idea that had been accepted. Here's Unicode TR51, with the whole set.[1] You can now specify skin tone, but cannot yet mix skin tone within a family group. Someone is probably already demanding that feature. http://www.unicode.org/reports/tr51/tr51-2.html http://www.unicode.org/reports/tr51/tr51-2.html
- legulere 10y ago
  > The advent of emojis and emoji ZWJ sequences make those kind of "characters" especially common. They're already common through MacOS filesystems. MacOS by standard decomposes characters (ä into ¨+a)
  - masklinn 10y ago
    > ä into ¨+a You've reversed the encoding (the combining character follows, so U+00E4 decomposes to U+0061 U+0308), and that's a simpler case as combining characters are combining, they have very specific properties which mark them as combining characters and as continuation of a grapheme. AFAIK that's not the case for ZWJ sequences. There's also a case to be made for reverse(WOMAN ZWJ HEAVY BLACK HEART ZWJ MAN) to be MAN ZWJ HEAVY BLACK HEART ZWJ WOMAN (sadly the latter does not seem to be handled correctly, at least by Apple's rendering system, it fails to combine into a single "couple" glyph)
jprzybyl 10y ago
> Shouldn't rev(1) reverse graphemes instead of code points? I honestly don't know. Is the intended purpose of this program to reverse bytes, or reverse characters, or reverse grapheme clusters? Or extended grapheme clusters? There's no spec - this has never been in POSIX. What is your expected behavior? Is it mine? For what it's worth, I needed rev recently, but forgot that it existed and did this: perl -ne 'chomp; print scalar reverse . "\n"' If I need it to handle UTF-8 in a certain way, I can use pragmas to change it's behavior. (I'm pretty sure that this, as it is, will ignore the surrounding locale.)
- stsp 10y ago
  UTF-8 can also be handled in several ways. There is a lot of middle ground between software that handles bytes, and typesetting software which is fully unicode-aware. The small unix utilities fall somewhere in there.
  - masklinn 10y ago
    > UTF-8 can also be handled in several ways. UTF8 can not be handled in several ways without breaking it, it's a pretty straightforward and strict encoding.
- kahirsch 10y ago
  What would you expect the output to be when the input is: nôn nôn Sending that through rev (with a UTF-8 locale), I get n̂on nôn By the way, did you know the perl -l flag removes newlines on input and adds them for a print, so your command could just be: perl -lne 'print scalar reverse' And, for a unicode-aware version: perl -CDS -lne 'print scalar reverse'
niccaluim 10y ago
FWIW the Unicode spec describes combining marks as characters in their own right. So if the intent is to reverse characters, page 21 does the job. The resulting sequences will potentially be defective but not ill-formed. That being said, an FAQ on combining characters points out that Unicode's definition of "character" may not match an end user's, and that it's best to use the word "grapheme" instead for clarity. (And that being said, if the typical end user knows what "grapheme" means, I'll eat my cat.) So from a practical standpoint, it's best to make sure that any input to rev is in one of the composed normal forms. (Incidentally, the proper sequence is <base character><combining character>…, not the other way around.)
- bhaak 10y ago
  > So from a practical standpoint, it's best to make sure that any input to rev is in one of the composed normal forms. But there are real world characters that don't have precomposed forms (IIRC e.g. indic scripts).
- kps 10y ago
  > Incidentally, the proper sequence is <base character><combining character>…, not the other way around. A mistake in Unicode, IMHO. The other way around, it would have been possible to identify the end of a combining sequence without looking past the sequence. Also, ‘dead keys’ could have directly generated the required combining characters just like normal characters, rather than requiring special processing.
stsp 10y ago
Yes, rev(1) probably should handle combined characters. But those are a property of Unicode, not UTF-8. UTF-8 encodes code points, and we often try to get away without decoding them. Of course the resulting Unicode can change its meaning but it's still valid Unicode (and valid UTF-8). In some cases we already look at Unicode properties (such as a character's column width). So perhaps we can find a nice way to fix this problem in rev(1), some day. There are many more interesting Unicode issues we don't address in OpenBSD's UTF-8 support (e.g. han unification, pre-composed vs de-composed normalization). But we have to start somewhere. Perhaps, eventually, someone will specify a minimal and sane variant of unicode, which removes all the ambiguities, edge cases, and silly symbols. We'd probably switch over in a heartbeat.
- ramshorns 10y ago
  What would a minimal and sane variant of Unicode be like? Removing the weird behaviour of Unicode would necessarily mean removing support for some characters, like those that only exist in decomposed form with combining diacritics, and some types of scripts like right-to-left. Mapping code points, characters and graphemes one-to-one seems like it would make text processing easier at the cost of excluding a large portion of the character set. I guess it would form a middle ground; US-ASCII is also a minimal subset of Unicode where text processing is easy.
  - karlmdavis 10y ago
    Ding ding! Hard things are hard. It seems... at least a bit arrogant for a developer that doesn't write any of the languages that rely on these features to claim that they're insane and excessive.
- Arnt 10y ago
  You'd switch, lose the ability to convert between unicode and those random 8-bit encoding, and end up having support for the encodings you can't solve to converting to unicode any more.
- ChoHag 10y ago
  Once we have a minimal and sane variant of humans without ambiguities, edge cases and silly symbols we can get right on that.
masklinn 10y ago
> then the textual output would actually be different, wouldn't it? Yes. The provided algorithm doesn't break UTF-8, but it will break the encoded text as it's not unicode-aware. I'm pretty sure it'll also leave through invalid UTF8 (lone surrogates or non-shortest encoding).

slededit 10y ago

It looks like the talk is limited to instructions on how to produce well formed utf-8 string transformations. Once you go farther than that your into the nitty gritty of internationalization which is itself a moving target as cultures change. For example how should strrev handle BIDI control characters? When is the unicode BIDI algorithm appropriate and when is it not? (hint: formulas look pretty messed up with it). At some point these become application specific.

kccqzy 10y ago

I would say, it should instead reverse extended grapheme clusters. The advent of emojis and emoji ZWJ sequences make those kind of "characters" especially common. For example the family emoji ‍‍‍‍‍‍ is thought of as a single "character" from the user's perspective but it actually has seven code points. EDIT: HN ate my emoji so there's a link to its emojipedia page: http://emojipedia.org/family-man-woman-girl-boy/ http://emojipedia.org/family-man-woman-girl-boy/

Animats 10y ago
I had no idea that had been accepted. Here's Unicode TR51, with the whole set.[1] You can now specify skin tone, but cannot yet mix skin tone within a family group. Someone is probably already demanding that feature. http://www.unicode.org/reports/tr51/tr51-2.html http://www.unicode.org/reports/tr51/tr51-2.html
legulere 10y ago
> The advent of emojis and emoji ZWJ sequences make those kind of "characters" especially common. They're already common through MacOS filesystems. MacOS by standard decomposes characters (ä into ¨+a)
- masklinn 10y ago
  > ä into ¨+a You've reversed the encoding (the combining character follows, so U+00E4 decomposes to U+0061 U+0308), and that's a simpler case as combining characters are combining, they have very specific properties which mark them as combining characters and as continuation of a grapheme. AFAIK that's not the case for ZWJ sequences. There's also a case to be made for reverse(WOMAN ZWJ HEAVY BLACK HEART ZWJ MAN) to be MAN ZWJ HEAVY BLACK HEART ZWJ WOMAN (sadly the latter does not seem to be handled correctly, at least by Apple's rendering system, it fails to combine into a single "couple" glyph)

jprzybyl 10y ago

> Shouldn't rev(1) reverse graphemes instead of code points? I honestly don't know. Is the intended purpose of this program to reverse bytes, or reverse characters, or reverse grapheme clusters? Or extended grapheme clusters? There's no spec - this has never been in POSIX. What is your expected behavior? Is it mine? For what it's worth, I needed rev recently, but forgot that it existed and did this: perl -ne 'chomp; print scalar reverse . "\n"' If I need it to handle UTF-8 in a certain way, I can use pragmas to change it's behavior. (I'm pretty sure that this, as it is, will ignore the surrounding locale.)

stsp 10y ago
UTF-8 can also be handled in several ways. There is a lot of middle ground between software that handles bytes, and typesetting software which is fully unicode-aware. The small unix utilities fall somewhere in there.
- masklinn 10y ago
  > UTF-8 can also be handled in several ways. UTF8 can not be handled in several ways without breaking it, it's a pretty straightforward and strict encoding.
kahirsch 10y ago
What would you expect the output to be when the input is: nôn nôn Sending that through rev (with a UTF-8 locale), I get n̂on nôn By the way, did you know the perl -l flag removes newlines on input and adds them for a print, so your command could just be: perl -lne 'print scalar reverse' And, for a unicode-aware version: perl -CDS -lne 'print scalar reverse'

niccaluim 10y ago

FWIW the Unicode spec describes combining marks as characters in their own right. So if the intent is to reverse characters, page 21 does the job. The resulting sequences will potentially be defective but not ill-formed. That being said, an FAQ on combining characters points out that Unicode's definition of "character" may not match an end user's, and that it's best to use the word "grapheme" instead for clarity. (And that being said, if the typical end user knows what "grapheme" means, I'll eat my cat.) So from a practical standpoint, it's best to make sure that any input to rev is in one of the composed normal forms. (Incidentally, the proper sequence is <base character><combining character>…, not the other way around.)

bhaak 10y ago
> So from a practical standpoint, it's best to make sure that any input to rev is in one of the composed normal forms. But there are real world characters that don't have precomposed forms (IIRC e.g. indic scripts).
kps 10y ago
> Incidentally, the proper sequence is <base character><combining character>…, not the other way around. A mistake in Unicode, IMHO. The other way around, it would have been possible to identify the end of a combining sequence without looking past the sequence. Also, ‘dead keys’ could have directly generated the required combining characters just like normal characters, rather than requiring special processing.

stsp 10y ago

Yes, rev(1) probably should handle combined characters. But those are a property of Unicode, not UTF-8. UTF-8 encodes code points, and we often try to get away without decoding them. Of course the resulting Unicode can change its meaning but it's still valid Unicode (and valid UTF-8). In some cases we already look at Unicode properties (such as a character's column width). So perhaps we can find a nice way to fix this problem in rev(1), some day. There are many more interesting Unicode issues we don't address in OpenBSD's UTF-8 support (e.g. han unification, pre-composed vs de-composed normalization). But we have to start somewhere. Perhaps, eventually, someone will specify a minimal and sane variant of unicode, which removes all the ambiguities, edge cases, and silly symbols. We'd probably switch over in a heartbeat.

ramshorns 10y ago
What would a minimal and sane variant of Unicode be like? Removing the weird behaviour of Unicode would necessarily mean removing support for some characters, like those that only exist in decomposed form with combining diacritics, and some types of scripts like right-to-left. Mapping code points, characters and graphemes one-to-one seems like it would make text processing easier at the cost of excluding a large portion of the character set. I guess it would form a middle ground; US-ASCII is also a minimal subset of Unicode where text processing is easy.
- karlmdavis 10y ago
  Ding ding! Hard things are hard. It seems... at least a bit arrogant for a developer that doesn't write any of the languages that rely on these features to claim that they're insane and excessive.
Arnt 10y ago
You'd switch, lose the ability to convert between unicode and those random 8-bit encoding, and end up having support for the encodings you can't solve to converting to unicode any more.
ChoHag 10y ago
Once we have a minimal and sane variant of humans without ambiguities, edge cases and silly symbols we can get right on that.

masklinn 10y ago

> then the textual output would actually be different, wouldn't it? Yes. The provided algorithm doesn't break UTF-8, but it will break the encoded text as it's not unicode-aware. I'm pretty sure it'll also leave through invalid UTF8 (lone surrogates or non-shortest encoding).