5 ms·

Unicode really is an impossibly bottomless well of trivia and bad decisions. As another example, the article's RFC warns against allowing legacy ASCII control c

Unicode really is an impossibly bottomless well of trivia and bad decisions. As another example, the article's RFC warns against allowing legacy ASCII control characters on the grounds that they can be confusing to display to humans, but says nothing about the Explicit Directional Overrides characters that https://www.unicode.org/reports/tr9/#Explicit_Directional_Overrides https://www.unicode.org/reports/tr9/#Explicit_Directional_Ov... suggests should "be avoided wherever possible, because of security concerns".

estebank 1y ago
The security concerns are those of "Trojan source", where the displayed text doesn't correspond to the bytes on the wire.[1] I don't think a wire protocol should necessarily restrict them, for the sake of compatibility with existing text corpus out there, but a fair observation. 1: https://trojansource.codes/ https://trojansource.codes/
- yencabulator 1y ago
  The enforcement is an app-level issue, depending on the semantics of the field. I agree it doesn't belong in the low-level transport protocol. The rules for "username", "display name", "biography", "email address", "email body" and "contents of uploaded file with name foo.txt" are not all going to be the same.
  - Waterluvian 1y ago
    Can a regular expression be used to restrict Unicode chars like the ones described? I’m imagining a listing of regex rules for the various gotchas, and then a validation-level use that unions the ones you want.
    - fluoridation 1y ago
      Why would you need a regular expression for that? It's just a list of characters.
      - Waterluvian 1y ago
        There’s cases where certain characters coming before or after others is what creates the issue.
arp242 1y ago
I always thought you kind of need those directional control characters to correctly render bidi text? e.g. if you write something in Hebrew but include a Latin word/name (or the reverse).
- dcrazy 1y ago
  This is the job of the Bidi Algorithm: https://www.unicode.org/reports/tr9/ https://www.unicode.org/reports/tr9/ Of course, this is an “annex”, not part of the core Unicode spec. So in situations where you can’t rely on the presentation layer’s (correct) implementation of the Bidi algorithm, you can fall back to directional override/embedding characters.
  - acdha 1y ago
    Over the years I’ve run into a few situations where the rules around neutral characters didn’t produce the right result and so we had to use the override characters to force the correct display. It’s completely a niche but very handy when you are mixing quotes within a complex text.
- layer8 1y ago
  Read the parent’s link. The characters “to be avoided” are a particular special-purpose subset, not directional control characters in general.
weinzierl 1y ago
I wouldn’t be so harsh. I think the Unicode Consortium not only started with good intentions but also did excellent work for the first decade or so. I just think they got distracted when the problems got harder, and instead of tackling them head-on, they now waste a lot of their resources on busywork - good intentions notwithstanding. Sure, it’s more fun standardizing sparkling disco balls than dealing with real-world pain points. That OpenType is a good and powerful standard which masks some of Unicode’s shortcomings doesn’t really help. It’s not too late, and I hope they will find their way back to their original mission and be braver in solving long-standing issues.
- yk 1y ago
  I would. The original sin of Unicode is really their manifold idea, at that point they stopped trying to write a string standard and started to become a kinda general description of how string standards should look like and hopefully string standards that more or less conform to this description are interoperable if you remember which direction "string".decode() and "string".encode() is.
- socalgal2 1y ago
  What could be better? Human languages are complex
  - deleted 1y ago
    [deleted]
  - weinzierl 1y ago
    Yes, exactly, human languages are complex and in my opinion Unicode used to be on a good track to tackle these complexities. I just think that nowadays they are not doing enough to help people around the world solving these problems.
    - pas 1y ago
      can you describe a few examples? what are you missing? or maybe are you aware of something they rejected that would be useful?

estebank 1y ago

The security concerns are those of "Trojan source", where the displayed text doesn't correspond to the bytes on the wire.[1] I don't think a wire protocol should necessarily restrict them, for the sake of compatibility with existing text corpus out there, but a fair observation. 1: https://trojansource.codes/ https://trojansource.codes/

yencabulator 1y ago
The enforcement is an app-level issue, depending on the semantics of the field. I agree it doesn't belong in the low-level transport protocol. The rules for "username", "display name", "biography", "email address", "email body" and "contents of uploaded file with name foo.txt" are not all going to be the same.
- Waterluvian 1y ago
  Can a regular expression be used to restrict Unicode chars like the ones described? I’m imagining a listing of regex rules for the various gotchas, and then a validation-level use that unions the ones you want.
  - fluoridation 1y ago
    Why would you need a regular expression for that? It's just a list of characters.
    - Waterluvian 1y ago
      There’s cases where certain characters coming before or after others is what creates the issue.

arp242 1y ago

I always thought you kind of need those directional control characters to correctly render bidi text? e.g. if you write something in Hebrew but include a Latin word/name (or the reverse).

dcrazy 1y ago
This is the job of the Bidi Algorithm: https://www.unicode.org/reports/tr9/ https://www.unicode.org/reports/tr9/ Of course, this is an “annex”, not part of the core Unicode spec. So in situations where you can’t rely on the presentation layer’s (correct) implementation of the Bidi algorithm, you can fall back to directional override/embedding characters.
- acdha 1y ago
  Over the years I’ve run into a few situations where the rules around neutral characters didn’t produce the right result and so we had to use the override characters to force the correct display. It’s completely a niche but very handy when you are mixing quotes within a complex text.
layer8 1y ago
Read the parent’s link. The characters “to be avoided” are a particular special-purpose subset, not directional control characters in general.

weinzierl 1y ago

I wouldn’t be so harsh. I think the Unicode Consortium not only started with good intentions but also did excellent work for the first decade or so. I just think they got distracted when the problems got harder, and instead of tackling them head-on, they now waste a lot of their resources on busywork - good intentions notwithstanding. Sure, it’s more fun standardizing sparkling disco balls than dealing with real-world pain points. That OpenType is a good and powerful standard which masks some of Unicode’s shortcomings doesn’t really help. It’s not too late, and I hope they will find their way back to their original mission and be braver in solving long-standing issues.

yk 1y ago
I would. The original sin of Unicode is really their manifold idea, at that point they stopped trying to write a string standard and started to become a kinda general description of how string standards should look like and hopefully string standards that more or less conform to this description are interoperable if you remember which direction "string".decode() and "string".encode() is.
socalgal2 1y ago
What could be better? Human languages are complex
- deleted 1y ago
  [deleted]
- weinzierl 1y ago
  Yes, exactly, human languages are complex and in my opinion Unicode used to be on a good track to tackle these complexities. I just think that nowadays they are not doing enough to help people around the world solving these problems.
  - pas 1y ago
    can you describe a few examples? what are you missing? or maybe are you aware of something they rejected that would be useful?