5 ms·

I sort of expect that nothing can be assumed when talking about strings and characters anymore. Waiting for the post on HN one day that says that its in the uni

I sort of expect that nothing can be assumed when talking about strings and characters anymore. Waiting for the post on HN one day that says that its in the unicode spec that characters can animate or be in superposition until observed…

tshaddox 5y ago
Or that some people’s names can only be represented as a bitmap or vector graphic. Or what if some people’s names can only be represented in a computer by that computer executing arbitrary code? Then all computer software that accepts the input of human names must by definition have arbitrary code execution vulnerabilities!
colanderman 5y ago
> unicode spec that characters can [...] be in superposition until observed You mean like language-dependent unified Han glyphs? [1] [1] https://en.wikipedia.org/wiki/Han_unification#Examples_of_language-dependent_glyphs https://en.wikipedia.org/wiki/Han_unification#Examples_of_la...
- junon 5y ago
  When I read GP that was the first thing I thought of. Animated emojis probably aren't too far off the horizon, and I could definitely see a world where multiple glyphs could be combined to affect the animation much like we do for skin tone or the family emojis. Sometimes I think we've gone too far.
  - phkahler 5y ago
    >> Sometimes I think we've gone too far. Agreed from a technical standpoint. But also dont forget we have the poop emoji and the love-hotel.
    - majou 5y ago
      That's just its Japanese origin.
    - jchw 5y ago
      Emojis in Unicode may have been massive scope creep, but I don’t think the poop emoji or love hotel emoji represent it. Love hotels are a legitimate phenomenon in Japan, where emoji originate (heck, even the word emoji is Japanese.) It fits in well alongside other map icons, even if the cultural clashes can be amusing. And the poop emoji may be silly, but it’s hard to deny its application for expression. Now what I find is less practical and more ridiculous is the sheer number of ways you can combine various codepoints to get different emoji representing families. On one hand, it’s impressive dedication to trying to be neutral and encompassing, on the other hand it is nightmarish just how many possibilities exist.
      - foepys 5y ago
        > It fits in well alongside other map icons, even if the cultural clashes can be amusing. Another example is the omission of the middle finger emoji in earlier Unicode versions but inclusion of various different hand emojis that insult people in East Asia or the Middle East. E.g. OK Hand Sign and Call me hand, which are both completely mislabeled when used in other cultures.
      - chriswarbo 5y ago
        > Now what I find is less practical and more ridiculous is the sheer number of ways you can combine various codepoints to get different emoji representing families. On one hand, it’s impressive dedication to trying to be neutral and encompassing, on the other hand it is nightmarish just how many possibilities exist. My problem with this is that it's trying to solve a font-level problem at the alphabet-level. For example, it's fair to ask why a male-looking construction worker appears on some particular emoji keyboard, whilst a female-looking construction worker doesn't; yet that bias exists in the choice of font, not in the Unicode standard (where U+1F477 simply defines 'construction worker'). Vendors like Apple created this problem when they moved away from simple silhouettes and (AFAIK) neutral 'smiley faces', to more detailed images which required a bunch of arbitrary choices to be made (gender, skin tone, etc.). Rather than making a bunch of alternative fonts, akin to light/bold/serif/monospace/etc. those choices were instead shoehorned into Unicode's modifier-character system :(
  - MengerSponge 5y ago
    Sometimes I think we haven't gone too far enough!
ahefner 5y ago
I'm sometimes believe that full general purpose embracing of unicode for text, with no clean distinction between "machine-friendly" text versus "for human" natural language text (in every script since the dawn of time plus every goofy emoji anyone dreams up, with all the complexity these entail) is a major mistake that has lead computing astray. I fear, though, that it is impractical to separate these things, short of entirely shunning the latter, and tempting as it is I can't quite advocate a return to pure ASCII.
- zionic 5y ago
  I’ll say it, it was a mistake. Every developer who learns swift for the next hundred years will curse the designers the first time they need to do “basic” string processing. It’s gotten slightly better, but it’s still a joke.
  - int_19h 5y ago
    As a developer whose native language is not English, I'm really glad that Swift has finally forced developers from Western countries (and especially US) contemplate what strings actually are and aren't, and how to properly use them in different contexts.
  - bouke 5y ago
    Care to explain? I believe Swift’s model to be quite sensible for human non-US text, having an API that puts Unicode before ASCII. The bytes are still available if you want to shoot yourself in the foot.
- jiggawatts 5y ago
  Programming languages really should have distinct string-like types: identifier -- printable ASCII characters only, an array of 8-bit chars ucs16 -- An array of 16-bit chars for compatibility with Windows, Java, and .NET utf8 -- Normalised, 100% valid UTF-8 with potentially some "reasonableness" constraints text -- Abstraction over arbitrary code pages, including both Unicode and legacy encodings. Languages like Rust kinda-sorta implement this. For example, the PathBuf type internally uses a "WTF8" encoding that is vaguely Utf-8 compatible, but allows the invalid code sequences that can turn up in Win32 system API calls. IMHO that's a good try but not ideal. The back-and-forth conversion is complex, requires temporary buffers, and is slow as molasses for many types of API calls. The ideal would be to have abstractions (traits, interfaces, whatever) that cover all the use-cases. E.g.: it should be possible to test if a 'utf8' string contains an 'identifier' string. It should be possible to compare strings without having to convert their formats. Etc...
hibbelig 5y ago
There are two German words that uppercase to the same: Masse (physical unit, measured in kg usually: mass) and Maße (plural of Maß: measurement). So downcasing MASSE either requires understanding the text or results in a superposition.
- c0l0 5y ago
  These days, we are lucky(?) enough to have ẞ (LATIN CAPITAL LETTER SHARP S) to avoid that particular conundrum ;)
  - ThreeFx 5y ago
    Except that it isn't used widely yet, if at all.
- vikingerik 5y ago
  This happens even in English. Downcasing POLISH could give you either an Eastern European nationality or the action you do to shine a shoe, and so needs textual understanding to disambiguate.

tshaddox 5y ago

Or that some people’s names can only be represented as a bitmap or vector graphic. Or what if some people’s names can only be represented in a computer by that computer executing arbitrary code? Then all computer software that accepts the input of human names must by definition have arbitrary code execution vulnerabilities!

colanderman 5y ago

> unicode spec that characters can [...] be in superposition until observed You mean like language-dependent unified Han glyphs? [1] [1] https://en.wikipedia.org/wiki/Han_unification#Examples_of_language-dependent_glyphs https://en.wikipedia.org/wiki/Han_unification#Examples_of_la...

junon 5y ago
When I read GP that was the first thing I thought of. Animated emojis probably aren't too far off the horizon, and I could definitely see a world where multiple glyphs could be combined to affect the animation much like we do for skin tone or the family emojis. Sometimes I think we've gone too far.
- phkahler 5y ago
  >> Sometimes I think we've gone too far. Agreed from a technical standpoint. But also dont forget we have the poop emoji and the love-hotel.
  - majou 5y ago
    That's just its Japanese origin.
  - jchw 5y ago
    Emojis in Unicode may have been massive scope creep, but I don’t think the poop emoji or love hotel emoji represent it. Love hotels are a legitimate phenomenon in Japan, where emoji originate (heck, even the word emoji is Japanese.) It fits in well alongside other map icons, even if the cultural clashes can be amusing. And the poop emoji may be silly, but it’s hard to deny its application for expression. Now what I find is less practical and more ridiculous is the sheer number of ways you can combine various codepoints to get different emoji representing families. On one hand, it’s impressive dedication to trying to be neutral and encompassing, on the other hand it is nightmarish just how many possibilities exist.
    - foepys 5y ago
      > It fits in well alongside other map icons, even if the cultural clashes can be amusing. Another example is the omission of the middle finger emoji in earlier Unicode versions but inclusion of various different hand emojis that insult people in East Asia or the Middle East. E.g. OK Hand Sign and Call me hand, which are both completely mislabeled when used in other cultures.
    - chriswarbo 5y ago
      > Now what I find is less practical and more ridiculous is the sheer number of ways you can combine various codepoints to get different emoji representing families. On one hand, it’s impressive dedication to trying to be neutral and encompassing, on the other hand it is nightmarish just how many possibilities exist. My problem with this is that it's trying to solve a font-level problem at the alphabet-level. For example, it's fair to ask why a male-looking construction worker appears on some particular emoji keyboard, whilst a female-looking construction worker doesn't; yet that bias exists in the choice of font, not in the Unicode standard (where U+1F477 simply defines 'construction worker'). Vendors like Apple created this problem when they moved away from simple silhouettes and (AFAIK) neutral 'smiley faces', to more detailed images which required a bunch of arbitrary choices to be made (gender, skin tone, etc.). Rather than making a bunch of alternative fonts, akin to light/bold/serif/monospace/etc. those choices were instead shoehorned into Unicode's modifier-character system :(
- MengerSponge 5y ago
  Sometimes I think we haven't gone too far enough!

ahefner 5y ago

I'm sometimes believe that full general purpose embracing of unicode for text, with no clean distinction between "machine-friendly" text versus "for human" natural language text (in every script since the dawn of time plus every goofy emoji anyone dreams up, with all the complexity these entail) is a major mistake that has lead computing astray. I fear, though, that it is impractical to separate these things, short of entirely shunning the latter, and tempting as it is I can't quite advocate a return to pure ASCII.

zionic 5y ago
I’ll say it, it was a mistake. Every developer who learns swift for the next hundred years will curse the designers the first time they need to do “basic” string processing. It’s gotten slightly better, but it’s still a joke.
- int_19h 5y ago
  As a developer whose native language is not English, I'm really glad that Swift has finally forced developers from Western countries (and especially US) contemplate what strings actually are and aren't, and how to properly use them in different contexts.
- bouke 5y ago
  Care to explain? I believe Swift’s model to be quite sensible for human non-US text, having an API that puts Unicode before ASCII. The bytes are still available if you want to shoot yourself in the foot.
jiggawatts 5y ago
Programming languages really should have distinct string-like types: identifier -- printable ASCII characters only, an array of 8-bit chars ucs16 -- An array of 16-bit chars for compatibility with Windows, Java, and .NET utf8 -- Normalised, 100% valid UTF-8 with potentially some "reasonableness" constraints text -- Abstraction over arbitrary code pages, including both Unicode and legacy encodings. Languages like Rust kinda-sorta implement this. For example, the PathBuf type internally uses a "WTF8" encoding that is vaguely Utf-8 compatible, but allows the invalid code sequences that can turn up in Win32 system API calls. IMHO that's a good try but not ideal. The back-and-forth conversion is complex, requires temporary buffers, and is slow as molasses for many types of API calls. The ideal would be to have abstractions (traits, interfaces, whatever) that cover all the use-cases. E.g.: it should be possible to test if a 'utf8' string contains an 'identifier' string. It should be possible to compare strings without having to convert their formats. Etc...

hibbelig 5y ago

There are two German words that uppercase to the same: Masse (physical unit, measured in kg usually: mass) and Maße (plural of Maß: measurement). So downcasing MASSE either requires understanding the text or results in a superposition.

c0l0 5y ago
These days, we are lucky(?) enough to have ẞ (LATIN CAPITAL LETTER SHARP S) to avoid that particular conundrum ;)
- ThreeFx 5y ago
  Except that it isn't used widely yet, if at all.
vikingerik 5y ago
This happens even in English. Downcasing POLISH could give you either an Eastern European nationality or the action you do to shine a shoe, and so needs textual understanding to disambiguate.