5 ms·

I'm coming from a German background and I sympathize with the author. German has 4 (7 if you consider cases) non-ASCII characters: äüöß(and upper-case umlauts)

I'm coming from a German background and I sympathize with the author. German has 4 (7 if you consider cases) non-ASCII characters: äüöß(and upper-case umlauts). All of these are unique, well-defined codepoints. That's not related to composing on a keyboard. In fact, although I'm German I'm using the US keyboard layout and HAD to compose these characters now. But I wouldn't need to and the result is a single codepoint again..

masklinn 12y ago
> But I wouldn't need to and the result is a single codepoint again.. Doesn't have to be though, it'd be perfectly correct for an IME to generate multiple codepoints. IIRC, that's what you'd get if you typed those in a filename on OSX then asked for the native file path, as HFS+ stores filenames in NFD. Meanwhile Safari does (used to do?) the opposite, text is automatically NFC'd before sending. Things get interesting when you don't expect it and don't do unicode-equivalent comparisons.
vilhelm_s 12y ago
Well, they have codepoints, but not unique ones (since they can be written both using combining characters or using the compatibility pre-combined form). Software libraries dealing with unicode strings needs to handle both versions, by applying unicode normalization before doing comparisons. The reason they have two representations is for backwards compatibility with previous character encoding standards, but the unicode standard is more complex because of this (it needs to specify more equivalences for normalization). I guess for languages which were not previously covered by any standards, the unicode consortium tries to represent things "as uniquely as possible".
the_mitsuhiko 12y ago
> German has 4 (7 if you consider cases) non-ASCII characters: äüöß(and upper-case umlauts). All of these are unique, well-defined codepoints. German does not consider "ä", "ö" and "ü" letters. Our alphabet has 26 letters none of which are the ones you mentioned. In fact, if you go back in History it becomes even clearer that those letters used to be ligatures in writing. They still are collated as the basic letters the represent, even if they sound different. That we use the uncomposed representation in Unicode usually, is merely a historical artifact because of iso-8859-1 and others, not because it logically makes sense. When you used an old typewriter you usually did not have those keys either, you composed them.
- darklajid 12y ago
  One by one: I'm confused by your use of 'our' and 'we'. It seems you're trying to write from the general point of view of a German, answering .. a German? Are umlauts letters? Yes. [1] [2] Maybe not the best source, but please provide a better one if you disagree so that I can actually understand where you're coming from. I understand - I hope? - composition. And I tend to agree that it shouldn't matter much if the input just works. If I press a key labeled ü and that letter shows up on the screen, I shouldn't really care if that is one codepoint or a composition of two (or more). I do think that the history you mention is an indicator that supports the author's argument. There IS a codepoint for ü (painful to type..). For 'legacy reasons' perhaps. And it feels to me that non-ASCII characters - for legacy reasons or whatever - have better support than the ones he is complaining about, if they originate in western Europe/in my home country. Typewriters and umlauts: http://i.ebayimg.com/00/s/Mzk2WDQwMA==/$T2eC16N,!)sE9swmYlFPBP8DdJ8zFQ~~48_20.JPG http://i.ebayimg.com/00/s/Mzk2WDQwMA==/$T2eC16N,!)sE9swmYlFP... (basically I searched for old typewriter models, 'Adler Schreibmaschinen' results in lots of hits like that). Note the separate umlaut keys. And these are typewriters from .. the 60s? Maybe?) 1: https://de.wikipedia.org/wiki/Alphabet https://de.wikipedia.org/wiki/Alphabet 2: https://de.wikipedia.org/wiki/Deutsches_Alphabet https://de.wikipedia.org/wiki/Deutsches_Alphabet
  - ptaipale 12y ago
    I am not entirely sure if Germans count umlauts as distinct characters or modified versions of the base character. And maybe it is not so important; they still do deserve their own code points. Note BTW that in e.g. Swedish and German alphabets, there are some overlapping non-ASCII characters (ä, ö) and some that are distinct to each language (å, ü). It is important that the Swedish ä and German ä are rendered to the same code point and same representation in files; this way I can use a computer localised for Swedish and type German text. Only when I need to type ü I need to compose it from ¨ and u, while ä and ö are right on the keyboard. The German alphabetical order supports the idea that umlauts are not so distinct from their bases: it is AÄBCDEFGHIJKLMNOÖPQRSßTUÜVWXYZ while the Swedish/Finnish one is ABCDEFGHIJKLMNOPQRSTUVWXYZÅÄÖ This has the obvious impacts on sorting order. BTW, traditionally Swedish/Finnish did not distinguish between V and W in sorting, thus a correct sorting order would be Vasa Westerlund Vinberg Vårdö - the W drops right in the middle, it's just an older way to write V. And Vå... is at the end of section V, while Va... is at the start.
1ris 12y ago
8 letters actually. 'ẞ' was added quite a while later.
- darklajid 12y ago
  Agreed, it exists. But then again, most systems in use today (as far as I'm aware) would turn a ß into SS, not ẞ. Actually I think I've never seen a ẞ in use, ever. Not once. Now I'm running around testing 'Try $programmingLanguage' services on the net. Try Clojure for example: > (.toUpperCase "ß") "SS"
  - 1ris 12y ago
    In Haskell: isLower $ toUpper 'ß' is True. I wonder how many security holes this unexpected behaviour causes.
    - darklajid 12y ago
      .Net seems to do the same thing, Javascript (according to jsfiddle) as well. So maybe this is more widespread than I thought (again - I have never seen that character in the wild)? Java (as in Try Clojure) seems to do the 'expected' SS thing. Trying the golang playground I get even worse: fmt.Println(strings.ToUpper("ßẞ")) returns ßẞ (yeah, unchanged?) So, while I agree that you're technically correct (ẞ exists!) I do stick to my ~7 letters list for now.. It seems that's both realistic and usable.
      - e12e 12y ago
        I think this is more related to the fact that there aren't many sane libraries implementing unicode and locales -- so you'll get either some c lib/c++ lib, system lib, java lib -- or an actual new implementation that's actually been done "seriously" -- as part of being able to say: "Yes, X does actually support unicode strings.". Python3 got a lot of flac for the decision to break away from it's byte sequences, to it's a unicode string. But I think that was the right choice. I still understand why people writing software that only cared about network, on-the-wire, pretend-to-be-text type strings. Then again, based on some other comments here, apparently there are still some dark corners: Python 3.2.3 (default, Feb 20 2013, 14:44:27) [GCC 4.7.2] on linux2 >>> s="Åßẞ" >>> s == s.upper().lower() False >>> s.lower() 'åßß' However, to complicate things: Python 3.4.2 (default, Dec 27 2014, 13:16:08) [GCC 4.9.2] on linux >>> s="Åßẞ" >>> s.lower() 'åßß' >>> s.lower().upper() 'ÅSSSS' >>> s == s.lower().upper() False >>> s.lower().upper() == 'ÅSSSS' True >>> 'SS'.lower() 'ss' >>> 'ß'.lower() 'ß' >>> 'ß'.lower().upper() 'SS' >>> 'ß'.lower().upper().lower() 'ss' So that's fun.

masklinn 12y ago

> But I wouldn't need to and the result is a single codepoint again.. Doesn't have to be though, it'd be perfectly correct for an IME to generate multiple codepoints. IIRC, that's what you'd get if you typed those in a filename on OSX then asked for the native file path, as HFS+ stores filenames in NFD. Meanwhile Safari does (used to do?) the opposite, text is automatically NFC'd before sending. Things get interesting when you don't expect it and don't do unicode-equivalent comparisons.

vilhelm_s 12y ago

Well, they have codepoints, but not unique ones (since they can be written both using combining characters or using the compatibility pre-combined form). Software libraries dealing with unicode strings needs to handle both versions, by applying unicode normalization before doing comparisons. The reason they have two representations is for backwards compatibility with previous character encoding standards, but the unicode standard is more complex because of this (it needs to specify more equivalences for normalization). I guess for languages which were not previously covered by any standards, the unicode consortium tries to represent things "as uniquely as possible".

the_mitsuhiko 12y ago

> German has 4 (7 if you consider cases) non-ASCII characters: äüöß(and upper-case umlauts). All of these are unique, well-defined codepoints. German does not consider "ä", "ö" and "ü" letters. Our alphabet has 26 letters none of which are the ones you mentioned. In fact, if you go back in History it becomes even clearer that those letters used to be ligatures in writing. They still are collated as the basic letters the represent, even if they sound different. That we use the uncomposed representation in Unicode usually, is merely a historical artifact because of iso-8859-1 and others, not because it logically makes sense. When you used an old typewriter you usually did not have those keys either, you composed them.

darklajid 12y ago
One by one: I'm confused by your use of 'our' and 'we'. It seems you're trying to write from the general point of view of a German, answering .. a German? Are umlauts letters? Yes. [1] [2] Maybe not the best source, but please provide a better one if you disagree so that I can actually understand where you're coming from. I understand - I hope? - composition. And I tend to agree that it shouldn't matter much if the input just works. If I press a key labeled ü and that letter shows up on the screen, I shouldn't really care if that is one codepoint or a composition of two (or more). I do think that the history you mention is an indicator that supports the author's argument. There IS a codepoint for ü (painful to type..). For 'legacy reasons' perhaps. And it feels to me that non-ASCII characters - for legacy reasons or whatever - have better support than the ones he is complaining about, if they originate in western Europe/in my home country. Typewriters and umlauts: http://i.ebayimg.com/00/s/Mzk2WDQwMA==/$T2eC16N,!)sE9swmYlFPBP8DdJ8zFQ~~48_20.JPG http://i.ebayimg.com/00/s/Mzk2WDQwMA==/$T2eC16N,!)sE9swmYlFP... (basically I searched for old typewriter models, 'Adler Schreibmaschinen' results in lots of hits like that). Note the separate umlaut keys. And these are typewriters from .. the 60s? Maybe?) 1: https://de.wikipedia.org/wiki/Alphabet https://de.wikipedia.org/wiki/Alphabet 2: https://de.wikipedia.org/wiki/Deutsches_Alphabet https://de.wikipedia.org/wiki/Deutsches_Alphabet
- ptaipale 12y ago
  I am not entirely sure if Germans count umlauts as distinct characters or modified versions of the base character. And maybe it is not so important; they still do deserve their own code points. Note BTW that in e.g. Swedish and German alphabets, there are some overlapping non-ASCII characters (ä, ö) and some that are distinct to each language (å, ü). It is important that the Swedish ä and German ä are rendered to the same code point and same representation in files; this way I can use a computer localised for Swedish and type German text. Only when I need to type ü I need to compose it from ¨ and u, while ä and ö are right on the keyboard. The German alphabetical order supports the idea that umlauts are not so distinct from their bases: it is AÄBCDEFGHIJKLMNOÖPQRSßTUÜVWXYZ while the Swedish/Finnish one is ABCDEFGHIJKLMNOPQRSTUVWXYZÅÄÖ This has the obvious impacts on sorting order. BTW, traditionally Swedish/Finnish did not distinguish between V and W in sorting, thus a correct sorting order would be Vasa Westerlund Vinberg Vårdö - the W drops right in the middle, it's just an older way to write V. And Vå... is at the end of section V, while Va... is at the start.

1ris 12y ago

8 letters actually. 'ẞ' was added quite a while later.

darklajid 12y ago
Agreed, it exists. But then again, most systems in use today (as far as I'm aware) would turn a ß into SS, not ẞ. Actually I think I've never seen a ẞ in use, ever. Not once. Now I'm running around testing 'Try $programmingLanguage' services on the net. Try Clojure for example: > (.toUpperCase "ß") "SS"
- 1ris 12y ago
  In Haskell: isLower $ toUpper 'ß' is True. I wonder how many security holes this unexpected behaviour causes.
  - darklajid 12y ago
    .Net seems to do the same thing, Javascript (according to jsfiddle) as well. So maybe this is more widespread than I thought (again - I have never seen that character in the wild)? Java (as in Try Clojure) seems to do the 'expected' SS thing. Trying the golang playground I get even worse: fmt.Println(strings.ToUpper("ßẞ")) returns ßẞ (yeah, unchanged?) So, while I agree that you're technically correct (ẞ exists!) I do stick to my ~7 letters list for now.. It seems that's both realistic and usable.
    - e12e 12y ago
      I think this is more related to the fact that there aren't many sane libraries implementing unicode and locales -- so you'll get either some c lib/c++ lib, system lib, java lib -- or an actual new implementation that's actually been done "seriously" -- as part of being able to say: "Yes, X does actually support unicode strings.". Python3 got a lot of flac for the decision to break away from it's byte sequences, to it's a unicode string. But I think that was the right choice. I still understand why people writing software that only cared about network, on-the-wire, pretend-to-be-text type strings. Then again, based on some other comments here, apparently there are still some dark corners: Python 3.2.3 (default, Feb 20 2013, 14:44:27) [GCC 4.7.2] on linux2 >>> s="Åßẞ" >>> s == s.upper().lower() False >>> s.lower() 'åßß' However, to complicate things: Python 3.4.2 (default, Dec 27 2014, 13:16:08) [GCC 4.9.2] on linux >>> s="Åßẞ" >>> s.lower() 'åßß' >>> s.lower().upper() 'ÅSSSS' >>> s == s.lower().upper() False >>> s.lower().upper() == 'ÅSSSS' True >>> 'SS'.lower() 'ss' >>> 'ß'.lower() 'ß' >>> 'ß'.lower().upper() 'SS' >>> 'ß'.lower().upper().lower() 'ss' So that's fun.