5 ms·

Hat tip to Guido van Rossum for passing (nearly) all the tests in Python 3. Is the "ffl-ligature to uppercase" test really relevant? Isn't that fixed by approp

by Pitarou 13y ago

Hat tip to Guido van Rossum for passing (nearly) all the tests in Python 3. Is the "ffl-ligature to uppercase" test really relevant? Isn't that fixed by appropriate use of string normalisation?

andrewcooke 13y ago
[edit:] ah, ok, so on python 3 it depends how it's constructed. [originally i had a post here saying i couldn't get the noel to work on 3.3.2]
- mercurial 13y ago
  Seems to work fine on 3.3.3 (Linux) Python 3.3.3 (default, Nov 23 2013, 09:49:26) [GCC 4.8.2] on linux Type "help", "copyright", "credits" or "license" for more information. >>> a = "noël" >>> len(a) 4 >>> a[::-1] 'lëon'
  - judofyr 13y ago
    This is the test case: https://eval.in/73766 https://eval.in/73766 print(len("noe\u0308l"))
    - mercurial 13y ago
      Ah, I see, the decomposed case indeed doesn't work as well.
- masklinn 13y ago
  The test case is decomposed.
unwind 13y ago
I also doubt the validity of the upper-casing, it feels like in an internationalization/localization context, converting a string to all upper case is not a valid thing to be doing. Not all languages (or even characters) have a well-defined upper-case versions of their glyphs. Even if they all did, I would expect the interpretation by a (human) reader to vary culturally.
- judofyr 13y ago
  The Unicode standard includes uppercase rules. If you're already representing strings using Unicode codepoints, why not follow the whole Unicode standard? EDIT: And yes, different languages can have different uppercasing rules. There's still a standard: https://github.com/lang/unicode_utils/blob/master/data/CaseFolding.txt https://github.com/lang/unicode_utils/blob/master/data/CaseF...
  - unwind 13y ago
    Thanks, I was not aware of that. I guess "uppercase this string" goes from being a tiny loop to a big ... thing based on a lot of hardcoded knowledge, which in turn might indicate that it's not a very simple operation any more.
- XorNot 13y ago
  The usual goal is to apply a consistent transform though, to smooth out interpretation differences - i.e. when looking for command input I either lowercase or uppercase things to smooth over the fact that "yes" "YES" "Yes" are all completely valid ways of saying the same thing with those characters. If there's only one way of expressing the thing - i.e. a single chinese character - then it would be valid to do nothing. It's just in english "y" and "Y" might change context, but as far as computer input is generally concerned they are the same thing.
  - TorKlingberg 13y ago
    For that use case it is better to compare case insensitively with "yes" instead of converting the input to lower case first.
    - zokier 13y ago
      How do you do case-insensitive comparison without normalizing the case of the operands?
      - 13y ago
riquito 13y ago
The ﬄ ligature passes $ python3 Python 3.3.2+ (default, Oct 9 2013, 14:50:09) [GCC 4.8.1] on linux Type "help", "copyright", "credits" or "license" for more information. >>> "baﬄe".upper() 'BAFFLE' Strange that the article claims that no languages passes it. It seems from another post that perl passes it too
- Pitarou 13y ago
  Embarrassing. I should have checked for myself. And it is strange. Maybe the author needs to check his locale settings?

by Pitarou 13y ago

Hat tip to Guido van Rossum for passing (nearly) all the tests in Python 3. Is the "ffl-ligature to uppercase" test really relevant? Isn't that fixed by appropriate use of string normalisation?

andrewcooke 13y ago

[edit:] ah, ok, so on python 3 it depends how it's constructed. [originally i had a post here saying i couldn't get the noel to work on 3.3.2]

mercurial 13y ago
Seems to work fine on 3.3.3 (Linux) Python 3.3.3 (default, Nov 23 2013, 09:49:26) [GCC 4.8.2] on linux Type "help", "copyright", "credits" or "license" for more information. >>> a = "noël" >>> len(a) 4 >>> a[::-1] 'lëon'
- judofyr 13y ago
  This is the test case: https://eval.in/73766 https://eval.in/73766 print(len("noe\u0308l"))
  - mercurial 13y ago
    Ah, I see, the decomposed case indeed doesn't work as well.
masklinn 13y ago
The test case is decomposed.

unwind 13y ago

I also doubt the validity of the upper-casing, it feels like in an internationalization/localization context, converting a string to all upper case is not a valid thing to be doing. Not all languages (or even characters) have a well-defined upper-case versions of their glyphs. Even if they all did, I would expect the interpretation by a (human) reader to vary culturally.

judofyr 13y ago
The Unicode standard includes uppercase rules. If you're already representing strings using Unicode codepoints, why not follow the whole Unicode standard? EDIT: And yes, different languages can have different uppercasing rules. There's still a standard: https://github.com/lang/unicode_utils/blob/master/data/CaseFolding.txt https://github.com/lang/unicode_utils/blob/master/data/CaseF...
- unwind 13y ago
  Thanks, I was not aware of that. I guess "uppercase this string" goes from being a tiny loop to a big ... thing based on a lot of hardcoded knowledge, which in turn might indicate that it's not a very simple operation any more.
XorNot 13y ago
The usual goal is to apply a consistent transform though, to smooth out interpretation differences - i.e. when looking for command input I either lowercase or uppercase things to smooth over the fact that "yes" "YES" "Yes" are all completely valid ways of saying the same thing with those characters. If there's only one way of expressing the thing - i.e. a single chinese character - then it would be valid to do nothing. It's just in english "y" and "Y" might change context, but as far as computer input is generally concerned they are the same thing.
- TorKlingberg 13y ago
  For that use case it is better to compare case insensitively with "yes" instead of converting the input to lower case first.
  - zokier 13y ago
    How do you do case-insensitive comparison without normalizing the case of the operands?
    - 13y ago

riquito 13y ago

The ﬄ ligature passes $ python3 Python 3.3.2+ (default, Oct 9 2013, 14:50:09) [GCC 4.8.1] on linux Type "help", "copyright", "credits" or "license" for more information. >>> "baﬄe".upper() 'BAFFLE' Strange that the article claims that no languages passes it. It seems from another post that perl passes it too

Pitarou 13y ago
Embarrassing. I should have checked for myself. And it is strange. Maybe the author needs to check his locale settings?