3 ms·

Another fun fact: Upper-casing is language dependent. In English uppercasing 'i' gets you 'I'. But Turkish has a dotted and un-dotted 'i', each with an upperc

by varenc 5y ago

Another fun fact: Upper-casing is language dependent. In English uppercasing 'i' gets you 'I'. But Turkish has a dotted and un-dotted 'i', each with an uppercase variant. So if your user's language was Turkish, uppercasing 'i' would give you 'İ', and lowercasing 'I' would give you 'ı'. Makes me wonder how case insensitive file systems handle this...and for more fun, handle the situation where the user changes the system language. I know that the Turkish 'I' delayed at least one big company's Turkish localization efforts for awhile.

toast0 5y ago
My understanding is that case insensitive filesystems that wish to be portable have a mapping table in the metadata. A quick search showed references to an 'upcase table', although I'm not sure of the accuracy of the source, so I won't link it. Just because the user changed the system language doesn't mean the system should be expected to change the upcase table though. That operation would need to be very carefully managed; you can't change the rules if there are existing files that would conflict in the new mapping. And you might have symbolic links that matched because of case insensitivity that won't anymore... Pretty tricky.
- DemocracyFTW 5y ago
  I think this goes to show that the file system is not the right place to implement natural language handling to its full extent, like it's probably not a good idea to implement time zones as a primitive function in your new programming language.
  - rini17 5y ago
    What is the right place to have two different systems agree on natural language handling, then? And of course there are widely recognized problems with time zones too.
lifthrasiir 5y ago
> Makes me wonder how case insensitive file systems handle this... They generally don't. It is true that several case-insensitive file systems (including NTFS, exFAT and ext4 [1]) maintain some sort of configurable case-folding maps but they are mostly used to guard against the future Unicode update and do not vary across locales. [1] https://dfir.ru/2021/07/15/playing-with-case-insensitive-file-names/ https://dfir.ru/2021/07/15/playing-with-case-insensitive-fil... (ext4 supports an optional case insensitivity)
danielbarla 5y ago
A somewhat surprising and interesting side effect of this can be found in the blog post "Hacking GitHub with Unicode's dotless i" [1], which is now fairly well known. [1] https://eng.getwisdom.io/hacking-github-with-unicode-dotless-i/ https://eng.getwisdom.io/hacking-github-with-unicode-dotless...
fallingknife 5y ago
Wouldn't the 2 Is be different characters than the ASCII I and therefore each be subject to different capitilazation rules?
- simonh 5y ago
  That's correct, but this could be very surprising for anyone not familiar with Turkish. For example if you have an English text which has a Turkish word in it, maybe a place or person name. Or for example when doing OCR.
- DemocracyFTW 5y ago
  This could've been a solution but is not what Unicode chose to do (in this particular case). Also relevant: discussion around unifying glyphs like A, B, C, E and so on across Cyrillic, Latin, and Greek, all with their language-specific casing behavior.
  - int_19h 5y ago
    Coincidentally, Cyrillic I/i is also a thing (and is separately encoded!): https://en.wikipedia.org/wiki/Dotted_I_(Cyrillic)#Computing_codes https://en.wikipedia.org/wiki/Dotted_I_(Cyrillic)#Computing_...
Sharlin 5y ago
Uppercasing is also language dependent in the trivial sense that most scripts in use have no concept of "upper" and "lower" case at all.
TimonKnigge 5y ago
Another example is that in Dutch, the bigram 'ij' is considered a single letter, and so at the beginning of a word, both have to be uppercased. See for example the Dutch Wikipedia page for Iceland: https://nl.wikipedia.org/wiki/IJsland https://nl.wikipedia.org/wiki/IJsland.
- TotempaaltJ 5y ago
  And it's present in Unicode also! Ĳ and ĳ are single characters (try selecting only the i or only the j in those). Their use is discouraged though: > They are included for compatibility and round-trip convertibility with legacy encodings, but their use is discouraged. Therefore, even with Unicode available, it is recommended to encode ij as two separate letters.
AlanYx 5y ago
>Upper-casing is language dependent. It goes beyond just language... uppercasing can also be locale-dependent. In Microsoft Word, for example, uppercasing an "é" gets you "É" in the French (Canada) locale but "E" in the French (France) locale.

by varenc 5y ago

toast0 5y ago

My understanding is that case insensitive filesystems that wish to be portable have a mapping table in the metadata. A quick search showed references to an 'upcase table', although I'm not sure of the accuracy of the source, so I won't link it. Just because the user changed the system language doesn't mean the system should be expected to change the upcase table though. That operation would need to be very carefully managed; you can't change the rules if there are existing files that would conflict in the new mapping. And you might have symbolic links that matched because of case insensitivity that won't anymore... Pretty tricky.

DemocracyFTW 5y ago
I think this goes to show that the file system is not the right place to implement natural language handling to its full extent, like it's probably not a good idea to implement time zones as a primitive function in your new programming language.
- rini17 5y ago
  What is the right place to have two different systems agree on natural language handling, then? And of course there are widely recognized problems with time zones too.

lifthrasiir 5y ago

> Makes me wonder how case insensitive file systems handle this... They generally don't. It is true that several case-insensitive file systems (including NTFS, exFAT and ext4 [1]) maintain some sort of configurable case-folding maps but they are mostly used to guard against the future Unicode update and do not vary across locales. [1] https://dfir.ru/2021/07/15/playing-with-case-insensitive-file-names/ https://dfir.ru/2021/07/15/playing-with-case-insensitive-fil... (ext4 supports an optional case insensitivity)

danielbarla 5y ago

A somewhat surprising and interesting side effect of this can be found in the blog post "Hacking GitHub with Unicode's dotless i" [1], which is now fairly well known. [1] https://eng.getwisdom.io/hacking-github-with-unicode-dotless-i/ https://eng.getwisdom.io/hacking-github-with-unicode-dotless...

fallingknife 5y ago

Wouldn't the 2 Is be different characters than the ASCII I and therefore each be subject to different capitilazation rules?

simonh 5y ago
That's correct, but this could be very surprising for anyone not familiar with Turkish. For example if you have an English text which has a Turkish word in it, maybe a place or person name. Or for example when doing OCR.
DemocracyFTW 5y ago
This could've been a solution but is not what Unicode chose to do (in this particular case). Also relevant: discussion around unifying glyphs like A, B, C, E and so on across Cyrillic, Latin, and Greek, all with their language-specific casing behavior.
- int_19h 5y ago
  Coincidentally, Cyrillic I/i is also a thing (and is separately encoded!): https://en.wikipedia.org/wiki/Dotted_I_(Cyrillic)#Computing_codes https://en.wikipedia.org/wiki/Dotted_I_(Cyrillic)#Computing_...

Sharlin 5y ago

Uppercasing is also language dependent in the trivial sense that most scripts in use have no concept of "upper" and "lower" case at all.

TimonKnigge 5y ago

Another example is that in Dutch, the bigram 'ij' is considered a single letter, and so at the beginning of a word, both have to be uppercased. See for example the Dutch Wikipedia page for Iceland: https://nl.wikipedia.org/wiki/IJsland https://nl.wikipedia.org/wiki/IJsland.

TotempaaltJ 5y ago
And it's present in Unicode also! Ĳ and ĳ are single characters (try selecting only the i or only the j in those). Their use is discouraged though: > They are included for compatibility and round-trip convertibility with legacy encodings, but their use is discouraged. Therefore, even with Unicode available, it is recommended to encode ij as two separate letters.

AlanYx 5y ago

>Upper-casing is language dependent. It goes beyond just language... uppercasing can also be locale-dependent. In Microsoft Word, for example, uppercasing an "é" gets you "É" in the French (Canada) locale but "E" in the French (France) locale.