5 ms·

Ive long thought programming languages need a "localizable string" (Aka user-facing string) type, different from regular utf8 strings. Something like what gette

Ive long thought programming languages need a "localizable string" (Aka user-facing string) type, different from regular utf8 strings. Something like what gettext and other i18n libraries fake for you, but native to the language. Behaviour like this is definitely a good reason why: sorting, changing case, etc should be consistent when dealing with strings used as constants and identifiers, but Python's .lower() behaviour makes sense in a localizable string context.

DougBTX 6y ago
Along the lines of this? https://docs.microsoft.com/en-us/dotnet/api/system.globalization.cultureinfo.invariantculture?view=netcore-3.1 https://docs.microsoft.com/en-us/dotnet/api/system.globaliza...
- wongarsu 6y ago
  .NET is one of the few ecoecosystems to get this right. It offers the invariant culture for identifier-like things, "fr" for French language and "fr-FR" for French language in France, allowing you to specify your intention to every string-modifying function. Support at the type level would be a lot less verbose, but support at the function level is already much better than many other popular languages.
  - kanox 6y ago
    It would be great if strings and especially date-time values always carried locale and timezone information with them. It would take slightly more memory but not significant on modern machines.
    - wongarsu 6y ago
      Putting the locale information on the string sounds like a good idea. However I'm not sure how that should handle combined strings with components from different locales. For example `logLevel + ": " + logMessage` might produce "info: bağlantı kesildi" in Turkish. How to annotate that? Neither English nor Turkish would work correctly, each would produce the wrong result when uppercasing. You could treat it as a series of string slices with different locales `[("info", "en"), (": ", ""), ("bağlantı kesildi", "tr")]`. That would work correctly, and you could now uppercase each slice according to its appropriate locale, but it wouldn't really be low overhead anymore. Maybe still worth it. It would be an interesting approach that might even be able to be implemented pretty seamlessly as a library in some languages (C++ or rust for example)
- layer8 6y ago
  In Java, there is Locale.ROOT, which can be used in a similar way. In particular, it is useful when performing locale-dependent operations in locale-independent contexts (e.g. working with case-insensitive identifiers) where you don’t want the behavior of your code to depend on the current default locale.
- scrollaway 6y ago
  That just seems to be a parameter for locale-dependent functions. Very useful, but no, I'm talking about splitting the unicode-string datatype in two: "user-facing unicode string" vs "internal unicode string". Example: logging.log("INFO", i"This is a localizable string") In the i18n world, we could gather i-strings just like gettext does (where it looks like `logging.log("INFO", _("This is a localizable string")`). The language could then have other useful hooks/behaviours into that datatype, and definitely one of them would be whether various methods have i18n behaviour enabled on them, versus using a C locale.
lazulicurio 6y ago
Along similar lines, I've thought that it would be useful if Unicode included language marks (i.e. codepoints to identify blocks of text as being written in a specific language). It would be strictly more useful than the barebones left-to-right/right-to-left marks (U+200E/U+200F) when deciding how to process and display text. And it would be a step towards correcting the mess that was Han unification.
- Ericson2314 6y ago
  What this gets right down to is that Unicode is a flawed idea: the meaning/behavior/whatever of characters is insanely dependent on their context. The problem was never gazillions of code pages, but our inability to write C to deal with that amount of complexity circa 1990. With modern machines, and good programming languages with good type systems, I absolutely think we could store a language per string, and concatenate into a polylinguistic rope if needed. This would hopefully push us away from stringly-typed crap in general.
  - throwaway_pdp09 6y ago
    > the meaning/behavior/whatever of characters is insanely dependent on their context I wish you would give an example instead of just proclaiming crapness. You know, so we n00bs can learn something.
    - lazulicurio 6y ago
      How about: case folding for the letter 'I' is dependent on whether the locale is Turkish or not. ;)
    - toast0 6y ago
      Different languages have different rules for change case (as seen here) or what to do when translitterating to 7-bit ascii, in French, you can mostly drop accents if you need to, in German, you need to transform an umlaut to an e following the vowel. Of course, many languages don't have a way to translitterate to 7-bit ascii. Sorting of strings is language dependent, but I don't know that there's a defined order for mixed language lists, so I guess user's context works if you're sorting for user purposes, but if you're sorting for machine purposes, you better not use the locale aware sort without telling it a hardcoded locale that doesn't change between localization library versions.
thomasahle 6y ago
That would be great! For example, in Python you currently have to do something like this import locale sorted(list_of_strings, key=locale.strxfrm) To sort using the current loacale, which many people forget.

DougBTX 6y ago

Along the lines of this? https://docs.microsoft.com/en-us/dotnet/api/system.globalization.cultureinfo.invariantculture?view=netcore-3.1 https://docs.microsoft.com/en-us/dotnet/api/system.globaliza...

wongarsu 6y ago
.NET is one of the few ecoecosystems to get this right. It offers the invariant culture for identifier-like things, "fr" for French language and "fr-FR" for French language in France, allowing you to specify your intention to every string-modifying function. Support at the type level would be a lot less verbose, but support at the function level is already much better than many other popular languages.
- kanox 6y ago
  It would be great if strings and especially date-time values always carried locale and timezone information with them. It would take slightly more memory but not significant on modern machines.
  - wongarsu 6y ago
    Putting the locale information on the string sounds like a good idea. However I'm not sure how that should handle combined strings with components from different locales. For example `logLevel + ": " + logMessage` might produce "info: bağlantı kesildi" in Turkish. How to annotate that? Neither English nor Turkish would work correctly, each would produce the wrong result when uppercasing. You could treat it as a series of string slices with different locales `[("info", "en"), (": ", ""), ("bağlantı kesildi", "tr")]`. That would work correctly, and you could now uppercase each slice according to its appropriate locale, but it wouldn't really be low overhead anymore. Maybe still worth it. It would be an interesting approach that might even be able to be implemented pretty seamlessly as a library in some languages (C++ or rust for example)
layer8 6y ago
In Java, there is Locale.ROOT, which can be used in a similar way. In particular, it is useful when performing locale-dependent operations in locale-independent contexts (e.g. working with case-insensitive identifiers) where you don’t want the behavior of your code to depend on the current default locale.
scrollaway 6y ago
That just seems to be a parameter for locale-dependent functions. Very useful, but no, I'm talking about splitting the unicode-string datatype in two: "user-facing unicode string" vs "internal unicode string". Example: logging.log("INFO", i"This is a localizable string") In the i18n world, we could gather i-strings just like gettext does (where it looks like `logging.log("INFO", _("This is a localizable string")`). The language could then have other useful hooks/behaviours into that datatype, and definitely one of them would be whether various methods have i18n behaviour enabled on them, versus using a C locale.

lazulicurio 6y ago

Along similar lines, I've thought that it would be useful if Unicode included language marks (i.e. codepoints to identify blocks of text as being written in a specific language). It would be strictly more useful than the barebones left-to-right/right-to-left marks (U+200E/U+200F) when deciding how to process and display text. And it would be a step towards correcting the mess that was Han unification.

Ericson2314 6y ago
What this gets right down to is that Unicode is a flawed idea: the meaning/behavior/whatever of characters is insanely dependent on their context. The problem was never gazillions of code pages, but our inability to write C to deal with that amount of complexity circa 1990. With modern machines, and good programming languages with good type systems, I absolutely think we could store a language per string, and concatenate into a polylinguistic rope if needed. This would hopefully push us away from stringly-typed crap in general.
- throwaway_pdp09 6y ago
  > the meaning/behavior/whatever of characters is insanely dependent on their context I wish you would give an example instead of just proclaiming crapness. You know, so we n00bs can learn something.
  - lazulicurio 6y ago
    How about: case folding for the letter 'I' is dependent on whether the locale is Turkish or not. ;)
  - toast0 6y ago
    Different languages have different rules for change case (as seen here) or what to do when translitterating to 7-bit ascii, in French, you can mostly drop accents if you need to, in German, you need to transform an umlaut to an e following the vowel. Of course, many languages don't have a way to translitterate to 7-bit ascii. Sorting of strings is language dependent, but I don't know that there's a defined order for mixed language lists, so I guess user's context works if you're sorting for user purposes, but if you're sorting for machine purposes, you better not use the locale aware sort without telling it a hardcoded locale that doesn't change between localization library versions.

thomasahle 6y ago

That would be great! For example, in Python you currently have to do something like this import locale sorted(list_of_strings, key=locale.strxfrm) To sort using the current loacale, which many people forget.