8 ms·

A years-long Turkish alphabet bug in the Kotlin compiler

161 points by Bogdanp 1y ago

carstenhag 1y ago
I was scrolling and scrolling, waiting for the author to mention the new methods, which of course every Android Dev had to migrate to at some point. And 99% of us probably thought how annoying this change is, even though it probably reduced the number of bugs for Turkish users :) Unrelated, but a month ago I found a weird behaviour where in a kotlin scratch file, `List.isEmpty()` is always true. Questioned my sanity for at least an hour there... https://youtrack.jetbrains.com/issue/KTIJ-35551/ https://youtrack.jetbrains.com/issue/KTIJ-35551/
- ajkjk 1y ago
  well now I wanna know what's going on there!
johnyzee 1y ago
Ugh, I've had the exact same problem in a Java project, which meant I had to go through thousands and thousands of lines of code and make sure that all 'toLowerCase()' on enum names included Locale.ENGLISH as parameter. As the article demonstrates, the error manifests in a completely inscrutable way. But once I saw the bug from a couple of users with Turkish sounding names, I zeroed in on it. And cursed a few times under my breath whoever messed up that character table so bad.
- nradov 1y ago
  Were you not using static analysis tools? All of the popular ones will warn about that issue with locales.
  - lucumo 1y ago
    They do. But a generic warning about locale-dependence doesn't really tell you that ASCII-strings will be broken. For nearly every purpose ASCII is the same in every locale. If you have a string that is guaranteed to be ASCII (like an enum constant is in most code styles), it's easy to think "not a problem here" and move on.
mikestew 1y ago
When I saw "Turkish alphabet bug", I just knew it was some version of toLower() gone horribly wrong. (I'm sure there's a good reason, but I find it odd that compiler message tags are invariably uppercase, but in this problem code they lowercased it to go do a lookup from an enum of lowercase names. Why isn't the enum uppercase, like the things you're going to lookup?)
- thaumasiotes 1y ago
  > Why isn't the enum uppercase, like the things you're going to lookup? Another question: why does the log record the string you intended to look up, instead of the string you actually did look up?
- kevin_thibedeau 1y ago
  With Turkish you can't safely case-fold with toupper() or tolower() in a C/US locale: i->I and I->i are both wrong. Uppercasing wouldn't work. You have to use Unicode or Latin-5 to manage it.
  - oneshtein 1y ago
    It's the bug in the Turkish locale. They hacked Latin alphabet instead of creating a separate letter with separate rules.
  - teo_zero 1y ago
    You misunderstood the parent post. They where suggesting to look up the exact string that ends up in the message, without any conversion. So if the message contains INFO, ERROR, etc. then look up "INFO", "ERROR"...
- kokada 1y ago
  Without looking at the source code I think it is because the log functions are lowercase, but I am not sure this is the reason.
charcircuit 1y ago
Everyone who has used Java has hit this before. Java really should force people to always specify the locale and get rid of the versions of the functions without locale parameters. There is so much hidden broken code out there.
- Uvix 1y ago
  That only helps if devs specify an invariant locale (ROOT for Java) where needed. In practice, I think you'll see devs blindly using using the user's current locale like it silently does today.
  - jeroenhd 1y ago
    The invariant locale can't parse the numbers I enter (my locale uses comma as a decimal separator). More than a few applications will reject perfectly valid numbers. Intel's driver control panel was even so fucked up that I needed to change my locale to make it parse its own UI layout files. Defaulting to ROOT makes a lot of sense for internal constants, like in the example in this article, but defaulting to ROOT for everything just exposes the problems that caused Sun to use the user locale by default in the first place.
    - Uvix 1y ago
      Agreed, there are cases where user locale is needed. So many so that I expect that to be devs’ default if required to specify, and that they won’t use ROOT where they should.
zettabomb 1y ago
I have always wondered why Turkey chose to Latinize in this way. I understand that the issue is having two similar vowels in Turkish, but not why they decided to invent the dotless I, when other diacritics already existed. Ĭ Î Ï Í Ì Į Ĩ and almost certainly a dozen other would've worked, unless there was already some significance to the dot in Turkish that's not obvious.
- mrighele 1y ago
  The issue is not the invention of the dotless I, it already exists, the issue is that the took a vowerl , i/I, and the assigned the lower case to one vowel, and the upper case to a different one, and invented what left missing. It's like they decided that the uppercase of "a" is "E" and the uppercase of "e" is "A".
  - pinkmuffinere 1y ago
    This is misleading, because it assumes that i/I naturally represent one vowel, which is just not the case. i/I represents one vowel in _English_, when written with a latin script. ̶I̶n̶ ̶f̶a̶c̶t̶ ̶e̶v̶e̶n̶ ̶t̶h̶i̶s̶ ̶i̶s̶n̶'̶t̶ ̶c̶o̶r̶r̶e̶c̶t̶,̶ ̶i̶/̶I̶ ̶r̶e̶p̶r̶e̶s̶e̶n̶t̶s̶ ̶o̶n̶e̶ ̶p̶h̶o̶n̶e̶m̶e̶,̶ ̶n̶o̶t̶ ̶o̶n̶e̶ ̶v̶o̶w̶e̶l̶.̶ <see troad's comment for correction> There is no reason to assume that the English representation is in general "correct", "standard", or even "first". The modern script for Turkish was adopted around the 1920's, so you could argue perhaps that most typewriters presented a standard that should have been followed. However, there was variation even between different typewriters, and I strongly suspect that typewriters weren't common in Turkey when the change was made.
    - ginko 1y ago
      >This is misleading, because it assumes that i/I naturally represent one vowel, which is just not the case. It does in literally any language using a latin alphabet other than Turkish.
      - pinkmuffinere 1y ago
okanat 1y ago
As a Turkish speaker who was using a Turkish-locale setup in my teenage years these kinds of bugs frustrated me infinitely. Half of the Java or Python apps I installed never run. My PHP webservers always had problems with random software. Ultimately, I had to change my system's language to English. However, US has godawful standards for everything: dates, measurement units, paper sizes. When I shared computers with my parents I had to switch languages back-and-forth all the time. This helped me learn English rather quickly but, I find it a huge accessibility and software design issue. If your program depends on letter cases, that is a badly designed program, period. If a language ships toUpper or a toLower function without a mandatory language field, it is badly designed too. The only slightly-better option is making toUpper and toLower ASCII-only and throwing error for any other character set. While half of the language design of C is questionable and outright dangerous, making its functions locale-sensitive by all popular OSes was an avoidable mistake. Yet everybody did that. Just the existence of this behavior is a reason I would like to get rid of anything GNU-based in the systems I develop today. I don't care if Unicode releases a conversion map. Natural-language behavior should always require natural language metadata too. Even modern languages like Rust did a crappy job of enforcing it: https://doc.rust-lang.org/std/primitive.char.html#method.to_uppercase https://doc.rust-lang.org/std/primitive.char.html#method.to_... . Yes it is significantly safer but converting 'ß' to 'SS' in German definitely has gotchas too.
- arccy 1y ago
  use Australian English: English but with same settings for everything else, including keyboard layout
  - Sesse__ 1y ago
    Many Linux distributions provide en_DK specifically for this purpose. English as it is used in Denmark. :-)
    - fph 1y ago
      Denmark doesn't have Euros as currency, unfortunately.
darkhorn 1y ago
Java; write once, run anywhere, except on Turkish Windows.
sjrd 1y ago
I am one of the maintainers is the Scala compiler, and this is one of the things that immediately jump to me when I review code that contains any casing operation. Always explicitly specify the locale. However, unlike TFA and other comments, I don't suggest `Locale.US`. That's a little too US-centric. The canonical locale is in fact `Locale.ROOT`. Granted, in practice it's equivalent, but I find it a little bit more sensible. Also, this is the last remaining major system-dependent default in Java. They made strict floating point the default in 17; UTF-8 the default encoding some versions later (21?); only the locale remains. I hope they make ROOT the default in an upcoming version. FWIW, in the Scala.js implementation, we've been using UTF-8 and ROOT as the defaults forever.
- JuniperMesos 1y ago
  > However, unlike TFA and other comments, I don't suggest `Locale.US`. That's a little too US-centric. The canonical locale is in fact `Locale.ROOT`. Granted, in practice it's equivalent, but I find it a little bit more sensible. I have no idea what `Locale.ROOT` refers to, and I'd be worried that it's accidentally the same as the system locale or something, exactly the sort of thing that will unexpectedly change when a Turkish-speaker uses a computer or what have you.
  - kevin_thibedeau 1y ago
    It is a programming language agnostic equivalent of POSIX C locale with Unicode enhancement.
  - layer8 1y ago
    > I'd be worried that it's accidentally the same as the system locale or something The API docs clearly specify that Locale.ROOT “is regarded as the base locale of all locales, and is used as the language/country neutral locale for the locale sensitive operations.”
  - troad 1y ago
    > However, unlike TFA and other comments, I don't suggest `Locale.US`. That's a little too US-centric. The canonical locale is in fact `Locale.ROOT`. Granted, in practice it's equivalent, but I find it a little bit more sensible. Isn't it kind of strange to say that Locale.US is too US centric, and therefore we'll invent a new, fictitious locale, the contents of which is all the US defaults, but which we'll call "the base locale of all locales"? That somehow seems even more US centric to me than just saying Locale.US. Setting the locale as Locale.US is at least comprehensible at a glance.
deleted 1y ago
[deleted]
esafak 1y ago
Kotlin keywords should be assumed to be English.
- pimlottc 1y ago
  Logging levels are not language keywords.
  - DamonHD 1y ago
    The implied locale of those logging levels is (US) English though. And so any recasing of them should be in that locale.
naniwaduni 1y ago
A stark reminder that all operations on strings are wrong.
- lifthrasiir 1y ago
  Or that strings are not human texts.
  - oneshtein 1y ago
    Kotlin is not for humans.
- hshdhdhehd 1y ago
  And all code is operations on strings. (The code starts out as a string).
liquidpele 1y ago
It’s always Turkish lol. That was our language of choice to QA anything… if it worked on that it would pretty much work on anything.
- ramses0 1y ago
  I'm shocked there's no mention of "The Turkey Test" https://blog.codinghorror.com/whats-wrong-with-turkey/ https://blog.codinghorror.com/whats-wrong-with-turkey/
  - OptionOfT 1y ago
    I really dislike the fact that that article uses the “” quotes instead of "". Makes copy-pasting not work.
emmelaich 1y ago
Could have been worse -- Ramazan Çalçoban sent his estranged wife Emine the text message: Zaten sen sıkışınca konuyu değiştiriyorsun. "Anyhow, whenever you can't answer an argument, you change the subject." Unfortunately, what she thought he wrote was: Zaten sen sikişınce konuyu değiştiriyorsun. "Anyhow, whenever they are fucking you, you change the subject." This led to a fight in which the woman was stabbed and died and the man committed suicide in prison. https://gizmodo.com/a-cellphones-missing-dot-kills-two-people-puts-three-m-382026 https://gizmodo.com/a-cellphones-missing-dot-kills-two-peopl...
- esafak 1y ago
  > sikişınce The last one should be an i too :)
flexagoon 1y ago
Wouldn't at least the first issue be solved by using Unicode case folding instead of lowercase? Python, for example, has separate .casefold() and .lower() methods, and AFAIK casefold would always turn I into i, and is much more appropriate for this use case.
- rhdunn 1y ago
  There are 3 types of case folding: 1. Simple one-to-one mappings -- E.g. `T` to `t`. These are typically the ones handled by `lower()` or similar methods as they work on single characters so can modify a string in place (the length of the string doesn't change). 2. More complex one-to-many mappings -- E.g. German `ß` to `ss`. These are covered by functions like `casefold()`. You can't modify the string in place so the function needs to always write to a new string buffer. 3. Locale-specific mappings -- This is what this bug is about. In Turkish `I` maps to `ı` whereas other languages/locales it maps to `i`. You can only implement this by passing the locale to the case function, irrespective of whether you are also doing (1) or (2).
  - zahlman 1y ago
    This is not quite right, at least for Python. .upper() and .lower() (and .casefold() as well) implement the default casing algorithms from the Unicode specification, which are one-to-many (but still locale-naive). Other languages, meanwhile, might well implement locale-aware mapping that defaults to the system locale rather than requiring a locale to be passed.
    - rhdunn 1y ago
      For one-to-one I was thinking of the Unicode Character Database single-character upper, lower, and title case entries in the UnicodeData.txt file [1] -- i.e. the simple case mappings [2]. The one-to-many mappings are specified in SpecialCasing.txt. Some are locale independent (like my German sharp/double S example) and others are locale aware (like the Turkish example). For the locale aware mappings you need to ensure you are setting the language/locale correctly in your language of choice, or that the language is doing the right thing. [1] https://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.txt https://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.tx... [2] https://www.unicode.org/reports/tr44/#Casemapping https://www.unicode.org/reports/tr44/#Casemapping [3] https://www.unicode.org/Public/UCD/latest/ucd/SpecialCasing.txt https://www.unicode.org/Public/UCD/latest/ucd/SpecialCasing....
Dwedit 1y ago
In C# programming, you are able to specify a culture every time you call a function such as numbers <-> strings, or case conversion. Or you specify the "Invariant Culture", which is basically US English. But the default culture is still based on your system's locale, you need to explicitly name the invariant culture everywhere. Because it involves a lot of filling in parameters for many different functions, people often leave it out, then their code breaks on systems where "," is the decimal separator. You can also change the default culture to the invariant culture and save all the headaches. Save the localized number conversion and such for situations where you actually need to interact with localized values.
- gf000 1y ago
  The same is true for Java/Kotlin (in this case at least). The problem is that there is a zero parameter version that implicitly depends on global state, so you may end up with the bug unless you were already familiar with the issue at hand - I think the same applies for C#. Though linters will routinely catch this particular issue FWIW.
hshdhdhehd 1y ago
Tldr. ToLowerCase is like converting a time to a string. For human display purposes only.
- DenisM 1y ago
  It stops working when people ask for cases-insensitive file names, as is their right.
jillesvangurp 1y ago
Interesting one. That and and relying on system character encodings is a source of subtle bugs. I've been bitten by that many times with e.g. XML parsing in the past. Modern Kotlin thankfully has very few (if any) places left where this can happen. Kotlin has parameters with default values. So anything that relies on a character encoding usually has a parameter of encoding set to UTF-8. The bug here was the default Java implementation that Kotlin uses on JVM. On kotlin-js both toLowerCase() and lowercase() do exactly the same thing. Also, the deprecation mechanism in Kotlin is kind of cool. The deprecated implementation is still there and you could use it with a compiler flag to disable the error. @Deprecated("Use lowercase() instead.", ReplaceWith("lowercase(Locale.getDefault())", "java.util.Locale")) @DeprecatedSinceKotlin(warningSince = "1.5", errorSince = "2.1") @kotlin.internal.InlineOnly public actual inline fun String.toLowerCase(): String = (this as java.lang.String).toLowerCase() /** * Returns a copy of this string converted to lower case using Unicode mapping rules of the invariant locale. * * This function supports one-to-many and many-to-one character mapping, * thus the length of the returned string can be different from the length of the original string. * * @sample samples.text.Strings.lowercase */ @SinceKotlin("1.5") @kotlin.internal.InlineOnly public actual inline fun String.lowercase(): String = (this as java.lang.String).toLowerCase(Locale.ROOT)
Someone 1y ago
FTA: “Less than a week later, they had a fix ready: (source: GitHub) map[name] = "box${primitiveType.javaKeywordName.capitalize(Locale.US)}" […] In September 2020, nearly a year after the coroutines bug had been fixed and forgotten […] When they came to fix this issue, the Kotlin team weren’t leaving anything to chance. They scoured the entire compiler codebase for case-conversion operations—calls to capitalize(), decapitalize(), toLowerCase(), and toUpperCase()” Bloody late, I would say. If something like this happened in OpenBSD, I think they would have done that, and more (the article doesn’t mention tooling to detect the introduction of new similar bugs ofof adding warnings to documentation), at the first spotting of such a bug. How come no reviewer mentioned such things when the first fix was reviewed? Also, why are they using Locale.US, and not Locale.ROOT (https://docs.oracle.com/javase/8/docs/api/java/util/Locale.html#ROOT https://docs.oracle.com/javase/8/docs/api/java/util/Locale.h...)?
phplovesong 1y ago
Wow this is bad. Even for a language like Java having vanilla strings as some sort of enum like value, and then even going further and downcasing them is a 100% bugmagnet waiting for the kaboom.
originHarbor1 1y ago
Every programmer learns about Turkish 'i' the hard way, usually at 3 AM in production.
voidUpdate 1y ago
> If capitalize() was an ambiguous name, what should its replacement be called? Can you think of a name that describes the function’s behaviour more clearly? In c#, setting every letter to its uppercase form is ToUpper, and I think capitalise is perfectly reasonable for setting the first character. I'm not sure I've ever referred to uppercasing a string as capitalising it
immibis 1y ago
For a while when I made Minecraft mods, I had my test environment set to Turkish for this exact reason (there's some simple command-line parameter you can pass to the JVM). Half the other mods installed in this environment would have broken textures, but mine never did since I tested it.
zahlman 1y ago
I knew from the headline that this would be the Turkish I thing, but I couldn't fathom why a compiler would care about case-folding. "I don't know Kotlin, but surely its syntax is case-sensitive like all the other commonly used languages nowadays?" > The code is part of a class named CompilerOutputParser, and is responsible for reading XML files containing messages from the Kotlin compiler. Those files look something like this: "Oh." "... Seriously?" As if I didn't hate XML enough already.
- _ZeD_ 1y ago
  what do you propose to handle translation messages? how do you think they should map the compiler codes to human messages?
  - colejohnson66 1y ago
    .NET ResX localization generates a source file. So localized messages are just `Resources.TheKey` - a named property like anything else in the language. It also catches key renaming bugs because the code will fail to compile if you remove/rename a key without updating users.
  - zahlman 1y ago
    ... Just about anything else? The baseline expectation in other data formats is that keys are case-sensitive. (Also, that there are keys, specifically designed for lookup of the needed data, rather than tag names.)
- James_K 1y ago
  >https://www.w3.org/TR/REC-xml/ https://www.w3.org/TR/REC-xml/ >match >[Definition: (Of strings or names:) Two strings or names being compared are identical. Characters with multiple possible representations in ISO/IEC 10646 (e.g. characters with both precomposed and base+diacritic forms) match only if they have the same representation in both strings. No case folding is performed. I'm quite fond of XML myself, and this is not an issue in XML.
  - 1y ago
iqfareez 1y ago
[dead]
iqfareez 1y ago
[dead]

161 points by Bogdanp 1y ago

I was scrolling and scrolling, waiting for the author to mention the new methods, which of course every Android Dev had to migrate to at some point. And 99% of us probably thought how annoying this change is, even though it probably reduced the number of bugs for Turkish users :) Unrelated, but a month ago I found a weird behaviour where in a kotlin scratch file, `List.isEmpty()` is always true. Questioned my sanity for at least an hour there... https://youtrack.jetbrains.com/issue/KTIJ-35551/ https://youtrack.jetbrains.com/issue/KTIJ-35551/

ajkjk 1y ago
well now I wanna know what's going on there!

johnyzee 1y ago

Ugh, I've had the exact same problem in a Java project, which meant I had to go through thousands and thousands of lines of code and make sure that all 'toLowerCase()' on enum names included Locale.ENGLISH as parameter. As the article demonstrates, the error manifests in a completely inscrutable way. But once I saw the bug from a couple of users with Turkish sounding names, I zeroed in on it. And cursed a few times under my breath whoever messed up that character table so bad.

nradov 1y ago
Were you not using static analysis tools? All of the popular ones will warn about that issue with locales.
- lucumo 1y ago
  They do. But a generic warning about locale-dependence doesn't really tell you that ASCII-strings will be broken. For nearly every purpose ASCII is the same in every locale. If you have a string that is guaranteed to be ASCII (like an enum constant is in most code styles), it's easy to think "not a problem here" and move on.

mikestew 1y ago

When I saw "Turkish alphabet bug", I just knew it was some version of toLower() gone horribly wrong. (I'm sure there's a good reason, but I find it odd that compiler message tags are invariably uppercase, but in this problem code they lowercased it to go do a lookup from an enum of lowercase names. Why isn't the enum uppercase, like the things you're going to lookup?)

thaumasiotes 1y ago
> Why isn't the enum uppercase, like the things you're going to lookup? Another question: why does the log record the string you intended to look up, instead of the string you actually did look up?
kevin_thibedeau 1y ago
With Turkish you can't safely case-fold with toupper() or tolower() in a C/US locale: i->I and I->i are both wrong. Uppercasing wouldn't work. You have to use Unicode or Latin-5 to manage it.
- oneshtein 1y ago
  It's the bug in the Turkish locale. They hacked Latin alphabet instead of creating a separate letter with separate rules.
- teo_zero 1y ago
  You misunderstood the parent post. They where suggesting to look up the exact string that ends up in the message, without any conversion. So if the message contains INFO, ERROR, etc. then look up "INFO", "ERROR"...
kokada 1y ago
Without looking at the source code I think it is because the log functions are lowercase, but I am not sure this is the reason.

charcircuit 1y ago

Everyone who has used Java has hit this before. Java really should force people to always specify the locale and get rid of the versions of the functions without locale parameters. There is so much hidden broken code out there.

Uvix 1y ago
That only helps if devs specify an invariant locale (ROOT for Java) where needed. In practice, I think you'll see devs blindly using using the user's current locale like it silently does today.
- jeroenhd 1y ago
  The invariant locale can't parse the numbers I enter (my locale uses comma as a decimal separator). More than a few applications will reject perfectly valid numbers. Intel's driver control panel was even so fucked up that I needed to change my locale to make it parse its own UI layout files. Defaulting to ROOT makes a lot of sense for internal constants, like in the example in this article, but defaulting to ROOT for everything just exposes the problems that caused Sun to use the user locale by default in the first place.
  - Uvix 1y ago
    Agreed, there are cases where user locale is needed. So many so that I expect that to be devs’ default if required to specify, and that they won’t use ROOT where they should.

zettabomb 1y ago

I have always wondered why Turkey chose to Latinize in this way. I understand that the issue is having two similar vowels in Turkish, but not why they decided to invent the dotless I, when other diacritics already existed. Ĭ Î Ï Í Ì Į Ĩ and almost certainly a dozen other would've worked, unless there was already some significance to the dot in Turkish that's not obvious.

mrighele 1y ago
The issue is not the invention of the dotless I, it already exists, the issue is that the took a vowerl , i/I, and the assigned the lower case to one vowel, and the upper case to a different one, and invented what left missing. It's like they decided that the uppercase of "a" is "E" and the uppercase of "e" is "A".
- pinkmuffinere 1y ago
  This is misleading, because it assumes that i/I naturally represent one vowel, which is just not the case. i/I represents one vowel in _English_, when written with a latin script. ̶I̶n̶ ̶f̶a̶c̶t̶ ̶e̶v̶e̶n̶ ̶t̶h̶i̶s̶ ̶i̶s̶n̶'̶t̶ ̶c̶o̶r̶r̶e̶c̶t̶,̶ ̶i̶/̶I̶ ̶r̶e̶p̶r̶e̶s̶e̶n̶t̶s̶ ̶o̶n̶e̶ ̶p̶h̶o̶n̶e̶m̶e̶,̶ ̶n̶o̶t̶ ̶o̶n̶e̶ ̶v̶o̶w̶e̶l̶.̶ <see troad's comment for correction> There is no reason to assume that the English representation is in general "correct", "standard", or even "first". The modern script for Turkish was adopted around the 1920's, so you could argue perhaps that most typewriters presented a standard that should have been followed. However, there was variation even between different typewriters, and I strongly suspect that typewriters weren't common in Turkey when the change was made.
  - ginko 1y ago
    >This is misleading, because it assumes that i/I naturally represent one vowel, which is just not the case. It does in literally any language using a latin alphabet other than Turkish.
    - pinkmuffinere 1y ago

okanat 1y ago

As a Turkish speaker who was using a Turkish-locale setup in my teenage years these kinds of bugs frustrated me infinitely. Half of the Java or Python apps I installed never run. My PHP webservers always had problems with random software. Ultimately, I had to change my system's language to English. However, US has godawful standards for everything: dates, measurement units, paper sizes. When I shared computers with my parents I had to switch languages back-and-forth all the time. This helped me learn English rather quickly but, I find it a huge accessibility and software design issue. If your program depends on letter cases, that is a badly designed program, period. If a language ships toUpper or a toLower function without a mandatory language field, it is badly designed too. The only slightly-better option is making toUpper and toLower ASCII-only and throwing error for any other character set. While half of the language design of C is questionable and outright dangerous, making its functions locale-sensitive by all popular OSes was an avoidable mistake. Yet everybody did that. Just the existence of this behavior is a reason I would like to get rid of anything GNU-based in the systems I develop today. I don't care if Unicode releases a conversion map. Natural-language behavior should always require natural language metadata too. Even modern languages like Rust did a crappy job of enforcing it: https://doc.rust-lang.org/std/primitive.char.html#method.to_uppercase https://doc.rust-lang.org/std/primitive.char.html#method.to_... . Yes it is significantly safer but converting 'ß' to 'SS' in German definitely has gotchas too.

arccy 1y ago
use Australian English: English but with same settings for everything else, including keyboard layout
- Sesse__ 1y ago
  Many Linux distributions provide en_DK specifically for this purpose. English as it is used in Denmark. :-)
  - fph 1y ago
    Denmark doesn't have Euros as currency, unfortunately.

darkhorn 1y ago

Java; write once, run anywhere, except on Turkish Windows.

sjrd 1y ago

I am one of the maintainers is the Scala compiler, and this is one of the things that immediately jump to me when I review code that contains any casing operation. Always explicitly specify the locale. However, unlike TFA and other comments, I don't suggest `Locale.US`. That's a little too US-centric. The canonical locale is in fact `Locale.ROOT`. Granted, in practice it's equivalent, but I find it a little bit more sensible. Also, this is the last remaining major system-dependent default in Java. They made strict floating point the default in 17; UTF-8 the default encoding some versions later (21?); only the locale remains. I hope they make ROOT the default in an upcoming version. FWIW, in the Scala.js implementation, we've been using UTF-8 and ROOT as the defaults forever.

JuniperMesos 1y ago
> However, unlike TFA and other comments, I don't suggest `Locale.US`. That's a little too US-centric. The canonical locale is in fact `Locale.ROOT`. Granted, in practice it's equivalent, but I find it a little bit more sensible. I have no idea what `Locale.ROOT` refers to, and I'd be worried that it's accidentally the same as the system locale or something, exactly the sort of thing that will unexpectedly change when a Turkish-speaker uses a computer or what have you.
- kevin_thibedeau 1y ago
  It is a programming language agnostic equivalent of POSIX C locale with Unicode enhancement.
- layer8 1y ago
  > I'd be worried that it's accidentally the same as the system locale or something The API docs clearly specify that Locale.ROOT “is regarded as the base locale of all locales, and is used as the language/country neutral locale for the locale sensitive operations.”
- troad 1y ago
  > However, unlike TFA and other comments, I don't suggest `Locale.US`. That's a little too US-centric. The canonical locale is in fact `Locale.ROOT`. Granted, in practice it's equivalent, but I find it a little bit more sensible. Isn't it kind of strange to say that Locale.US is too US centric, and therefore we'll invent a new, fictitious locale, the contents of which is all the US defaults, but which we'll call "the base locale of all locales"? That somehow seems even more US centric to me than just saying Locale.US. Setting the locale as Locale.US is at least comprehensible at a glance.

deleted 1y ago

[deleted]

esafak 1y ago

Kotlin keywords should be assumed to be English.

pimlottc 1y ago
Logging levels are not language keywords.
- DamonHD 1y ago
  The implied locale of those logging levels is (US) English though. And so any recasing of them should be in that locale.

naniwaduni 1y ago

A stark reminder that all operations on strings are wrong.

lifthrasiir 1y ago
Or that strings are not human texts.
- oneshtein 1y ago
  Kotlin is not for humans.
hshdhdhehd 1y ago
And all code is operations on strings. (The code starts out as a string).

liquidpele 1y ago

It’s always Turkish lol. That was our language of choice to QA anything… if it worked on that it would pretty much work on anything.

ramses0 1y ago
I'm shocked there's no mention of "The Turkey Test" https://blog.codinghorror.com/whats-wrong-with-turkey/ https://blog.codinghorror.com/whats-wrong-with-turkey/
- OptionOfT 1y ago
  I really dislike the fact that that article uses the “” quotes instead of "". Makes copy-pasting not work.

emmelaich 1y ago

Could have been worse -- Ramazan Çalçoban sent his estranged wife Emine the text message: Zaten sen sıkışınca konuyu değiştiriyorsun. "Anyhow, whenever you can't answer an argument, you change the subject." Unfortunately, what she thought he wrote was: Zaten sen sikişınce konuyu değiştiriyorsun. "Anyhow, whenever they are fucking you, you change the subject." This led to a fight in which the woman was stabbed and died and the man committed suicide in prison. https://gizmodo.com/a-cellphones-missing-dot-kills-two-people-puts-three-m-382026 https://gizmodo.com/a-cellphones-missing-dot-kills-two-peopl...

esafak 1y ago
> sikişınce The last one should be an i too :)

flexagoon 1y ago

Wouldn't at least the first issue be solved by using Unicode case folding instead of lowercase? Python, for example, has separate .casefold() and .lower() methods, and AFAIK casefold would always turn I into i, and is much more appropriate for this use case.

rhdunn 1y ago
There are 3 types of case folding: 1. Simple one-to-one mappings -- E.g. `T` to `t`. These are typically the ones handled by `lower()` or similar methods as they work on single characters so can modify a string in place (the length of the string doesn't change). 2. More complex one-to-many mappings -- E.g. German `ß` to `ss`. These are covered by functions like `casefold()`. You can't modify the string in place so the function needs to always write to a new string buffer. 3. Locale-specific mappings -- This is what this bug is about. In Turkish `I` maps to `ı` whereas other languages/locales it maps to `i`. You can only implement this by passing the locale to the case function, irrespective of whether you are also doing (1) or (2).
- zahlman 1y ago
  This is not quite right, at least for Python. .upper() and .lower() (and .casefold() as well) implement the default casing algorithms from the Unicode specification, which are one-to-many (but still locale-naive). Other languages, meanwhile, might well implement locale-aware mapping that defaults to the system locale rather than requiring a locale to be passed.
  - rhdunn 1y ago
    For one-to-one I was thinking of the Unicode Character Database single-character upper, lower, and title case entries in the UnicodeData.txt file [1] -- i.e. the simple case mappings [2]. The one-to-many mappings are specified in SpecialCasing.txt. Some are locale independent (like my German sharp/double S example) and others are locale aware (like the Turkish example). For the locale aware mappings you need to ensure you are setting the language/locale correctly in your language of choice, or that the language is doing the right thing. [1] https://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.txt https://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.tx... [2] https://www.unicode.org/reports/tr44/#Casemapping https://www.unicode.org/reports/tr44/#Casemapping [3] https://www.unicode.org/Public/UCD/latest/ucd/SpecialCasing.txt https://www.unicode.org/Public/UCD/latest/ucd/SpecialCasing....

Dwedit 1y ago

In C# programming, you are able to specify a culture every time you call a function such as numbers <-> strings, or case conversion. Or you specify the "Invariant Culture", which is basically US English. But the default culture is still based on your system's locale, you need to explicitly name the invariant culture everywhere. Because it involves a lot of filling in parameters for many different functions, people often leave it out, then their code breaks on systems where "," is the decimal separator. You can also change the default culture to the invariant culture and save all the headaches. Save the localized number conversion and such for situations where you actually need to interact with localized values.

gf000 1y ago
The same is true for Java/Kotlin (in this case at least). The problem is that there is a zero parameter version that implicitly depends on global state, so you may end up with the bug unless you were already familiar with the issue at hand - I think the same applies for C#. Though linters will routinely catch this particular issue FWIW.

hshdhdhehd 1y ago

Tldr. ToLowerCase is like converting a time to a string. For human display purposes only.

DenisM 1y ago
It stops working when people ask for cases-insensitive file names, as is their right.

jillesvangurp 1y ago

Interesting one. That and and relying on system character encodings is a source of subtle bugs. I've been bitten by that many times with e.g. XML parsing in the past. Modern Kotlin thankfully has very few (if any) places left where this can happen. Kotlin has parameters with default values. So anything that relies on a character encoding usually has a parameter of encoding set to UTF-8. The bug here was the default Java implementation that Kotlin uses on JVM. On kotlin-js both toLowerCase() and lowercase() do exactly the same thing. Also, the deprecation mechanism in Kotlin is kind of cool. The deprecated implementation is still there and you could use it with a compiler flag to disable the error. @Deprecated("Use lowercase() instead.", ReplaceWith("lowercase(Locale.getDefault())", "java.util.Locale")) @DeprecatedSinceKotlin(warningSince = "1.5", errorSince = "2.1") @kotlin.internal.InlineOnly public actual inline fun String.toLowerCase(): String = (this as java.lang.String).toLowerCase() /** * Returns a copy of this string converted to lower case using Unicode mapping rules of the invariant locale. * * This function supports one-to-many and many-to-one character mapping, * thus the length of the returned string can be different from the length of the original string. * * @sample samples.text.Strings.lowercase */ @SinceKotlin("1.5") @kotlin.internal.InlineOnly public actual inline fun String.lowercase(): String = (this as java.lang.String).toLowerCase(Locale.ROOT)

Someone 1y ago

FTA: “Less than a week later, they had a fix ready: (source: GitHub) map[name] = "box${primitiveType.javaKeywordName.capitalize(Locale.US)}" […] In September 2020, nearly a year after the coroutines bug had been fixed and forgotten […] When they came to fix this issue, the Kotlin team weren’t leaving anything to chance. They scoured the entire compiler codebase for case-conversion operations—calls to capitalize(), decapitalize(), toLowerCase(), and toUpperCase()” Bloody late, I would say. If something like this happened in OpenBSD, I think they would have done that, and more (the article doesn’t mention tooling to detect the introduction of new similar bugs ofof adding warnings to documentation), at the first spotting of such a bug. How come no reviewer mentioned such things when the first fix was reviewed? Also, why are they using Locale.US, and not Locale.ROOT (https://docs.oracle.com/javase/8/docs/api/java/util/Locale.html#ROOT https://docs.oracle.com/javase/8/docs/api/java/util/Locale.h...)?

phplovesong 1y ago

Wow this is bad. Even for a language like Java having vanilla strings as some sort of enum like value, and then even going further and downcasing them is a 100% bugmagnet waiting for the kaboom.

originHarbor1 1y ago

Every programmer learns about Turkish 'i' the hard way, usually at 3 AM in production.

voidUpdate 1y ago

> If capitalize() was an ambiguous name, what should its replacement be called? Can you think of a name that describes the function’s behaviour more clearly? In c#, setting every letter to its uppercase form is ToUpper, and I think capitalise is perfectly reasonable for setting the first character. I'm not sure I've ever referred to uppercasing a string as capitalising it

immibis 1y ago

For a while when I made Minecraft mods, I had my test environment set to Turkish for this exact reason (there's some simple command-line parameter you can pass to the JVM). Half the other mods installed in this environment would have broken textures, but mine never did since I tested it.

zahlman 1y ago

I knew from the headline that this would be the Turkish I thing, but I couldn't fathom why a compiler would care about case-folding. "I don't know Kotlin, but surely its syntax is case-sensitive like all the other commonly used languages nowadays?" > The code is part of a class named CompilerOutputParser, and is responsible for reading XML files containing messages from the Kotlin compiler. Those files look something like this: "Oh." "... Seriously?" As if I didn't hate XML enough already.

_ZeD_ 1y ago
what do you propose to handle translation messages? how do you think they should map the compiler codes to human messages?
- colejohnson66 1y ago
  .NET ResX localization generates a source file. So localized messages are just `Resources.TheKey` - a named property like anything else in the language. It also catches key renaming bugs because the code will fail to compile if you remove/rename a key without updating users.
- zahlman 1y ago
  ... Just about anything else? The baseline expectation in other data formats is that keys are case-sensitive. (Also, that there are keys, specifically designed for lookup of the needed data, rather than tag names.)
James_K 1y ago
>https://www.w3.org/TR/REC-xml/ https://www.w3.org/TR/REC-xml/ >match >[Definition: (Of strings or names:) Two strings or names being compared are identical. Characters with multiple possible representations in ISO/IEC 10646 (e.g. characters with both precomposed and base+diacritic forms) match only if they have the same representation in both strings. No case folding is performed. I'm quite fond of XML myself, and this is not an issue in XML.
- 1y ago

iqfareez 1y ago

[dead]

iqfareez 1y ago

[dead]