Other languages have similar problems, although not as striking. It does not make sense to special-case Azeri, Crimean Tatar, Kurdish, Turkish, Tatar as you pro

Other languages have similar problems, although not as striking. It does not make sense to special-case Azeri, Crimean Tatar, Kurdish, Turkish, Tatar as you propose. The real problem is giving in to the notion that having an unknown mix of languages in a text is an acceptable state of affairs. It's not; we should recognise the problem for what it is and work towards a future where it occurs less. Documents carry metadata about the language content, e.g. HTML and OpenDocument, precisely so that the algorithms which depend on a language like case-folding or word matching work correctly. The situation is analogous to having a document with an unknown/undeclared text encoding: rely on the metadata where supplied, otherwise make a guess, perhaps informed by statistical properties of the text (e.g. uchardet) rather than system locale. It's not ideal, but works more often than not.