13 ms·

A popular but wrong way to convert a string to uppercase or lowercase

209 points by ingve 2y ago

vardump 2y ago
As always, Raymond is right. (And as usually, I could guess it's him before even clicking the link.) That said, 99% time when doing upper- or lowercase operation you're interested just in the 7-bit ASCII range of characters. For the remaining 1%, there's ICU library. Just like Raymond Chen mentioned.
- fhars 2y ago
  No, when you are doing string manipulation, you are almost never interestet in just the seven bit ASCII range, as there is almost no language that can be written using just that.
  - daemin 2y ago
    I would argue that for most programs when you're doing string manipulation you're doing it for internal programming reasons - logs, error messages, etc. In that case you are in nearly full control of the strings and therefore can declare that you're only working with ASCII. The other normal cases of string usage are file paths and user interface, and the needed operations can be done with simple string functions, and even in UTF8 encoding the characters you care about are in the ASCII range. With file paths the manipulations that you're most often doing is path based so you only care about '/', '\', ':', and '.' ASCII characters. With user interface elements you're likely to be using them as just static data and only substituting values into placeholders when necessary.
    - pistoleer 2y ago
      > I would argue that for most programs when you're doing string manipulation you're doing it for internal programming reasons - logs, error messages, etc. In that case you are in nearly full control of the strings and therefore can declare that you're only working with ASCII. Why would you argue that? In my experience it's about formatting things that are addressed to the user, where the hardest and most annoying localization problems matter a lot. That includes sorting the last name "van den Berg" just after "Bakker", stylizing it as "Berg, van den", and making sure this capitalization is correct and not "Van Den Berg". There is no built in standard library function in any language that does any of that. It's so much larger than ascii and even larger than unicode. Another user said that the main takeaway is that you can't process strings until you know their language (locale), and that is exactly correct.
      - daemin 2y ago
        I would maintain that your program has more string manipulation for error messages and logging than for generating localised formatted names. Further I do say that if you're creating text for presenting to the user then the most common operation would be replacement of some field in pre-defined text. In your case I would design it so that the correctly capitalised first name, surname, and variations of those for sorting would be generated at the data entry point (manually or automatically) and then just used when needed in user facing text generation. Therefore the only string operation needed would be replacement of placeholders like the fmt and standard library provide. This uses more memory and storage but these are cheaper now.
        pistoleer 2y ago
        I agree, but the logging formatters don't really do much beyond trivially pasting in placeholders. And as for data entry... Maybe in an ideal world. In the current world, marred by importing previously mangled datasets, a common solution in the few companies I've worked at is to just not do anything, which leaves ugly edges, yet is "good enough".
    - BoringTimesGang 2y ago
      Now double all of that effort, so you can get it to work with Windows' UTF-16 wstrings.
      - account42 2y ago
        Better to just convert WTF-16 (Windows filenames re not guaranteed to be valid UTF-16) to/from WTF-8 at the API boundary and then do the same processing internally on all platforms.
    - heisenzombie 2y ago
      File paths? I think filesystem paths are generally “bags of bytes” that the OS might interpret as UTF-16 (Windows) or UTF-8 (macOS, Linux). For example: https://en.m.wikipedia.org/wiki/Program_Files#Localization https://en.m.wikipedia.org/wiki/Program_Files#Localization
      - vardump 2y ago
        File paths are scary. The last I checked (which is admittedly a while ago), Windows didn't for example care about correct UTF-16 surrogate pairs at all, it'd happily accept invalid UTF-16 strings. So use standard string processing libraries on path names at your own peril. It's a good idea to consider file paths as a bag of bytes.
        Someone 2y ago
        > It's a good idea to consider file paths as a bag of bytes (Nitpick: sequence of bytes) Also very limiting. If you do that, you can’t, for example, show a file name to the user as a string or easily use a shell to process data in your file system (do you type “/bin” or “\x2F\x62\x69\x6E”?) Unix, from the start, claimed file names where byte sequences, yet assumed many of those to encode ascii. That’s part of why Plan 9 made the choice “names may contain any printable character (that is, any character outside hexadecimal 00-1F and 80-9F)” (https://9fans.github.io/plan9port/man/man9/intro.html https://9fans.github.io/plan9port/man/man9/intro.html)
        daemin 2y ago
        That's what I mean, you treat filesystem paths as bags of bytes separated by known ASCII characters, as the only path manipulation that you generally need to do is to append a path, remove a path, change extension, things that only care about those ASCII characters. You only modify the path strings at those known characters and leave everything in between as is (with some exceptions using OS API specific functions as needed).
        netsharc 2y ago
        IIRC, the FAT filesystem (before Windows 95) allowed lowercase letters, but there's a layer in the filesystem driver that converted everything to uppercase, e.g. if you did the command "more readme.txt", the more command would ask the filesystem for "readme.txt" and it would search for "README.TXT" in the file allocation table. I think I once hex-edited the FA-table to change a filename to have a lowercase name (or maybe it was disk corruption), trying to delete that file didn't work because it would be trying to delete "FOO", and couldn't find it because the file was named "FOo".
  - vardump 2y ago
    > as there is almost no language that can be written using just that. 99% of use cases I've seen have nothing to do with human language. 1% human language case that is needs to be handled properly using a proper Unicode library. Your mileage (percentages) may vary depending on your job.
    - 9dev 2y ago
      It's funny how software developers live in bubbles so much. Whether you deal with human language a lot or almost not at all depends entirely on your specific domain. Anyone working on user interfaces of any kind must accommodate for proper encoding, for example; that includes pretty much every line-of-business app out there, which is a lot of code.
    - inexcf 2y ago
      Why do you need upper- or lowercase conversion in cases that have nothing to do with human language?
      - vardump 2y ago
        Here's an example. Hypothetically say you want to build an HTML parser. You might encounter tags like <html>, <HTML>, <Html>, etc., but you want to perform a hash table lookup. So first you're going to normalize to either lower- or uppercase.
        Muromec 2y ago
        But but, I want to have a custom web component and register it under my own name, which can only be properly written in Ukrainian Cyrillic. How dare you not let me have it.
        inexcf 2y ago
        Ah, i see, we disagree on what is "human language". An abbreviation like HTML and it's different capitalisations to me sound a lot like a feature of human language.
- sebstefan 2y ago
  Yes please, keep making software that mangles my actual last name at every step of the way. 99% of the world loves it when you only care about the USA.
  - Muromec 2y ago
    If it needs to uppercase names it probably interfaces with something forsaken like Sabre/Amadeus that only understands ASCII anyway. The real problem is accepting non-ASCII input from user where you later assume it's ASCII-only and safe to bitfuck around.
    - sebstefan 2y ago
      From experience anything banking adjacent will usually fuck it up as well For some reason they have a hard-on for putting last names in capital letters and they still have systems in place that use ASCII
      - Muromec 2y ago
        If it uses ASCII anyway, what's the problem then? Don't accept non-ASCII user input.
        sebstefan 2y ago
        First off: And exclude 70% of the world? Usually they'll accept it, but some parts of the backend are still running code from the 60's. So you get your name rendered properly on the web interface, and most core features, but one day you're wandering off from the beaten path, by, like, requesting some insurance contract, and you'll see your name at the top with some characters mangled, depending on what your name's like. Mine is just accented latin characters so it usually drops the accents ; not sure how it would work if your name was in an entirely different alphabet
        Muromec 2y ago
        >First off: And exclude 70% of the world? Guess what, I'm part of this 70% and I also work in a bank and I know exactly how. Not a single letter in my name (any of them) can be represented with ASCII. When it is represented in UTF-8, most of the people who have to see it can't read it anyway. So my identity document issued by the country which doesn't use Latin alphabet includes ASCII-representation of my name in addition to canonical form in Ukrainian Cyrillic. That ASCII-rendering is happily accepted by all kinds of systems that only speak ASCII. People still can't pronounce it and it got misspelled like yesterday when dictated over the phone. Now regarding the accents, it's illegal to not support them per GDPR (as per case law, discussed here few years ago).
- crazygringo 2y ago
  > That said, 99% time when doing upper- or lowercase operation you're interested just in the 7-bit ASCII range of characters. I think it's more the exact opposite. The only times I'm dealing with 7-bit ASCII is for internal identifiers like variable names or API endpoints. Which is a lot of the time, but I can't ever think of when I've needed my code to change their case. It might literally be never. On the other hand, needing to switch between upper, lower, and title case happens all the time, always with people's names and article titles and product names and whatnot. Which are never in ASCII because this isn't 1990.
  - hinkley 2y ago
    And you could argue that if the internal identifiers need to be capitalized or lower-cased, you've already lost. On an enterprise app these little string manipulations are a drop in the bucket. In a game they might not be. Sort that stuff out at compile time, or commit time.
    - account42 2y ago
      You can't always control the case you get but often you can not care about anything outside ASCII. Scripts and configuration or text-based data formats are common examples.
  - bigstrat2003 2y ago
    > Which are never in ASCII because this isn't 1990. This is a very silly statement. I'm willing to believe that you have lots of cases where those things are outside the ASCII range. Perhaps even most of the cases, depending on where you live. But I do not believe for one second that it never happens.
    - crazygringo 2y ago
      Never stored in ASCII, never limited to ASCII. They're UTF-8, usually. If somebody's name happens to fit into ASCII that's irrelevant because it's not guaranteed, so you can never blindly do an ASCII case conversion. For text data meant for users, I literally cannot remember the last time I used a string in ASCII format as opposed to UTF-8 (or UTF-16 in JS). It's certainly over a decade ago. So yes, when I say never, I literally mean never. Nothing "very silly" about it, sorry. (Again, excepting identifiers, where case conversion is not generally applicable.)
- PaulDavisThe1st 2y ago
  He may be right, but approximately 75% of the problems he describes are all Microsoft-ecosystem specific. In Unix-land we don't use wchar_t or UTF-16, and his article is a good demonstrations of why not.
  - pjmlp 2y ago
    UNIX land is even worse in international languages support. As in, there isn't even something on POSIX at the level other operaring systems support for localisation. Yes there is some locale stuff, however not enough for all stuff, hence why every modern programming language happens to have this as part of their standard library.
    - PaulDavisThe1st 2y ago
      When the notion of what a "string" is differs so much from language to language, i18n is never going to be an effective part of POSIX.
    - account42 2y ago
      Is there a platform where you can't use ICU?
- yas_hmaheshwari 2y ago
  Wow, I came here to write exactly that, and its heartening to see that I am not crazy Just reading the title, with microsoft.com in bracket, I knew two things: 1. It would be written by Raymond Chen 2. That article is going to be awesome
- account42 2y ago
  > That said, 99% time when doing upper- or lowercase operation you're interested just in the 7-bit ASCII range of characters. And std::tolower/toupper is the wrong tool for that as well.
blenderob 2y ago
It is issues like this due to which I gave up on C++. There are so many ways to do something and every way is freaking wrong! An acceptable solution is given at the end of the article: > If you use the International Components for Unicode (ICU) library, you can use u_strToUpper and u_strToLower. Makes you wonder why this isn't part of the C++ standard library itself. Every revision of the C++ standard brings with itself more syntax and more complexity in the language. But as a user of C++ I don't need more syntax and more complexity in the language. But I do need more standard library functions that solves these ordinary real-world programming problems.
- BoringTimesGang 2y ago
  >It is issues like this due to which I gave up on C++. There are so many ways to do something and every way is freaking wrong! These are mostly unicode or linguistics problems.
  - tralarpa 2y ago
    The fact that the standard library works against you doesn't help (to_lower takes an int, but only kind of works (sometimes) correctly on unsigned char, and wchar_t is implicitly promoted to int).
    - BoringTimesGang 2y ago
      to_lower is in the std namespace but is actually just part of the C89 standard, meaning it predates both UTF8 and UTF16. Is the alternative that it should be made unusable, and more existing code broken? A modern user has to include one of the c-prefix headers to use it, already hinting to them that 'here be dragons'. But there are always dragons. It's strings. The mere assumption that they can be transformed int-by-int, irrespective of encoding, is wrong. As is the assumption that a sensible transformation to lower case without error handling exists.
      - account42 2y ago
        > Is the alternative that it should be made unusable, and more existing code broken? It should be marked [[deprecated]], yes. There is no good reason to use std::tolower/toupper anywhere - they can neither do unicode properly nor are they anywhere close to efficient for ASCII. And their behavior depends on the process-global locale.
appointment 2y ago
The key takeaway here is that you can't correctly process a string if you don't what language it's in. That includes variants of the same language with different rules, eg en-US and en-UK or es-MX and es-ES. If you are handling multilingual text the locale is mandatory metadata.
- zarzavat 2y ago
  Different parts of a string can be in different languages too[1]. The lowercase of "DON'T FUSS ABOUT FUSSBALL" is "don't fuss about fußball". Unless you're in Switzerland. [1] https://en.wikipedia.org/wiki/Code-switching https://en.wikipedia.org/wiki/Code-switching
  - schoen 2y ago
    Probably "don't fuss about Fußball" for the same reasons, right?
  - thiht 2y ago
    I thought the German language deprecated the use of ß years ago, no? I learned German for a year and that's what the teacher told us, but maybe it's not the whole story
    - 47282847 2y ago
      Incorrect. ẞ is still a thing.
      - CamperBob2 2y ago
        Going by what you and the grandparent wrote, it's not just a thing, but two different things: ẞ ß It is probably time for an Esperanto advocate to show up and set us all straight.
        D-Coder 2y ago
        Pri kio vi parolas? En Esperanto, unu letero egalas unu sonon. What are you talking about? In Esperanto, one letter equals one sound.
ahartmetz 2y ago
...and that is why you use QString if you are using the Qt framework. QString is a string class that actually does what you want when used in the obvious way. It probably helps that it was mostly created by people with "ASCII+" native languages. Or with customers that expect not exceedingly dumb behavior. The methods are called QString::toUpper() and QString::toLower() and take only the implicit "this" argument, unlike Win32 LCMapStringEx() which takes 5-8 arguments...
- vardump 2y ago
  You just want a banana, but you also get the gorilla. And the jungle.
- cannam 2y ago
  QString::toUpper/toLower are not locale-aware (https://doc.qt.io/qt-6/qstring.html#toLower https://doc.qt.io/qt-6/qstring.html#toLower) Qt does have a locale-aware equivalent (QLocale::toUpper/toLower) which calls out to ICU if available. Otherwise it falls back to the QString functions, so you have to be confident about how your build is configured. Whether it works or not has very little to do with the design of QString.
  - ahartmetz 2y ago
    I don't see a problem with that. You can have it done locale-aware or not and "not" seems like a sane default. QString will uppercase 'ü' to 'Ü' just fine without locale-awareness whereas std::string doesn't handle non-ASCII according to the article. The cases where locale matters are probably very rare and the result will probably be reasonable anyway.
    - account42 2y ago
      That attitude is how you end up with exploits because your case folding is different from some other system you interact with.
- aetherspawn 2y ago
  I will admit I don’t love the Qt licensing model, but most things in Qt just work as they are supposed to, and on every platform too.
cyxxon 2y ago
Small nitpick: the example "LATIN SMALL LETTER SHARP S (“ß” U+00DF) uppercases to the two-character sequence “SS”:³ Straße ⇒ STRASSE" is slightly wrong, it seems to me, as we now do actually have a uppercase version of that, so it should uppercase to "Latin Capital Letter Sharp S" (U+1E9E). The double-S thing is still widely used, though.
- mkayokay 2y ago
  Duden mentions this: "Bei Verwendung von Großbuchstaben steht traditionellerweise SS für ß. In manchen Schriften gibt es aber auch einen entsprechenden Großbuchstaben; seine Verwendung ist fakultativ ‹§ 25 E3›." But isn't it also dependent on the available glyphs in the font used? So f.e. it needs to be ensured that U+1E9E exists?
  - Kwpolska 2y ago
    I don't think there exists any code that makes uppercasing decisions based on the selected font. Besides, if it doesn't exist in the current font, there's probably a fallback font.
  - NoInkling 2y ago
    According to Wikipedia: > "Since 2024 the capital ⟨ẞ⟩ is preferred over ⟨SS⟩." https://en.wikipedia.org/wiki/%C3%9F https://en.wikipedia.org/wiki/%C3%9F Check reference #5 and compare it to the older wording in reference #12.
- Muromec 2y ago
  But what if you need to uppercase the historical record in a vital records registry from 1950ies, but and OCRed last week? Now you need to not just be locale-aware, but you locale should be versioned.
- pjmlp 2y ago
  Lowering case is even better, because a Swiss user would expect the two-character sequence “SS“ to be converted into “ss“ and not “ß“. And thus we add country specific locale to the party.
serbuvlad 2y ago
The real insights here are that strings in C++ suck and UTF-16 is extremely unintuitive.
- criddell 2y ago
  Strings in C++ standard library do suck (and C++ is my favorite language). As for UTF-16, well, I don't know that UTF-8 is a whole lot more intuitive: > And for UTF-8 data, you have the same issues discussed before: Multibyte characters will not be converted properly, and it breaks for case mappings that alter string lengths.
  - recursive 2y ago
    UTF-16 has all the complexity of UTF-8 plus surrogate pairs.
    - zahlman 2y ago
      Surrogate pairs aren't more complex than UTF-8's scheme for determining the number of bytes used to represent a code point. (Arguably the logic is slightly simpler.) But the important point is that UTF-16 pretends to be a constant-length encoding while actually having the surrogate-pair loophole - that's because it's a hack on top of UCS-2 (which originally worked well enough for Microsoft to get married to; but then the BMP turned out not to be enough code points). UTF-8 is clearly designed from scratch to be a multi-byte encoding (and, while the standard now makes the corresponding sequences illegal, the scheme was designed to be able to support much higher code points - up to 2^42 if we extend the logic all the way; hypothetical 6-byte sequences starting with values FC or FD would neatly map up to 2^31).
PhilipRoman 2y ago
Thought this was going to be about and-not-ing bytes with 0x20. Wrong for most inputs but sure as hell faster than anything else.
high_na_euv 2y ago
In cpp basic things are hard
- onemoresoop 2y ago
  It's subjective but I find C++ extremely ugly.
- johnnyjeans 2y ago
  nothing about working with locales, or text in general, is basic. we were decades into working with digital computers before we moved past switchboards and LEDs. don't take for granted just how high of a perch upon the shoulders of giants you have. that's exactly how the mistakes in the blog post get made.
  - high_na_euv 2y ago
    Ive worked in various languages like C#, C and CPP and I know where Ive been fighting what kinds of problems. People always had some fancy reasoning about why things that should just work are not, but then a few years pass and things are improved. C++ is getting closer and closer to langs like C# in terms of making it harder to shot yourself, but still there is a huge room for improvement
SleepyMyroslav 2y ago
In gamedev there is simple rule: don't try to do any of that. If it is text game needs to show to user then every version of the text that is needed is a translated text. Programmer will never know if context or locale will need word order changes or anything complicated. Just trust the translation team. If text is coming from user - then change design until its not needed to 'convert'. There are major issues just to show user back what he entered! Because the font for editing and displayed text could be different. Not even mentioning RTL and other issues. Once ppl learn about localization the questions like why a programming language does not do this 'simple text operation' are just a newcomer detector. :)
- zahlman 2y ago
  >If text is coming from user - then change design until its not needed to 'convert' In games, you can possibly get away with this. Most other people need to worry about things like string collation (locale-aware sorting) for user-supplied text.
  - makeitdouble 2y ago
    TBF, if you are caring about string collation, you're already at the entrance of the rabbit hole and probably should go down to the deep end anyway. I'd assume SleepyMyroslav doesn't apply to devs willing to spend weeks at time to handle all the complexity in full.
- fluoridation 2y ago
  >Once ppl learn about localization the questions like why a programming language does not do this 'simple text operation' are just a newcomer detector. :) I think you are purposefully misinterpreting the question. They're not asking about converting the case of any Unicode string with locale sensitivity, they're asking about converting the case of ASCII characters. What if your game needs to talk to a server and do some string manipulation in between requests? Are you really going to architect everything so that the client doesn't need to handle any of that ever?
  - 2y ago
HPsquared 2y ago
I thought this was going to be about adding or subtracting 32. Old school.
- klyrs 2y ago
  I do hope you mean bitwise "addition" and "subtraction" -- (c => c&0xdf) or (c => c|0x20)
  - HPsquared 2y ago
    Tbh I come at this as a plebeian Excel user
the_gorilla 2y ago
Why are some functions addressable in C++ and others not? Seems like a pointless design oversight.
- bialpio 2y ago
  Footnote in the article provides the following explanation: "The standard imposes this limitation because the implementation may need to add default function parameters, template default parameters, or overloads in order to accomplish the various requirements of the standard."
flareback 2y ago
He gave 4 examples of how it's done incorrectly, but zero actual examples of doing it correctly.
- commandlinefan 2y ago
  for (int i = 0; i < strlen(s); i++) { s[i] ^= 0x20; }
  - vardump 2y ago
    Surely you meant: s[i] &= ~0x20; We're talking about converting to upper case after all! As an added benefit, every space character (0x20) is now a NUL byte!
    - account42 2y ago
      Free strtok!
  - calibas 2y ago
    Thank you for this universal approach. I can now toggle capitalization on/off for any character, instead of just being limited to alphabetic ones! Jokes aside, I was kinda hoping for a good answer that doesn't rely on a Windows API or an external library, but I'm not sure there is one. It's a rather complex problem when you account for more than just ASCII and the English language.
    - TZubiri 2y ago
      Next up, check out our vector addition implementation of Hello+World. Spoiler alert, the result is Zalgo
- TheGeminon 2y ago
  > Okay, so those are the problems. What’s the solution? > If you need to perform a case mapping on a string, you can use LCMapStringEx with LCMAP_LOWERCASE or LCMAP_UPPERCASE, possibly with other flags like LCMAP_LINGUISTIC_CASING. If you use the International Components for Unicode (ICU) library, you can use u_strToUpper and u_strToLower.
deleted 2y ago
[deleted]
deleted 2y ago
[deleted]
PoignardAzur 2y ago
So I'm going to be that guy and say it: Man, I'm happy we don't need to deal with this crap in Rust, and we can just use String::to_lowercase. Not having to worry about things makes coding fun.
- lilyball 2y ago
  While certainly much better, you still need to be aware that doing case conversion absent any locale information will never be perfect. If you want proper locale-aware conversion you can use the icu crate (https://docs.rs/icu/latest/icu/ https://docs.rs/icu/latest/icu/).
  - account42 2y ago
    Exactly, simple "unicode-aware" case conversions are a trap. You are always going to need much more.
himinlomax 2y ago
> And in certain forms of the French language, capitalizing an accented character causes the accent to be dropped: à Paris ⇒ A PARIS. That's incorrect, using diacritics on capital letters is always the preferred form, it's just that dropping them is acceptable as it was often done for technical reasons.
codr7 2y ago
C++, where every line of code is a book waiting to be written.
deleted 2y ago
[deleted]
zzo38computer 2y ago
First, you should consider if you even need case folding; for many uses it will be unnecessary, anyways. Furthermore, the proper way to do case folding will depend on such things as the character set, the language, the specific context of the text being converted (e.g. in some cases specific letters are required, such as abbreviations of the names of SI units), etc. And then, it is not necessarily only "uppercase" and "lowercase", anyways. There might even be different ways to do by the same language, with possibly disagreements about usage (e.g. the German Eszett did not have an official capital form until 2017, although apparently some type designers did it anyways (and it was in Unicode before then, despite that)). If the character set is Unicode, then there is not actually the correct way to do it, despite what the Unicode Conspiracy insists otherwise. Also, for some uses the way that it will need to be done, there will be a specific way that it is required (due to the way that a file format or a protocol or whatever is working), so in such a case if the character set is something other than ASCII then you cannot just assume that it will always work in the same way. You also cannot necessarily depend on the locale for such a thing, since it might depend on the data, as well. These things can be as bad as they are, but Unicode just makes these things worse than that. If a program requires a specific case folding and then it will not work because it is the wrong version of Unicode and it is possible to be a security issue and/or other problems. (Another problem, which applies even if you do not use case folding, is that some people think that all text is or should be Unicode and that one character set is suitable for everything. Actually, one character set cannot be suitable for everything, regardless of what character set it is. Even if it was (which it isn't), it wouldn't be Unicode.)
alkonaut 2y ago
Handle text in two ways: either it's controlled by you and you can do simple, efficient, and naive processing, or it's not (it's translated resources, or user input) and you can't. For the former case, you don't need any complex logic. A very typical example would be: i'm serializing a field or constructing a url so I want the variable name "Someproperty" as a lower case string. The lowercase transform is completely naive. I know exactly what the range of possible characters are and they aren't going to be Turkish or emoji, not least because I have asserted they won't be. And THIS is what the regular programming functions for upper/lower case are for. They are important, and they are most often correct. Because for all the other cases (i18n, user input, ...) you probably don't want to do toUpper/toLower at all to begin with! Example, if you present a message to the user from resources so your code is translate("USER_DIALOG_QUESTION_ABOUT_FISH") which you want to lookup knowing it will be in sentence case, and present as uppercase, what will you do? Here you likely can't, and shouldn't, do toUpper(translate(resourceKey)). Just use two resources if you want to correctly transform text. The toUpper function isn't made for this. Trying to use a complex i18n-ready toUpper/toLower only helps part of the way. It still might not understand whether two S are contracted or whether something is a proper Noun and must stay capitalized. So it adds complexity and still isn't correct. Just use two resources!
- account42 2y ago
  > For the former case, you don't need any complex logic. A very typical example would be: i'm serializing a field or constructing a url so I want the variable name "Someproperty" as a lower case string. The lowercase transform is completely naive. I know exactly what the range of possible characters are and they aren't going to be Turkish or emoji, not least because I have asserted they won't be. And THIS is what the regular programming functions for upper/lower case are for. They are important, and they are most often correct. Because for all the other cases (i18n, user input, ...) you probably don't want to do toUpper/toLower at all to begin with! C++ std::tolower/toupper (which are really just C tolower/toupper) are the wrong tool for that too though because they depend on the process locale which makes them a) horribly inefficient and b) prone to blow your program up in interesting ways on customer systems. Not quite as bad as the locale-dependent standard number parsing functions that want . in some localses and , in others but still should never be used.
Animats 2y ago
> From the article: "I find it quaint that Unicode character names are ALL IN CAPITAL LETTERS, in case you need to put them in a Baudot telegram or something." I had to do that. When we had our steampunk telegraph office at steampunk conventions [1], people could text in a message via SMS, it would be printed on a Model 14 or 15 Teletype, put in an envelope, and hand-delivered. People would use emoji in messages, and the device could only print Baudot, or International Telegraphic Alphabet #2, which is upper case only with some symbols. Emoji translation would cause the machine to hammer out (RED-HEART) or whatever emoji description was needed. Used the emoji list at [2], an older version. [1] https://vimeo.com/124065314 https://vimeo.com/124065314 [2] http://unicode.org/emoji/charts-beta/full-emoji-list.html http://unicode.org/emoji/charts-beta/full-emoji-list.html
guerrilla 2y ago
C is hard. It seems like C++ just made things way harder. I don't regret skipping it. Why not just go right to Java, C#, JS, Haskell, etc. and do what you need in C.
ChrisMarshallNY 2y ago
I generally just use the language-supported tolower/upper() (or similar) routines. I assume that they take things like UTF and alternative type systems into account. I'm not sure about other languages, but Swift has pretty intense String support[0], and can go quite a long ways. Someone actually wrote a whole book about just Swift Strings[1]. [0] https://docs.swift.org/swift-book/documentation/the-swift-programming-language/stringsandcharacters/ https://docs.swift.org/swift-book/documentation/the-swift-pr... [1] https://flight.school/books/strings/ https://flight.school/books/strings/
deleted 2y ago
[deleted]
account42 2y ago
A popular but wrong way to do Unicode > wchar_t

209 points by ingve 2y ago

As always, Raymond is right. (And as usually, I could guess it's him before even clicking the link.) That said, 99% time when doing upper- or lowercase operation you're interested just in the 7-bit ASCII range of characters. For the remaining 1%, there's ICU library. Just like Raymond Chen mentioned.

fhars 2y ago
No, when you are doing string manipulation, you are almost never interestet in just the seven bit ASCII range, as there is almost no language that can be written using just that.
- daemin 2y ago
  I would argue that for most programs when you're doing string manipulation you're doing it for internal programming reasons - logs, error messages, etc. In that case you are in nearly full control of the strings and therefore can declare that you're only working with ASCII. The other normal cases of string usage are file paths and user interface, and the needed operations can be done with simple string functions, and even in UTF8 encoding the characters you care about are in the ASCII range. With file paths the manipulations that you're most often doing is path based so you only care about '/', '\', ':', and '.' ASCII characters. With user interface elements you're likely to be using them as just static data and only substituting values into placeholders when necessary.
  - pistoleer 2y ago
    > I would argue that for most programs when you're doing string manipulation you're doing it for internal programming reasons - logs, error messages, etc. In that case you are in nearly full control of the strings and therefore can declare that you're only working with ASCII. Why would you argue that? In my experience it's about formatting things that are addressed to the user, where the hardest and most annoying localization problems matter a lot. That includes sorting the last name "van den Berg" just after "Bakker", stylizing it as "Berg, van den", and making sure this capitalization is correct and not "Van Den Berg". There is no built in standard library function in any language that does any of that. It's so much larger than ascii and even larger than unicode. Another user said that the main takeaway is that you can't process strings until you know their language (locale), and that is exactly correct.
    - daemin 2y ago
      I would maintain that your program has more string manipulation for error messages and logging than for generating localised formatted names. Further I do say that if you're creating text for presenting to the user then the most common operation would be replacement of some field in pre-defined text. In your case I would design it so that the correctly capitalised first name, surname, and variations of those for sorting would be generated at the data entry point (manually or automatically) and then just used when needed in user facing text generation. Therefore the only string operation needed would be replacement of placeholders like the fmt and standard library provide. This uses more memory and storage but these are cheaper now.
      - pistoleer 2y ago
        I agree, but the logging formatters don't really do much beyond trivially pasting in placeholders. And as for data entry... Maybe in an ideal world. In the current world, marred by importing previously mangled datasets, a common solution in the few companies I've worked at is to just not do anything, which leaves ugly edges, yet is "good enough".
  - BoringTimesGang 2y ago
    Now double all of that effort, so you can get it to work with Windows' UTF-16 wstrings.
    - account42 2y ago
      Better to just convert WTF-16 (Windows filenames re not guaranteed to be valid UTF-16) to/from WTF-8 at the API boundary and then do the same processing internally on all platforms.
  - heisenzombie 2y ago
    File paths? I think filesystem paths are generally “bags of bytes” that the OS might interpret as UTF-16 (Windows) or UTF-8 (macOS, Linux). For example: https://en.m.wikipedia.org/wiki/Program_Files#Localization https://en.m.wikipedia.org/wiki/Program_Files#Localization
    - vardump 2y ago
      File paths are scary. The last I checked (which is admittedly a while ago), Windows didn't for example care about correct UTF-16 surrogate pairs at all, it'd happily accept invalid UTF-16 strings. So use standard string processing libraries on path names at your own peril. It's a good idea to consider file paths as a bag of bytes.
      - Someone 2y ago
        > It's a good idea to consider file paths as a bag of bytes (Nitpick: sequence of bytes) Also very limiting. If you do that, you can’t, for example, show a file name to the user as a string or easily use a shell to process data in your file system (do you type “/bin” or “\x2F\x62\x69\x6E”?) Unix, from the start, claimed file names where byte sequences, yet assumed many of those to encode ascii. That’s part of why Plan 9 made the choice “names may contain any printable character (that is, any character outside hexadecimal 00-1F and 80-9F)” (https://9fans.github.io/plan9port/man/man9/intro.html https://9fans.github.io/plan9port/man/man9/intro.html)
      - daemin 2y ago
        That's what I mean, you treat filesystem paths as bags of bytes separated by known ASCII characters, as the only path manipulation that you generally need to do is to append a path, remove a path, change extension, things that only care about those ASCII characters. You only modify the path strings at those known characters and leave everything in between as is (with some exceptions using OS API specific functions as needed).
      - netsharc 2y ago
        IIRC, the FAT filesystem (before Windows 95) allowed lowercase letters, but there's a layer in the filesystem driver that converted everything to uppercase, e.g. if you did the command "more readme.txt", the more command would ask the filesystem for "readme.txt" and it would search for "README.TXT" in the file allocation table. I think I once hex-edited the FA-table to change a filename to have a lowercase name (or maybe it was disk corruption), trying to delete that file didn't work because it would be trying to delete "FOO", and couldn't find it because the file was named "FOo".
- vardump 2y ago
  > as there is almost no language that can be written using just that. 99% of use cases I've seen have nothing to do with human language. 1% human language case that is needs to be handled properly using a proper Unicode library. Your mileage (percentages) may vary depending on your job.
  - 9dev 2y ago
    It's funny how software developers live in bubbles so much. Whether you deal with human language a lot or almost not at all depends entirely on your specific domain. Anyone working on user interfaces of any kind must accommodate for proper encoding, for example; that includes pretty much every line-of-business app out there, which is a lot of code.
  - inexcf 2y ago
    Why do you need upper- or lowercase conversion in cases that have nothing to do with human language?
    - vardump 2y ago
      Here's an example. Hypothetically say you want to build an HTML parser. You might encounter tags like <html>, <HTML>, <Html>, etc., but you want to perform a hash table lookup. So first you're going to normalize to either lower- or uppercase.
      - Muromec 2y ago
        But but, I want to have a custom web component and register it under my own name, which can only be properly written in Ukrainian Cyrillic. How dare you not let me have it.
      - inexcf 2y ago
        Ah, i see, we disagree on what is "human language". An abbreviation like HTML and it's different capitalisations to me sound a lot like a feature of human language.
sebstefan 2y ago
Yes please, keep making software that mangles my actual last name at every step of the way. 99% of the world loves it when you only care about the USA.
- Muromec 2y ago
  If it needs to uppercase names it probably interfaces with something forsaken like Sabre/Amadeus that only understands ASCII anyway. The real problem is accepting non-ASCII input from user where you later assume it's ASCII-only and safe to bitfuck around.
  - sebstefan 2y ago
    From experience anything banking adjacent will usually fuck it up as well For some reason they have a hard-on for putting last names in capital letters and they still have systems in place that use ASCII
    - Muromec 2y ago
      If it uses ASCII anyway, what's the problem then? Don't accept non-ASCII user input.
      - sebstefan 2y ago
        First off: And exclude 70% of the world? Usually they'll accept it, but some parts of the backend are still running code from the 60's. So you get your name rendered properly on the web interface, and most core features, but one day you're wandering off from the beaten path, by, like, requesting some insurance contract, and you'll see your name at the top with some characters mangled, depending on what your name's like. Mine is just accented latin characters so it usually drops the accents ; not sure how it would work if your name was in an entirely different alphabet
        Muromec 2y ago
        >First off: And exclude 70% of the world? Guess what, I'm part of this 70% and I also work in a bank and I know exactly how. Not a single letter in my name (any of them) can be represented with ASCII. When it is represented in UTF-8, most of the people who have to see it can't read it anyway. So my identity document issued by the country which doesn't use Latin alphabet includes ASCII-representation of my name in addition to canonical form in Ukrainian Cyrillic. That ASCII-rendering is happily accepted by all kinds of systems that only speak ASCII. People still can't pronounce it and it got misspelled like yesterday when dictated over the phone. Now regarding the accents, it's illegal to not support them per GDPR (as per case law, discussed here few years ago).
crazygringo 2y ago
> That said, 99% time when doing upper- or lowercase operation you're interested just in the 7-bit ASCII range of characters. I think it's more the exact opposite. The only times I'm dealing with 7-bit ASCII is for internal identifiers like variable names or API endpoints. Which is a lot of the time, but I can't ever think of when I've needed my code to change their case. It might literally be never. On the other hand, needing to switch between upper, lower, and title case happens all the time, always with people's names and article titles and product names and whatnot. Which are never in ASCII because this isn't 1990.
- hinkley 2y ago
  And you could argue that if the internal identifiers need to be capitalized or lower-cased, you've already lost. On an enterprise app these little string manipulations are a drop in the bucket. In a game they might not be. Sort that stuff out at compile time, or commit time.
  - account42 2y ago
    You can't always control the case you get but often you can not care about anything outside ASCII. Scripts and configuration or text-based data formats are common examples.
- bigstrat2003 2y ago
  > Which are never in ASCII because this isn't 1990. This is a very silly statement. I'm willing to believe that you have lots of cases where those things are outside the ASCII range. Perhaps even most of the cases, depending on where you live. But I do not believe for one second that it never happens.
  - crazygringo 2y ago
    Never stored in ASCII, never limited to ASCII. They're UTF-8, usually. If somebody's name happens to fit into ASCII that's irrelevant because it's not guaranteed, so you can never blindly do an ASCII case conversion. For text data meant for users, I literally cannot remember the last time I used a string in ASCII format as opposed to UTF-8 (or UTF-16 in JS). It's certainly over a decade ago. So yes, when I say never, I literally mean never. Nothing "very silly" about it, sorry. (Again, excepting identifiers, where case conversion is not generally applicable.)
PaulDavisThe1st 2y ago
He may be right, but approximately 75% of the problems he describes are all Microsoft-ecosystem specific. In Unix-land we don't use wchar_t or UTF-16, and his article is a good demonstrations of why not.
- pjmlp 2y ago
  UNIX land is even worse in international languages support. As in, there isn't even something on POSIX at the level other operaring systems support for localisation. Yes there is some locale stuff, however not enough for all stuff, hence why every modern programming language happens to have this as part of their standard library.
  - PaulDavisThe1st 2y ago
    When the notion of what a "string" is differs so much from language to language, i18n is never going to be an effective part of POSIX.
  - account42 2y ago
    Is there a platform where you can't use ICU?
yas_hmaheshwari 2y ago
Wow, I came here to write exactly that, and its heartening to see that I am not crazy Just reading the title, with microsoft.com in bracket, I knew two things: 1. It would be written by Raymond Chen 2. That article is going to be awesome
account42 2y ago
> That said, 99% time when doing upper- or lowercase operation you're interested just in the 7-bit ASCII range of characters. And std::tolower/toupper is the wrong tool for that as well.

blenderob 2y ago

It is issues like this due to which I gave up on C++. There are so many ways to do something and every way is freaking wrong! An acceptable solution is given at the end of the article: > If you use the International Components for Unicode (ICU) library, you can use u_strToUpper and u_strToLower. Makes you wonder why this isn't part of the C++ standard library itself. Every revision of the C++ standard brings with itself more syntax and more complexity in the language. But as a user of C++ I don't need more syntax and more complexity in the language. But I do need more standard library functions that solves these ordinary real-world programming problems.

BoringTimesGang 2y ago
>It is issues like this due to which I gave up on C++. There are so many ways to do something and every way is freaking wrong! These are mostly unicode or linguistics problems.
- tralarpa 2y ago
  The fact that the standard library works against you doesn't help (to_lower takes an int, but only kind of works (sometimes) correctly on unsigned char, and wchar_t is implicitly promoted to int).
  - BoringTimesGang 2y ago
    to_lower is in the std namespace but is actually just part of the C89 standard, meaning it predates both UTF8 and UTF16. Is the alternative that it should be made unusable, and more existing code broken? A modern user has to include one of the c-prefix headers to use it, already hinting to them that 'here be dragons'. But there are always dragons. It's strings. The mere assumption that they can be transformed int-by-int, irrespective of encoding, is wrong. As is the assumption that a sensible transformation to lower case without error handling exists.
    - account42 2y ago
      > Is the alternative that it should be made unusable, and more existing code broken? It should be marked [[deprecated]], yes. There is no good reason to use std::tolower/toupper anywhere - they can neither do unicode properly nor are they anywhere close to efficient for ASCII. And their behavior depends on the process-global locale.

appointment 2y ago

The key takeaway here is that you can't correctly process a string if you don't what language it's in. That includes variants of the same language with different rules, eg en-US and en-UK or es-MX and es-ES. If you are handling multilingual text the locale is mandatory metadata.

zarzavat 2y ago
Different parts of a string can be in different languages too[1]. The lowercase of "DON'T FUSS ABOUT FUSSBALL" is "don't fuss about fußball". Unless you're in Switzerland. [1] https://en.wikipedia.org/wiki/Code-switching https://en.wikipedia.org/wiki/Code-switching
- schoen 2y ago
  Probably "don't fuss about Fußball" for the same reasons, right?
- thiht 2y ago
  I thought the German language deprecated the use of ß years ago, no? I learned German for a year and that's what the teacher told us, but maybe it's not the whole story
  - 47282847 2y ago
    Incorrect. ẞ is still a thing.
    - CamperBob2 2y ago
      Going by what you and the grandparent wrote, it's not just a thing, but two different things: ẞ ß It is probably time for an Esperanto advocate to show up and set us all straight.
      - D-Coder 2y ago
        Pri kio vi parolas? En Esperanto, unu letero egalas unu sonon. What are you talking about? In Esperanto, one letter equals one sound.

ahartmetz 2y ago

...and that is why you use QString if you are using the Qt framework. QString is a string class that actually does what you want when used in the obvious way. It probably helps that it was mostly created by people with "ASCII+" native languages. Or with customers that expect not exceedingly dumb behavior. The methods are called QString::toUpper() and QString::toLower() and take only the implicit "this" argument, unlike Win32 LCMapStringEx() which takes 5-8 arguments...

vardump 2y ago
You just want a banana, but you also get the gorilla. And the jungle.
cannam 2y ago
QString::toUpper/toLower are not locale-aware (https://doc.qt.io/qt-6/qstring.html#toLower https://doc.qt.io/qt-6/qstring.html#toLower) Qt does have a locale-aware equivalent (QLocale::toUpper/toLower) which calls out to ICU if available. Otherwise it falls back to the QString functions, so you have to be confident about how your build is configured. Whether it works or not has very little to do with the design of QString.
- ahartmetz 2y ago
  I don't see a problem with that. You can have it done locale-aware or not and "not" seems like a sane default. QString will uppercase 'ü' to 'Ü' just fine without locale-awareness whereas std::string doesn't handle non-ASCII according to the article. The cases where locale matters are probably very rare and the result will probably be reasonable anyway.
  - account42 2y ago
    That attitude is how you end up with exploits because your case folding is different from some other system you interact with.
aetherspawn 2y ago
I will admit I don’t love the Qt licensing model, but most things in Qt just work as they are supposed to, and on every platform too.

cyxxon 2y ago

Small nitpick: the example "LATIN SMALL LETTER SHARP S (“ß” U+00DF) uppercases to the two-character sequence “SS”:³ Straße ⇒ STRASSE" is slightly wrong, it seems to me, as we now do actually have a uppercase version of that, so it should uppercase to "Latin Capital Letter Sharp S" (U+1E9E). The double-S thing is still widely used, though.

mkayokay 2y ago
Duden mentions this: "Bei Verwendung von Großbuchstaben steht traditionellerweise SS für ß. In manchen Schriften gibt es aber auch einen entsprechenden Großbuchstaben; seine Verwendung ist fakultativ ‹§ 25 E3›." But isn't it also dependent on the available glyphs in the font used? So f.e. it needs to be ensured that U+1E9E exists?
- Kwpolska 2y ago
  I don't think there exists any code that makes uppercasing decisions based on the selected font. Besides, if it doesn't exist in the current font, there's probably a fallback font.
- NoInkling 2y ago
  According to Wikipedia: > "Since 2024 the capital ⟨ẞ⟩ is preferred over ⟨SS⟩." https://en.wikipedia.org/wiki/%C3%9F https://en.wikipedia.org/wiki/%C3%9F Check reference #5 and compare it to the older wording in reference #12.
Muromec 2y ago
But what if you need to uppercase the historical record in a vital records registry from 1950ies, but and OCRed last week? Now you need to not just be locale-aware, but you locale should be versioned.
pjmlp 2y ago
Lowering case is even better, because a Swiss user would expect the two-character sequence “SS“ to be converted into “ss“ and not “ß“. And thus we add country specific locale to the party.

serbuvlad 2y ago

The real insights here are that strings in C++ suck and UTF-16 is extremely unintuitive.

criddell 2y ago
Strings in C++ standard library do suck (and C++ is my favorite language). As for UTF-16, well, I don't know that UTF-8 is a whole lot more intuitive: > And for UTF-8 data, you have the same issues discussed before: Multibyte characters will not be converted properly, and it breaks for case mappings that alter string lengths.
- recursive 2y ago
  UTF-16 has all the complexity of UTF-8 plus surrogate pairs.
  - zahlman 2y ago
    Surrogate pairs aren't more complex than UTF-8's scheme for determining the number of bytes used to represent a code point. (Arguably the logic is slightly simpler.) But the important point is that UTF-16 pretends to be a constant-length encoding while actually having the surrogate-pair loophole - that's because it's a hack on top of UCS-2 (which originally worked well enough for Microsoft to get married to; but then the BMP turned out not to be enough code points). UTF-8 is clearly designed from scratch to be a multi-byte encoding (and, while the standard now makes the corresponding sequences illegal, the scheme was designed to be able to support much higher code points - up to 2^42 if we extend the logic all the way; hypothetical 6-byte sequences starting with values FC or FD would neatly map up to 2^31).

PhilipRoman 2y ago

Thought this was going to be about and-not-ing bytes with 0x20. Wrong for most inputs but sure as hell faster than anything else.

high_na_euv 2y ago

In cpp basic things are hard

onemoresoop 2y ago
It's subjective but I find C++ extremely ugly.
johnnyjeans 2y ago
nothing about working with locales, or text in general, is basic. we were decades into working with digital computers before we moved past switchboards and LEDs. don't take for granted just how high of a perch upon the shoulders of giants you have. that's exactly how the mistakes in the blog post get made.
- high_na_euv 2y ago
  Ive worked in various languages like C#, C and CPP and I know where Ive been fighting what kinds of problems. People always had some fancy reasoning about why things that should just work are not, but then a few years pass and things are improved. C++ is getting closer and closer to langs like C# in terms of making it harder to shot yourself, but still there is a huge room for improvement

SleepyMyroslav 2y ago

In gamedev there is simple rule: don't try to do any of that. If it is text game needs to show to user then every version of the text that is needed is a translated text. Programmer will never know if context or locale will need word order changes or anything complicated. Just trust the translation team. If text is coming from user - then change design until its not needed to 'convert'. There are major issues just to show user back what he entered! Because the font for editing and displayed text could be different. Not even mentioning RTL and other issues. Once ppl learn about localization the questions like why a programming language does not do this 'simple text operation' are just a newcomer detector. :)

zahlman 2y ago
>If text is coming from user - then change design until its not needed to 'convert' In games, you can possibly get away with this. Most other people need to worry about things like string collation (locale-aware sorting) for user-supplied text.
- makeitdouble 2y ago
  TBF, if you are caring about string collation, you're already at the entrance of the rabbit hole and probably should go down to the deep end anyway. I'd assume SleepyMyroslav doesn't apply to devs willing to spend weeks at time to handle all the complexity in full.
fluoridation 2y ago
>Once ppl learn about localization the questions like why a programming language does not do this 'simple text operation' are just a newcomer detector. :) I think you are purposefully misinterpreting the question. They're not asking about converting the case of any Unicode string with locale sensitivity, they're asking about converting the case of ASCII characters. What if your game needs to talk to a server and do some string manipulation in between requests? Are you really going to architect everything so that the client doesn't need to handle any of that ever?
- 2y ago

HPsquared 2y ago

I thought this was going to be about adding or subtracting 32. Old school.

klyrs 2y ago
I do hope you mean bitwise "addition" and "subtraction" -- (c => c&0xdf) or (c => c|0x20)
- HPsquared 2y ago
  Tbh I come at this as a plebeian Excel user

the_gorilla 2y ago

Why are some functions addressable in C++ and others not? Seems like a pointless design oversight.

bialpio 2y ago
Footnote in the article provides the following explanation: "The standard imposes this limitation because the implementation may need to add default function parameters, template default parameters, or overloads in order to accomplish the various requirements of the standard."

flareback 2y ago

He gave 4 examples of how it's done incorrectly, but zero actual examples of doing it correctly.

commandlinefan 2y ago
for (int i = 0; i < strlen(s); i++) { s[i] ^= 0x20; }
- vardump 2y ago
  Surely you meant: s[i] &= ~0x20; We're talking about converting to upper case after all! As an added benefit, every space character (0x20) is now a NUL byte!
  - account42 2y ago
    Free strtok!
- calibas 2y ago
  Thank you for this universal approach. I can now toggle capitalization on/off for any character, instead of just being limited to alphabetic ones! Jokes aside, I was kinda hoping for a good answer that doesn't rely on a Windows API or an external library, but I'm not sure there is one. It's a rather complex problem when you account for more than just ASCII and the English language.
  - TZubiri 2y ago
    Next up, check out our vector addition implementation of Hello+World. Spoiler alert, the result is Zalgo
TheGeminon 2y ago
> Okay, so those are the problems. What’s the solution? > If you need to perform a case mapping on a string, you can use LCMapStringEx with LCMAP_LOWERCASE or LCMAP_UPPERCASE, possibly with other flags like LCMAP_LINGUISTIC_CASING. If you use the International Components for Unicode (ICU) library, you can use u_strToUpper and u_strToLower.

deleted 2y ago

[deleted]

deleted 2y ago

[deleted]

PoignardAzur 2y ago

So I'm going to be that guy and say it: Man, I'm happy we don't need to deal with this crap in Rust, and we can just use String::to_lowercase. Not having to worry about things makes coding fun.

lilyball 2y ago
While certainly much better, you still need to be aware that doing case conversion absent any locale information will never be perfect. If you want proper locale-aware conversion you can use the icu crate (https://docs.rs/icu/latest/icu/ https://docs.rs/icu/latest/icu/).
- account42 2y ago
  Exactly, simple "unicode-aware" case conversions are a trap. You are always going to need much more.

himinlomax 2y ago

> And in certain forms of the French language, capitalizing an accented character causes the accent to be dropped: à Paris ⇒ A PARIS. That's incorrect, using diacritics on capital letters is always the preferred form, it's just that dropping them is acceptable as it was often done for technical reasons.

codr7 2y ago

C++, where every line of code is a book waiting to be written.

deleted 2y ago

[deleted]

zzo38computer 2y ago

First, you should consider if you even need case folding; for many uses it will be unnecessary, anyways. Furthermore, the proper way to do case folding will depend on such things as the character set, the language, the specific context of the text being converted (e.g. in some cases specific letters are required, such as abbreviations of the names of SI units), etc. And then, it is not necessarily only "uppercase" and "lowercase", anyways. There might even be different ways to do by the same language, with possibly disagreements about usage (e.g. the German Eszett did not have an official capital form until 2017, although apparently some type designers did it anyways (and it was in Unicode before then, despite that)). If the character set is Unicode, then there is not actually the correct way to do it, despite what the Unicode Conspiracy insists otherwise. Also, for some uses the way that it will need to be done, there will be a specific way that it is required (due to the way that a file format or a protocol or whatever is working), so in such a case if the character set is something other than ASCII then you cannot just assume that it will always work in the same way. You also cannot necessarily depend on the locale for such a thing, since it might depend on the data, as well. These things can be as bad as they are, but Unicode just makes these things worse than that. If a program requires a specific case folding and then it will not work because it is the wrong version of Unicode and it is possible to be a security issue and/or other problems. (Another problem, which applies even if you do not use case folding, is that some people think that all text is or should be Unicode and that one character set is suitable for everything. Actually, one character set cannot be suitable for everything, regardless of what character set it is. Even if it was (which it isn't), it wouldn't be Unicode.)

alkonaut 2y ago

Handle text in two ways: either it's controlled by you and you can do simple, efficient, and naive processing, or it's not (it's translated resources, or user input) and you can't. For the former case, you don't need any complex logic. A very typical example would be: i'm serializing a field or constructing a url so I want the variable name "Someproperty" as a lower case string. The lowercase transform is completely naive. I know exactly what the range of possible characters are and they aren't going to be Turkish or emoji, not least because I have asserted they won't be. And THIS is what the regular programming functions for upper/lower case are for. They are important, and they are most often correct. Because for all the other cases (i18n, user input, ...) you probably don't want to do toUpper/toLower at all to begin with! Example, if you present a message to the user from resources so your code is translate("USER_DIALOG_QUESTION_ABOUT_FISH") which you want to lookup knowing it will be in sentence case, and present as uppercase, what will you do? Here you likely can't, and shouldn't, do toUpper(translate(resourceKey)). Just use two resources if you want to correctly transform text. The toUpper function isn't made for this. Trying to use a complex i18n-ready toUpper/toLower only helps part of the way. It still might not understand whether two S are contracted or whether something is a proper Noun and must stay capitalized. So it adds complexity and still isn't correct. Just use two resources!

account42 2y ago
> For the former case, you don't need any complex logic. A very typical example would be: i'm serializing a field or constructing a url so I want the variable name "Someproperty" as a lower case string. The lowercase transform is completely naive. I know exactly what the range of possible characters are and they aren't going to be Turkish or emoji, not least because I have asserted they won't be. And THIS is what the regular programming functions for upper/lower case are for. They are important, and they are most often correct. Because for all the other cases (i18n, user input, ...) you probably don't want to do toUpper/toLower at all to begin with! C++ std::tolower/toupper (which are really just C tolower/toupper) are the wrong tool for that too though because they depend on the process locale which makes them a) horribly inefficient and b) prone to blow your program up in interesting ways on customer systems. Not quite as bad as the locale-dependent standard number parsing functions that want . in some localses and , in others but still should never be used.

Animats 2y ago

> From the article: "I find it quaint that Unicode character names are ALL IN CAPITAL LETTERS, in case you need to put them in a Baudot telegram or something." I had to do that. When we had our steampunk telegraph office at steampunk conventions [1], people could text in a message via SMS, it would be printed on a Model 14 or 15 Teletype, put in an envelope, and hand-delivered. People would use emoji in messages, and the device could only print Baudot, or International Telegraphic Alphabet #2, which is upper case only with some symbols. Emoji translation would cause the machine to hammer out (RED-HEART) or whatever emoji description was needed. Used the emoji list at [2], an older version. [1] https://vimeo.com/124065314 https://vimeo.com/124065314 [2] http://unicode.org/emoji/charts-beta/full-emoji-list.html http://unicode.org/emoji/charts-beta/full-emoji-list.html

guerrilla 2y ago

C is hard. It seems like C++ just made things way harder. I don't regret skipping it. Why not just go right to Java, C#, JS, Haskell, etc. and do what you need in C.

ChrisMarshallNY 2y ago

I generally just use the language-supported tolower/upper() (or similar) routines. I assume that they take things like UTF and alternative type systems into account. I'm not sure about other languages, but Swift has pretty intense String support[0], and can go quite a long ways. Someone actually wrote a whole book about just Swift Strings[1]. [0] https://docs.swift.org/swift-book/documentation/the-swift-programming-language/stringsandcharacters/ https://docs.swift.org/swift-book/documentation/the-swift-pr... [1] https://flight.school/books/strings/ https://flight.school/books/strings/

deleted 2y ago

[deleted]

account42 2y ago

A popular but wrong way to do Unicode > wchar_t