7 ms·

As always, Raymond is right. (And as usually, I could guess it's him before even clicking the link.) That said, 99% time when doing upper- or lowercase operati

As always, Raymond is right. (And as usually, I could guess it's him before even clicking the link.) That said, 99% time when doing upper- or lowercase operation you're interested just in the 7-bit ASCII range of characters. For the remaining 1%, there's ICU library. Just like Raymond Chen mentioned.

fhars 2y ago
No, when you are doing string manipulation, you are almost never interestet in just the seven bit ASCII range, as there is almost no language that can be written using just that.
- daemin 2y ago
  I would argue that for most programs when you're doing string manipulation you're doing it for internal programming reasons - logs, error messages, etc. In that case you are in nearly full control of the strings and therefore can declare that you're only working with ASCII. The other normal cases of string usage are file paths and user interface, and the needed operations can be done with simple string functions, and even in UTF8 encoding the characters you care about are in the ASCII range. With file paths the manipulations that you're most often doing is path based so you only care about '/', '\', ':', and '.' ASCII characters. With user interface elements you're likely to be using them as just static data and only substituting values into placeholders when necessary.
  - pistoleer 2y ago
    > I would argue that for most programs when you're doing string manipulation you're doing it for internal programming reasons - logs, error messages, etc. In that case you are in nearly full control of the strings and therefore can declare that you're only working with ASCII. Why would you argue that? In my experience it's about formatting things that are addressed to the user, where the hardest and most annoying localization problems matter a lot. That includes sorting the last name "van den Berg" just after "Bakker", stylizing it as "Berg, van den", and making sure this capitalization is correct and not "Van Den Berg". There is no built in standard library function in any language that does any of that. It's so much larger than ascii and even larger than unicode. Another user said that the main takeaway is that you can't process strings until you know their language (locale), and that is exactly correct.
    - daemin 2y ago
      I would maintain that your program has more string manipulation for error messages and logging than for generating localised formatted names. Further I do say that if you're creating text for presenting to the user then the most common operation would be replacement of some field in pre-defined text. In your case I would design it so that the correctly capitalised first name, surname, and variations of those for sorting would be generated at the data entry point (manually or automatically) and then just used when needed in user facing text generation. Therefore the only string operation needed would be replacement of placeholders like the fmt and standard library provide. This uses more memory and storage but these are cheaper now.
      - pistoleer 2y ago
        I agree, but the logging formatters don't really do much beyond trivially pasting in placeholders. And as for data entry... Maybe in an ideal world. In the current world, marred by importing previously mangled datasets, a common solution in the few companies I've worked at is to just not do anything, which leaves ugly edges, yet is "good enough".
  - BoringTimesGang 2y ago
    Now double all of that effort, so you can get it to work with Windows' UTF-16 wstrings.
    - account42 2y ago
      Better to just convert WTF-16 (Windows filenames re not guaranteed to be valid UTF-16) to/from WTF-8 at the API boundary and then do the same processing internally on all platforms.
  - heisenzombie 2y ago
    File paths? I think filesystem paths are generally “bags of bytes” that the OS might interpret as UTF-16 (Windows) or UTF-8 (macOS, Linux). For example: https://en.m.wikipedia.org/wiki/Program_Files#Localization https://en.m.wikipedia.org/wiki/Program_Files#Localization
    - vardump 2y ago
      File paths are scary. The last I checked (which is admittedly a while ago), Windows didn't for example care about correct UTF-16 surrogate pairs at all, it'd happily accept invalid UTF-16 strings. So use standard string processing libraries on path names at your own peril. It's a good idea to consider file paths as a bag of bytes.
      - Someone 2y ago
        > It's a good idea to consider file paths as a bag of bytes (Nitpick: sequence of bytes) Also very limiting. If you do that, you can’t, for example, show a file name to the user as a string or easily use a shell to process data in your file system (do you type “/bin” or “\x2F\x62\x69\x6E”?) Unix, from the start, claimed file names where byte sequences, yet assumed many of those to encode ascii. That’s part of why Plan 9 made the choice “names may contain any printable character (that is, any character outside hexadecimal 00-1F and 80-9F)” (https://9fans.github.io/plan9port/man/man9/intro.html https://9fans.github.io/plan9port/man/man9/intro.html)
      - daemin 2y ago
        That's what I mean, you treat filesystem paths as bags of bytes separated by known ASCII characters, as the only path manipulation that you generally need to do is to append a path, remove a path, change extension, things that only care about those ASCII characters. You only modify the path strings at those known characters and leave everything in between as is (with some exceptions using OS API specific functions as needed).
      - netsharc 2y ago
        IIRC, the FAT filesystem (before Windows 95) allowed lowercase letters, but there's a layer in the filesystem driver that converted everything to uppercase, e.g. if you did the command "more readme.txt", the more command would ask the filesystem for "readme.txt" and it would search for "README.TXT" in the file allocation table. I think I once hex-edited the FA-table to change a filename to have a lowercase name (or maybe it was disk corruption), trying to delete that file didn't work because it would be trying to delete "FOO", and couldn't find it because the file was named "FOo".
- vardump 2y ago
  > as there is almost no language that can be written using just that. 99% of use cases I've seen have nothing to do with human language. 1% human language case that is needs to be handled properly using a proper Unicode library. Your mileage (percentages) may vary depending on your job.
  - 9dev 2y ago
    It's funny how software developers live in bubbles so much. Whether you deal with human language a lot or almost not at all depends entirely on your specific domain. Anyone working on user interfaces of any kind must accommodate for proper encoding, for example; that includes pretty much every line-of-business app out there, which is a lot of code.
  - inexcf 2y ago
    Why do you need upper- or lowercase conversion in cases that have nothing to do with human language?
    - vardump 2y ago
      Here's an example. Hypothetically say you want to build an HTML parser. You might encounter tags like <html>, <HTML>, <Html>, etc., but you want to perform a hash table lookup. So first you're going to normalize to either lower- or uppercase.
      - Muromec 2y ago
        But but, I want to have a custom web component and register it under my own name, which can only be properly written in Ukrainian Cyrillic. How dare you not let me have it.
      - inexcf 2y ago
        Ah, i see, we disagree on what is "human language". An abbreviation like HTML and it's different capitalisations to me sound a lot like a feature of human language.
sebstefan 2y ago
Yes please, keep making software that mangles my actual last name at every step of the way. 99% of the world loves it when you only care about the USA.
- Muromec 2y ago
  If it needs to uppercase names it probably interfaces with something forsaken like Sabre/Amadeus that only understands ASCII anyway. The real problem is accepting non-ASCII input from user where you later assume it's ASCII-only and safe to bitfuck around.
  - sebstefan 2y ago
    From experience anything banking adjacent will usually fuck it up as well For some reason they have a hard-on for putting last names in capital letters and they still have systems in place that use ASCII
    - Muromec 2y ago
      If it uses ASCII anyway, what's the problem then? Don't accept non-ASCII user input.
      - sebstefan 2y ago
        First off: And exclude 70% of the world? Usually they'll accept it, but some parts of the backend are still running code from the 60's. So you get your name rendered properly on the web interface, and most core features, but one day you're wandering off from the beaten path, by, like, requesting some insurance contract, and you'll see your name at the top with some characters mangled, depending on what your name's like. Mine is just accented latin characters so it usually drops the accents ; not sure how it would work if your name was in an entirely different alphabet
        Muromec 2y ago
        >First off: And exclude 70% of the world? Guess what, I'm part of this 70% and I also work in a bank and I know exactly how. Not a single letter in my name (any of them) can be represented with ASCII. When it is represented in UTF-8, most of the people who have to see it can't read it anyway. So my identity document issued by the country which doesn't use Latin alphabet includes ASCII-representation of my name in addition to canonical form in Ukrainian Cyrillic. That ASCII-rendering is happily accepted by all kinds of systems that only speak ASCII. People still can't pronounce it and it got misspelled like yesterday when dictated over the phone. Now regarding the accents, it's illegal to not support them per GDPR (as per case law, discussed here few years ago).
crazygringo 2y ago
> That said, 99% time when doing upper- or lowercase operation you're interested just in the 7-bit ASCII range of characters. I think it's more the exact opposite. The only times I'm dealing with 7-bit ASCII is for internal identifiers like variable names or API endpoints. Which is a lot of the time, but I can't ever think of when I've needed my code to change their case. It might literally be never. On the other hand, needing to switch between upper, lower, and title case happens all the time, always with people's names and article titles and product names and whatnot. Which are never in ASCII because this isn't 1990.
- hinkley 2y ago
  And you could argue that if the internal identifiers need to be capitalized or lower-cased, you've already lost. On an enterprise app these little string manipulations are a drop in the bucket. In a game they might not be. Sort that stuff out at compile time, or commit time.
  - account42 2y ago
    You can't always control the case you get but often you can not care about anything outside ASCII. Scripts and configuration or text-based data formats are common examples.
- bigstrat2003 2y ago
  > Which are never in ASCII because this isn't 1990. This is a very silly statement. I'm willing to believe that you have lots of cases where those things are outside the ASCII range. Perhaps even most of the cases, depending on where you live. But I do not believe for one second that it never happens.
  - crazygringo 2y ago
    Never stored in ASCII, never limited to ASCII. They're UTF-8, usually. If somebody's name happens to fit into ASCII that's irrelevant because it's not guaranteed, so you can never blindly do an ASCII case conversion. For text data meant for users, I literally cannot remember the last time I used a string in ASCII format as opposed to UTF-8 (or UTF-16 in JS). It's certainly over a decade ago. So yes, when I say never, I literally mean never. Nothing "very silly" about it, sorry. (Again, excepting identifiers, where case conversion is not generally applicable.)
PaulDavisThe1st 2y ago
He may be right, but approximately 75% of the problems he describes are all Microsoft-ecosystem specific. In Unix-land we don't use wchar_t or UTF-16, and his article is a good demonstrations of why not.
- pjmlp 2y ago
  UNIX land is even worse in international languages support. As in, there isn't even something on POSIX at the level other operaring systems support for localisation. Yes there is some locale stuff, however not enough for all stuff, hence why every modern programming language happens to have this as part of their standard library.
  - PaulDavisThe1st 2y ago
    When the notion of what a "string" is differs so much from language to language, i18n is never going to be an effective part of POSIX.
  - account42 2y ago
    Is there a platform where you can't use ICU?
yas_hmaheshwari 2y ago
Wow, I came here to write exactly that, and its heartening to see that I am not crazy Just reading the title, with microsoft.com in bracket, I knew two things: 1. It would be written by Raymond Chen 2. That article is going to be awesome
account42 2y ago
> That said, 99% time when doing upper- or lowercase operation you're interested just in the 7-bit ASCII range of characters. And std::tolower/toupper is the wrong tool for that as well.

fhars 2y ago

No, when you are doing string manipulation, you are almost never interestet in just the seven bit ASCII range, as there is almost no language that can be written using just that.

daemin 2y ago
I would argue that for most programs when you're doing string manipulation you're doing it for internal programming reasons - logs, error messages, etc. In that case you are in nearly full control of the strings and therefore can declare that you're only working with ASCII. The other normal cases of string usage are file paths and user interface, and the needed operations can be done with simple string functions, and even in UTF8 encoding the characters you care about are in the ASCII range. With file paths the manipulations that you're most often doing is path based so you only care about '/', '\', ':', and '.' ASCII characters. With user interface elements you're likely to be using them as just static data and only substituting values into placeholders when necessary.
- pistoleer 2y ago
  > I would argue that for most programs when you're doing string manipulation you're doing it for internal programming reasons - logs, error messages, etc. In that case you are in nearly full control of the strings and therefore can declare that you're only working with ASCII. Why would you argue that? In my experience it's about formatting things that are addressed to the user, where the hardest and most annoying localization problems matter a lot. That includes sorting the last name "van den Berg" just after "Bakker", stylizing it as "Berg, van den", and making sure this capitalization is correct and not "Van Den Berg". There is no built in standard library function in any language that does any of that. It's so much larger than ascii and even larger than unicode. Another user said that the main takeaway is that you can't process strings until you know their language (locale), and that is exactly correct.
  - daemin 2y ago
    I would maintain that your program has more string manipulation for error messages and logging than for generating localised formatted names. Further I do say that if you're creating text for presenting to the user then the most common operation would be replacement of some field in pre-defined text. In your case I would design it so that the correctly capitalised first name, surname, and variations of those for sorting would be generated at the data entry point (manually or automatically) and then just used when needed in user facing text generation. Therefore the only string operation needed would be replacement of placeholders like the fmt and standard library provide. This uses more memory and storage but these are cheaper now.
    - pistoleer 2y ago
      I agree, but the logging formatters don't really do much beyond trivially pasting in placeholders. And as for data entry... Maybe in an ideal world. In the current world, marred by importing previously mangled datasets, a common solution in the few companies I've worked at is to just not do anything, which leaves ugly edges, yet is "good enough".
- BoringTimesGang 2y ago
  Now double all of that effort, so you can get it to work with Windows' UTF-16 wstrings.
  - account42 2y ago
    Better to just convert WTF-16 (Windows filenames re not guaranteed to be valid UTF-16) to/from WTF-8 at the API boundary and then do the same processing internally on all platforms.
- heisenzombie 2y ago
  File paths? I think filesystem paths are generally “bags of bytes” that the OS might interpret as UTF-16 (Windows) or UTF-8 (macOS, Linux). For example: https://en.m.wikipedia.org/wiki/Program_Files#Localization https://en.m.wikipedia.org/wiki/Program_Files#Localization
  - vardump 2y ago
    File paths are scary. The last I checked (which is admittedly a while ago), Windows didn't for example care about correct UTF-16 surrogate pairs at all, it'd happily accept invalid UTF-16 strings. So use standard string processing libraries on path names at your own peril. It's a good idea to consider file paths as a bag of bytes.
    - Someone 2y ago
      > It's a good idea to consider file paths as a bag of bytes (Nitpick: sequence of bytes) Also very limiting. If you do that, you can’t, for example, show a file name to the user as a string or easily use a shell to process data in your file system (do you type “/bin” or “\x2F\x62\x69\x6E”?) Unix, from the start, claimed file names where byte sequences, yet assumed many of those to encode ascii. That’s part of why Plan 9 made the choice “names may contain any printable character (that is, any character outside hexadecimal 00-1F and 80-9F)” (https://9fans.github.io/plan9port/man/man9/intro.html https://9fans.github.io/plan9port/man/man9/intro.html)
    - daemin 2y ago
      That's what I mean, you treat filesystem paths as bags of bytes separated by known ASCII characters, as the only path manipulation that you generally need to do is to append a path, remove a path, change extension, things that only care about those ASCII characters. You only modify the path strings at those known characters and leave everything in between as is (with some exceptions using OS API specific functions as needed).
    - netsharc 2y ago
      IIRC, the FAT filesystem (before Windows 95) allowed lowercase letters, but there's a layer in the filesystem driver that converted everything to uppercase, e.g. if you did the command "more readme.txt", the more command would ask the filesystem for "readme.txt" and it would search for "README.TXT" in the file allocation table. I think I once hex-edited the FA-table to change a filename to have a lowercase name (or maybe it was disk corruption), trying to delete that file didn't work because it would be trying to delete "FOO", and couldn't find it because the file was named "FOo".
vardump 2y ago
> as there is almost no language that can be written using just that. 99% of use cases I've seen have nothing to do with human language. 1% human language case that is needs to be handled properly using a proper Unicode library. Your mileage (percentages) may vary depending on your job.
- 9dev 2y ago
  It's funny how software developers live in bubbles so much. Whether you deal with human language a lot or almost not at all depends entirely on your specific domain. Anyone working on user interfaces of any kind must accommodate for proper encoding, for example; that includes pretty much every line-of-business app out there, which is a lot of code.
- inexcf 2y ago
  Why do you need upper- or lowercase conversion in cases that have nothing to do with human language?
  - vardump 2y ago
    Here's an example. Hypothetically say you want to build an HTML parser. You might encounter tags like <html>, <HTML>, <Html>, etc., but you want to perform a hash table lookup. So first you're going to normalize to either lower- or uppercase.
    - Muromec 2y ago
      But but, I want to have a custom web component and register it under my own name, which can only be properly written in Ukrainian Cyrillic. How dare you not let me have it.
    - inexcf 2y ago
      Ah, i see, we disagree on what is "human language". An abbreviation like HTML and it's different capitalisations to me sound a lot like a feature of human language.

sebstefan 2y ago

Yes please, keep making software that mangles my actual last name at every step of the way. 99% of the world loves it when you only care about the USA.

Muromec 2y ago
If it needs to uppercase names it probably interfaces with something forsaken like Sabre/Amadeus that only understands ASCII anyway. The real problem is accepting non-ASCII input from user where you later assume it's ASCII-only and safe to bitfuck around.
- sebstefan 2y ago
  From experience anything banking adjacent will usually fuck it up as well For some reason they have a hard-on for putting last names in capital letters and they still have systems in place that use ASCII
  - Muromec 2y ago
    If it uses ASCII anyway, what's the problem then? Don't accept non-ASCII user input.
    - sebstefan 2y ago
      First off: And exclude 70% of the world? Usually they'll accept it, but some parts of the backend are still running code from the 60's. So you get your name rendered properly on the web interface, and most core features, but one day you're wandering off from the beaten path, by, like, requesting some insurance contract, and you'll see your name at the top with some characters mangled, depending on what your name's like. Mine is just accented latin characters so it usually drops the accents ; not sure how it would work if your name was in an entirely different alphabet
      - Muromec 2y ago
        >First off: And exclude 70% of the world? Guess what, I'm part of this 70% and I also work in a bank and I know exactly how. Not a single letter in my name (any of them) can be represented with ASCII. When it is represented in UTF-8, most of the people who have to see it can't read it anyway. So my identity document issued by the country which doesn't use Latin alphabet includes ASCII-representation of my name in addition to canonical form in Ukrainian Cyrillic. That ASCII-rendering is happily accepted by all kinds of systems that only speak ASCII. People still can't pronounce it and it got misspelled like yesterday when dictated over the phone. Now regarding the accents, it's illegal to not support them per GDPR (as per case law, discussed here few years ago).

crazygringo 2y ago

> That said, 99% time when doing upper- or lowercase operation you're interested just in the 7-bit ASCII range of characters. I think it's more the exact opposite. The only times I'm dealing with 7-bit ASCII is for internal identifiers like variable names or API endpoints. Which is a lot of the time, but I can't ever think of when I've needed my code to change their case. It might literally be never. On the other hand, needing to switch between upper, lower, and title case happens all the time, always with people's names and article titles and product names and whatnot. Which are never in ASCII because this isn't 1990.

hinkley 2y ago
And you could argue that if the internal identifiers need to be capitalized or lower-cased, you've already lost. On an enterprise app these little string manipulations are a drop in the bucket. In a game they might not be. Sort that stuff out at compile time, or commit time.
- account42 2y ago
  You can't always control the case you get but often you can not care about anything outside ASCII. Scripts and configuration or text-based data formats are common examples.
bigstrat2003 2y ago
> Which are never in ASCII because this isn't 1990. This is a very silly statement. I'm willing to believe that you have lots of cases where those things are outside the ASCII range. Perhaps even most of the cases, depending on where you live. But I do not believe for one second that it never happens.
- crazygringo 2y ago
  Never stored in ASCII, never limited to ASCII. They're UTF-8, usually. If somebody's name happens to fit into ASCII that's irrelevant because it's not guaranteed, so you can never blindly do an ASCII case conversion. For text data meant for users, I literally cannot remember the last time I used a string in ASCII format as opposed to UTF-8 (or UTF-16 in JS). It's certainly over a decade ago. So yes, when I say never, I literally mean never. Nothing "very silly" about it, sorry. (Again, excepting identifiers, where case conversion is not generally applicable.)

PaulDavisThe1st 2y ago

He may be right, but approximately 75% of the problems he describes are all Microsoft-ecosystem specific. In Unix-land we don't use wchar_t or UTF-16, and his article is a good demonstrations of why not.

pjmlp 2y ago
UNIX land is even worse in international languages support. As in, there isn't even something on POSIX at the level other operaring systems support for localisation. Yes there is some locale stuff, however not enough for all stuff, hence why every modern programming language happens to have this as part of their standard library.
- PaulDavisThe1st 2y ago
  When the notion of what a "string" is differs so much from language to language, i18n is never going to be an effective part of POSIX.
- account42 2y ago
  Is there a platform where you can't use ICU?

yas_hmaheshwari 2y ago

Wow, I came here to write exactly that, and its heartening to see that I am not crazy Just reading the title, with microsoft.com in bracket, I knew two things: 1. It would be written by Raymond Chen 2. That article is going to be awesome

account42 2y ago

> That said, 99% time when doing upper- or lowercase operation you're interested just in the 7-bit ASCII range of characters. And std::tolower/toupper is the wrong tool for that as well.