3 ms·

Aha, I have uncovered it. There are two problems. One is that apparently std::locale is horribly, horribly broken in the C++ standard library for Mac OS X. Y

by gchpaco 17y ago

Aha, I have uncovered it. There are two problems. One is that apparently std::locale is horribly, horribly broken in the C++ standard library for Mac OS X. You can work around this by using C, like Real Programmers. The other problem is that without some form of setlocale, the standard library will run in "C" or "POSIX" mode, which can do nothing at all interesting with Unicode. By running setlocale, or by running std::locale::global (std::locale ("")) on a platform with a C++ library that works, it will print appropriately. Interestingly on my Ubuntu box, before I put the std::locale in it was printing "I have EUR100 to my name.", which is a pretty cool fallback.

timr 17y ago
Yes, the article title is wildly unfair. But an article titled "the g++ std::locale implementation under Mac OSX is broken" isn't nearly as interesting as claiming that C++ might not be able to support Unicode. I mean, really...C++ can work with individual bits; it can obviously support Unicode strings. The problem here is one of making the compiler aware of the details of the platform on which it's being used. The blame here appears to lie on Apple: http://lists.apple.com/archives/darwin-dev/2008/Mar/msg00015.html http://lists.apple.com/archives/darwin-dev/2008/Mar/msg00015... http://lists.apple.com/archives/unix-porting/2006/Dec/msg00007.html http://lists.apple.com/archives/unix-porting/2006/Dec/msg000...
- gchpaco 17y ago
  Yes, that's right, for those just tuning in, this has been broken since at least 2006. Probably earlier. I'm just speechless.
  - deleted 17y ago
    [deleted]
fauigerzigerk 17y ago
setlocale() in C and C++ affects the entire process. Setting it to anything other than the "C" locale very likely breaks stuff elsewhere and it's completely unpredictable. Realistically, for serious text processing you need to leave the entire broken POSIX locale system and C++ wchar_t alone and work with libraries such as ICU.
- timr 17y ago
  ...which isn't the end of the world, since ICU comes with a bunch of stuff that makes life bearable when developing international software, like language-aware collation and regular expressions. If you're going to be developing real i18n software in C++, you're always going to be using ICU or something like it. There's no reason to bother with std::locale. In fact, I'm guessing that Apple never got around to fixing this because they probably don't use std::locale for real i18n work internally. (update: the ICU homepage says that Mac OSX uses ICU internally -- http://site.icu-project.org/ http://site.icu-project.org/)
  - fauigerzigerk 17y ago
    Seems my post was very unclear and I apologize for that. What I meant to say is: Try to avoid POSIX locales at all cost. Use ICU!
- gchpaco 17y ago
  I'm guessing you're an American, because everybody else is very ill served by the C/POSIX locale. Locales are about something different than general text munging, and if setting the locale to something other than C (which, BTW, is done by default on Ubuntu since many years ago; been en_US.UTF-8 since as far back as I can remember) broke software, that software is seriously stupidly designed. Locales are about making the date come out in the order the user was expecting, the default text encoding, error messages. They are fundamentally about interfacing to the user in the way that the user expects. And yes, they are flawed, seriously so, mostly due to a lack of ambition. setlocale assumes that the locale the user is currently working in is also the locale the user has always been working in since time immemorial, and it assumes there is only one user. If you have a multilingual document, you're in trouble. If you use the locale date/time printer for anything other than immediate input and output to the user, you're in trouble (and there are circles of hell that certain Usenet newsreader authors will burn in for all eternity for this one). Any time you write a file out to disk with one locale and read it in with another you are potentially in horrible danger. This is where something like ICU comes in really handy; any time you have to deal with data from several different origins, you will be ill served by locale. So in that sense, you're quite right. But all this is a very different problem from making printf work the way it is advertised as doing, from making it possible to output a Unicode string that you have told the computer in as many ways that you wish to output in UTF-8, write in UTF-8. If you want to do simple text output and not deal with ICU, GNU recode, whatever, you need to use setlocale or your libc will mutely suppress everything but 7-bit ASCII, and you will be very, very confused.

by gchpaco 17y ago

timr 17y ago

Yes, the article title is wildly unfair. But an article titled "the g++ std::locale implementation under Mac OSX is broken" isn't nearly as interesting as claiming that C++ might not be able to support Unicode. I mean, really...C++ can work with individual bits; it can obviously support Unicode strings. The problem here is one of making the compiler aware of the details of the platform on which it's being used. The blame here appears to lie on Apple: http://lists.apple.com/archives/darwin-dev/2008/Mar/msg00015.html http://lists.apple.com/archives/darwin-dev/2008/Mar/msg00015... http://lists.apple.com/archives/unix-porting/2006/Dec/msg00007.html http://lists.apple.com/archives/unix-porting/2006/Dec/msg000...

gchpaco 17y ago
Yes, that's right, for those just tuning in, this has been broken since at least 2006. Probably earlier. I'm just speechless.
- deleted 17y ago
  [deleted]

fauigerzigerk 17y ago

setlocale() in C and C++ affects the entire process. Setting it to anything other than the "C" locale very likely breaks stuff elsewhere and it's completely unpredictable. Realistically, for serious text processing you need to leave the entire broken POSIX locale system and C++ wchar_t alone and work with libraries such as ICU.

timr 17y ago
...which isn't the end of the world, since ICU comes with a bunch of stuff that makes life bearable when developing international software, like language-aware collation and regular expressions. If you're going to be developing real i18n software in C++, you're always going to be using ICU or something like it. There's no reason to bother with std::locale. In fact, I'm guessing that Apple never got around to fixing this because they probably don't use std::locale for real i18n work internally. (update: the ICU homepage says that Mac OSX uses ICU internally -- http://site.icu-project.org/ http://site.icu-project.org/)
- fauigerzigerk 17y ago
  Seems my post was very unclear and I apologize for that. What I meant to say is: Try to avoid POSIX locales at all cost. Use ICU!
gchpaco 17y ago
I'm guessing you're an American, because everybody else is very ill served by the C/POSIX locale. Locales are about something different than general text munging, and if setting the locale to something other than C (which, BTW, is done by default on Ubuntu since many years ago; been en_US.UTF-8 since as far back as I can remember) broke software, that software is seriously stupidly designed. Locales are about making the date come out in the order the user was expecting, the default text encoding, error messages. They are fundamentally about interfacing to the user in the way that the user expects. And yes, they are flawed, seriously so, mostly due to a lack of ambition. setlocale assumes that the locale the user is currently working in is also the locale the user has always been working in since time immemorial, and it assumes there is only one user. If you have a multilingual document, you're in trouble. If you use the locale date/time printer for anything other than immediate input and output to the user, you're in trouble (and there are circles of hell that certain Usenet newsreader authors will burn in for all eternity for this one). Any time you write a file out to disk with one locale and read it in with another you are potentially in horrible danger. This is where something like ICU comes in really handy; any time you have to deal with data from several different origins, you will be ill served by locale. So in that sense, you're quite right. But all this is a very different problem from making printf work the way it is advertised as doing, from making it possible to output a Unicode string that you have told the computer in as many ways that you wish to output in UTF-8, write in UTF-8. If you want to do simple text output and not deal with ICU, GNU recode, whatever, you need to use setlocale or your libc will mutely suppress everything but 7-bit ASCII, and you will be very, very confused.