21 ms·

OT question: I was playing with some C over the weekend (not C++), trying to figure out how to handle Unicode in a way that would work on Mac, Linux, and Window

OT question: I was playing with some C over the weekend (not C++), trying to figure out how to handle Unicode in a way that would work on Mac, Linux, and Windows. Despite a couple hours googling and reading, I couldn't answer really basic stuff, like: - Do I use char* for strings? It sounds like wchar_t is 16 bits on some systems and 32 on others, so I should avoid it? - If I want to read & write UTF-8 files, how do I turn the bytes into 32-bit wide Unicode strings? - Are there special functions I should use for handling Unicode strings? More generally, where do you go to discover C libraries you can use? I am (was) reasonably proficient in C, but I haven't used it much for over ten years. I'm surprised how many things I just don't know how to do! Sorry this is not a C++ question. I'd like to get back into that also, but I'm trying to work my way up from the basics. :-)

wisty 12y ago
Look here: http://userguide.icu-project.org http://userguide.icu-project.org You really need a library when dealing with unicode, if you are doing anything advanced. How do you split a sentence into words? Spaces, right? Oh, Chinese doesn't have spaces. How about sentences? Paragraphs? Even characters are a pain to iterate over.
- Nursie 12y ago
  Agreed, ICU is a good place to start looking at this stuff, used it in a few large-scale commercial projects now.
- jonstewart 12y ago
  It really is abominable how bad the standard support for Unicode is. ICU is great, though the C API is somewhat limited compared to the full C++ API. ICU's build process, OTOH, leaves something to be desired, especially if you want to build it with a cross-compiler.
zanny 12y ago
Why not just write C style C++? I never understand the arguments to use C anymore - you can write C style C++ and get on with your life, and cherry pick the features you want. In C++ unicode is as simple as: string s = u8"This is always a utf8 string.";
- pmelendez 12y ago
  Well there are people who really hates C++[1] and to be honest C++03 had some flaws an inconveniences that made the language a bit awkward. Right now it feels like it is going in the right direction. [1]http://article.gmane.org/gmane.comp.version-control.git/57918 http://article.gmane.org/gmane.comp.version-control.git/5791...
- akira2501 12y ago
  Is 'extern "C"' still a thing to avoid name mangling? Some C++ features are easier not to use than others.
  - mpyne 12y ago
    Yes, extern "C" is still a thing to interoperate with the C FFI. It is required if for nothing else than to use C headers.
- conistonwater 12y ago
  The main problem with using C++ as C is that everyone has different ideas about which ideas to cherry pick. So without a lot of discipline and a good coding standard, lots of (maybe subjectively) bad ideas will make it into the code as well. And once they do, you can't do anything about them any more.
- programmarchy 12y ago
  Firmware development. For some microcontrollers, you'll be lucky to even have a C compiler let alone a C++ compiler. Even then, I'd rather use C than C++ for firmware. I often find myself running into odd compiler issues, having to inspect/modify assembly, deal with resource constraints, etc. and I imagine C++ would make that more difficult.
- millstone 12y ago
  Unicode is much more complex than that. For example, how would you then convert that string to uppercase, or truncate it? For Unicode strings in C++, you can either limp along with the wchar_t support, or leverage something like ICU. Neither is especially easy.
maxlybbert 12y ago
C11 has the uchar.h header (say, http://www.cplusplus.com/reference/cuchar/ http://www.cplusplus.com/reference/cuchar/ ), but it wasn't in Visual Studio last I checked. Most programmers will tell you to use UTF-8 for in-memory strings. But it's not easy to figure out if a particular char* is already encoded as UTF-8, and it's common for people to forget that Unicode characters can take up to 6 bytes in UTF-8. I know I'm in the minority, but I prefer to use UTF-16 or UTF-32, because nobody makes the kind of mistakes with UTF-16/UTF-32 that they make with UTF-8. Plus, you can't accidentally pass a UTF-16-encoded string to a non-Unicode-aware function.
- syncsynchalt 12y ago
  Unicode code points can only take 4 bytes in UTF-8, this has been true ever since the planes were capped at U+10FFFF. Or did you mean UTF-8 encoded surrogate pairs?
  - maxlybbert 12y ago
    Thanks for the correction. I don't know where I originally got the "up top six bytes" number from, but I've been using it for a while. Apparently, it's out of date (looking at the original proposal, https://en.wikipedia.org/wiki/UTF-8#Description https://en.wikipedia.org/wiki/UTF-8#Description , some code points were expected to need 6 bytes, but as you say, that's no longer true).
    - syncsynchalt 12y ago
      It was true until 2003, so in this case reality has been retconned on you.
- nly 12y ago
  UTF-16 has surrogate pairs as well. It's the worst of both worlds.
  - maxlybbert 12y ago
    I prefer UTF-32, but I'm very lonely. The benefit with UTF-16 is that you can't accidentally pass a string to, say, strlen(). But, yes, people will forget that not all Unicode code points fit in 16 bits, and won't test with the right kind of input to find out that they've done it wrong. So errors can still creep in; but I will continue to argue that those errors are less common because the fact that you're dealing with Unicode (and not ASCII or something else) is more obvious and you're more likely to need library calls (that will do the right thing) to do anything useful.
nly 12y ago
C++'s support for unicode is minimal. There are unicode string literals for UTF-8, UTF-16 and UTF-32, and conversion facilities between those encodings and the host platforms native 'wide char' format. > - Do I use char* for strings? It sounds like wchar_t is 16 bits on some systems and 32 on others, so I should avoid it? In general you should use std::string and UTF-8, since UTF-8 is a relatively safe encoding for sub-string search and replace, and (locale insensitive) lexical comparison/ordering. Windows uses a 16 bit wchar_t (UTF-16). Mostly everyone else uses 32 bit (UTF-32) > - If I want to read & write UTF-8 files, how do I turn the bytes into 32-bit wide Unicode strings? http://en.cppreference.com/w/cpp/locale/wstring_convert http://en.cppreference.com/w/cpp/locale/wstring_convert > - Are there special functions I should use for handling Unicode strings? Nope. You need a library for that. Manipulating natural language is a tricky business anyway, beyond single mortals, and you will likely get it wrong.

wisty 12y ago

Look here: http://userguide.icu-project.org http://userguide.icu-project.org You really need a library when dealing with unicode, if you are doing anything advanced. How do you split a sentence into words? Spaces, right? Oh, Chinese doesn't have spaces. How about sentences? Paragraphs? Even characters are a pain to iterate over.

Nursie 12y ago
Agreed, ICU is a good place to start looking at this stuff, used it in a few large-scale commercial projects now.
jonstewart 12y ago
It really is abominable how bad the standard support for Unicode is. ICU is great, though the C API is somewhat limited compared to the full C++ API. ICU's build process, OTOH, leaves something to be desired, especially if you want to build it with a cross-compiler.

zanny 12y ago

Why not just write C style C++? I never understand the arguments to use C anymore - you can write C style C++ and get on with your life, and cherry pick the features you want. In C++ unicode is as simple as: string s = u8"This is always a utf8 string.";

pmelendez 12y ago
Well there are people who really hates C++[1] and to be honest C++03 had some flaws an inconveniences that made the language a bit awkward. Right now it feels like it is going in the right direction. [1]http://article.gmane.org/gmane.comp.version-control.git/57918 http://article.gmane.org/gmane.comp.version-control.git/5791...
akira2501 12y ago
Is 'extern "C"' still a thing to avoid name mangling? Some C++ features are easier not to use than others.
- mpyne 12y ago
  Yes, extern "C" is still a thing to interoperate with the C FFI. It is required if for nothing else than to use C headers.
conistonwater 12y ago
The main problem with using C++ as C is that everyone has different ideas about which ideas to cherry pick. So without a lot of discipline and a good coding standard, lots of (maybe subjectively) bad ideas will make it into the code as well. And once they do, you can't do anything about them any more.
programmarchy 12y ago
Firmware development. For some microcontrollers, you'll be lucky to even have a C compiler let alone a C++ compiler. Even then, I'd rather use C than C++ for firmware. I often find myself running into odd compiler issues, having to inspect/modify assembly, deal with resource constraints, etc. and I imagine C++ would make that more difficult.
millstone 12y ago
Unicode is much more complex than that. For example, how would you then convert that string to uppercase, or truncate it? For Unicode strings in C++, you can either limp along with the wchar_t support, or leverage something like ICU. Neither is especially easy.

maxlybbert 12y ago

C11 has the uchar.h header (say, http://www.cplusplus.com/reference/cuchar/ http://www.cplusplus.com/reference/cuchar/ ), but it wasn't in Visual Studio last I checked. Most programmers will tell you to use UTF-8 for in-memory strings. But it's not easy to figure out if a particular char* is already encoded as UTF-8, and it's common for people to forget that Unicode characters can take up to 6 bytes in UTF-8. I know I'm in the minority, but I prefer to use UTF-16 or UTF-32, because nobody makes the kind of mistakes with UTF-16/UTF-32 that they make with UTF-8. Plus, you can't accidentally pass a UTF-16-encoded string to a non-Unicode-aware function.

syncsynchalt 12y ago
Unicode code points can only take 4 bytes in UTF-8, this has been true ever since the planes were capped at U+10FFFF. Or did you mean UTF-8 encoded surrogate pairs?
- maxlybbert 12y ago
  Thanks for the correction. I don't know where I originally got the "up top six bytes" number from, but I've been using it for a while. Apparently, it's out of date (looking at the original proposal, https://en.wikipedia.org/wiki/UTF-8#Description https://en.wikipedia.org/wiki/UTF-8#Description , some code points were expected to need 6 bytes, but as you say, that's no longer true).
  - syncsynchalt 12y ago
    It was true until 2003, so in this case reality has been retconned on you.
nly 12y ago
UTF-16 has surrogate pairs as well. It's the worst of both worlds.
- maxlybbert 12y ago
  I prefer UTF-32, but I'm very lonely. The benefit with UTF-16 is that you can't accidentally pass a string to, say, strlen(). But, yes, people will forget that not all Unicode code points fit in 16 bits, and won't test with the right kind of input to find out that they've done it wrong. So errors can still creep in; but I will continue to argue that those errors are less common because the fact that you're dealing with Unicode (and not ASCII or something else) is more obvious and you're more likely to need library calls (that will do the right thing) to do anything useful.

nly 12y ago

C++'s support for unicode is minimal. There are unicode string literals for UTF-8, UTF-16 and UTF-32, and conversion facilities between those encodings and the host platforms native 'wide char' format. > - Do I use char* for strings? It sounds like wchar_t is 16 bits on some systems and 32 on others, so I should avoid it? In general you should use std::string and UTF-8, since UTF-8 is a relatively safe encoding for sub-string search and replace, and (locale insensitive) lexical comparison/ordering. Windows uses a 16 bit wchar_t (UTF-16). Mostly everyone else uses 32 bit (UTF-32) > - If I want to read & write UTF-8 files, how do I turn the bytes into 32-bit wide Unicode strings? http://en.cppreference.com/w/cpp/locale/wstring_convert http://en.cppreference.com/w/cpp/locale/wstring_convert > - Are there special functions I should use for handling Unicode strings? Nope. You need a library for that. Manipulating natural language is a tricky business anyway, beyond single mortals, and you will likely get it wrong.