32 ms·
UTF-8 Everywhere
- angersock 13y agoWhat is currently the best way of dealing with UTF-8 strings in a cross-platform manner? It sounds like widechars and std::string just won't cut it.
- 90minuteAPI 13y agoYeah, you really need some specialized interface that encapsulates all the things you may need from your string processing, but can spit out a representation of that string in various encodings. In line with the spirit of this article, that interface should use UTF-8 storage internally as well, but this should be transparent to the programmer anyway. Dealing with encoded strings directly is a recipe for heartache unless you're actually writing such a library.
- lmm 13y agoA higher level language, honestly. Perl, Ruby, Python3 and Haskell have excellent crossplatform utf8 support, and I'd be amazed if OCaml didn't. But if you want to write C++ code that works on windows, you're in for some pain.
- belluchan 13y agoAnd software developers, don't forget to implement the 4 byte characters too please. Utter nightmare dealing with MySQL. I believe 4 byte characters still even break github comments.
- 90minuteAPI 13y agoAbout 2 weeks I tried to file a bug with our backend guys about 4-byte characters wreaking havoc on our API. My example broke the bug tracker's (bugzilla) comment system as well. I chuckled.
- jeltz 13y agoI assume Bugzilla broke due to being backed by MySQL. Bugzilla itself is written in Perl so should have no problem.
- lmm 13y agoAt my last job I checked in a test case for our astral character handling. And broke the build server.
- MBCook 13y agoYeah. We've noticed, to our own amusement, that Jira (we're on an older version) can't handle non-ASCII. Makes entering tickets involving other languages fun.
- ggreer 13y agoI ran into that issue with MySQL's utf-8 handling. It was 𝒜wesome: http://geoff.greer.fm/2012/08/12/character-encoding-bugs-are-𝒜wesome/ http://geoff.greer.fm/2012/08/12/character-encoding-bugs-are...
- guard-of-terra 13y agoMySQL was a Norwegian company at the beginning, but you won't believe. It's one of the worst products when it comes to I18N and especially to Unicode. They still kind of store unicode text as a chain of encoded utf8 characters, if I'm not wrong. Their stupid command line utility still defaults to LATIN1 for input and output despite my locale clearly saying .UTF-8. People in their IRC channel still refuse to admit anything of this as a problem.
- jeltz 13y agoThe horrible unicode support in MySQL fits well with its general careless attitude toward data integrity. Minor correction: MySQL was Swedish and InnoDB Finnish.
- jasonjei 13y agoWe constantly have to deal with Win32 as a build platform and we write our apps natively for that platform using wchar. I think the main difficulty is that most developers hate adding another library to their stack, and to make matters worse, displaying this text in Windows GUI would require conversion to wchar. That's why I think they are up for a lot of resistance, at least in the Windows world. If the Windows APIs were friendlier to UTF-8, there might be hope. But as it stands right now, using UTF-8 requires the CA2W/CW2A macros, which is just a lot of dancing to keep your strings in UTF-8 which ultimately must be rendered in wchar/UTF-16. Maybe there might be a shot in getting developers to switch if Windows GUIs/native API would render Unicode text presented in UTF-8. But right now, it's back to encoding/decoding.
- deleted 13y ago[deleted]
- jbk 13y agoThis resonate so much for me, in VLC. VLC has a very large number of users on Windows (80% of our users), yet almost none of the dev use Windows to code. Therefore, we use UTF-8 char* everywhere, notably in the core. We use UTF-16 conversions only in the necessary Windows modules, that use Windows APIs. Being sure we were UTF-8 everywhere took a lot of time, tbh... But the worse are formats like ASF (WMV) or MMS that use UTF-16/UCS-2 (without correctly specifying) them and that we need to support on all other platforms, like OSX or Linux...
- lelandbatey 13y agoWow, I've learned even more about VLC (and programming in general) and I'm even more impressed. Thanks for this little insight!
- username42 13y agoI am also convinced that wide char should be avoided and are one of the bad thing of java. It is very nice to see that more and more people agree and that important projects like VLC already apply this principle. Will 2014 become the year we kill wide chars ?
- randomfool 13y ago"This is what made UTF-8 the favorite choice in the Web world, where English HTML/XML tags are intermixed with any-language text." Except that Javascript is UTF-16, so no luck with 4 byte chars there.
- lisper 13y ago> Javascript is UTF-16 No it isn't. Javascript is no different from any other text. It can be encoded in any encoding. Where did you get the idea that JS is UTF-16? EDIT: I misunderstood the intent of the comment I was responding to. JS uses (unbeknownst to me) UTF-16 as its internal representation of strings.
- ender7 13y agoJavaScript source can be encoded in any way that the browser can handle, yes. Within the JS language, strings are represented as sort-of-UCS-2-sort-of-UTF-16 [0]. This is one of the few problems with JS that I think merits a backwards-compatibility-breaking change. [0] http://mathiasbynens.be/notes/javascript-encoding http://mathiasbynens.be/notes/javascript-encoding
- thurn 13y agoGP means string literals. To quote from the spec: "4.3.16 String value: primitive value that is a finite ordered sequence of zero or more 16-bit unsigned integer... Each integer value in the sequence usually represents a single 16-bit unit of UTF-16 text."
- jgraham 13y agoThe "usually" there turns out to be important. Javascript "strings" are, as the spec says, just arrays of 16 bit integers internally. Since Unicode introduced characters outside the Basic Multilingual Plane (BMP) i.e. those with codepoints greater than 0xFFFF it has no longer been possible to store all characters as a single 16 bit integer. But it turns out that you can store non-BMP character using a pair of 16 bit integers. In a UTF-16 implementation it would be impossible to store one half of a surrogate pair without the other, indexing characters would no longer be O(1) and the length of a string would not necessarily be equal to the number of 16 bit integers, since it would have to account for the possibility of a four byte sequence representing a single character. In javascript none of these things are true. This turns out to be quite a significant difference. For example it is impossible in general to represent a javascript "string" using a conforming UTF-8 implementation, since that will choke on lone surrogates. If you are building an application that is supposed to interact with javascript — for example a web browser — this prevents you from using UTF-8 internally for the encoding, at least for those parts that are accessible from javascript.
- andystanton 13y ago"Even though one can argue that source codes of programs, web pages and XML files, OS file names and other computer-to-computer text interfaces should never have existed, as long as they do exist, text is not only for human readers." I'm a little confused by this statement. Can someone clarify?
- claudius 13y agoI think the author wants to say that a) computers should use appropriate binary formats for communication between them (e.g. from a web server to a browser), b) they don’t and use plain text in many cases (e.g. HTML) and hence c) text is not only read by humans but also by computers. I am not entirely sure whether that makes any sense, though.
- asgard1024 13y agoThis may be tangential, but I think that computer languages should have a different type (and literal notation) for human text (strings that may be read by human, may be translated, won't affect program semantics) and for computer string (strings that are strictly defined, not to be translated, and may affect program semantics). Then we could put all the human language problems into human text type, and leave the simpler computer string type with easier semantics. In Python, although there are no tools for that, I typically use the following convention: single quotes for computer text and double quotes for human text. I guess you could use byte arrays for computer text as well, but it would be more painful.
- bcbrown 13y agoIn traditional Windows development, the way that was handled was by having human-strings in a resource file (which you could replace with a file in a different language), while the computer-strings were hardcoded constants.
- tel 13y agoHaskell has the different semantics though they share the literal type. In Haskell there's Text for Unicode strings (UTF-16 encoded) and ByteString for, well, byte-strings (fixed vectors of Word8s). The semantics make it hard to confuse them, though if you turn on the OverloadedStrings extension then "foo" is ambiguous. If you don't like that you can just not turn on that extension and then there is no literal syntax for either type and you must convert literal strings (really just lists of characters, [Char]) to either type manually Data.Text.pack "foo" :: Text Data.ByteString.pack "foo" :: ByteString Finally, you must use manual encoding functions to interconvert them Data.Text.Encoding.encodeUtf8 :: Text -> ByteString
- Pxtl 13y agoI was horrified to discover that Microsoft SQL Server's text import/export tools don't even support UTF-8. Like, at all. You can either use their bastardized wrongendian pseudo-UTF-16, or just pick a code-page and go pure 8-bit.
- deleted 13y ago[deleted]
- josteink 13y agoI'm not sure what modules or tools you are talking about, but if you use Sql Server Integration Services (formerly SQL Servrer Data Transformation Services), you basically have a data processing pipeline which supports everything and all transformations on the planet. And obviously it supports arbitrary text-encodings, although sometimes you will need to be explicit about it. If you used the simplified wizards, all the options may not have been there, but you should have been given the option to export/save the job as a package, and then you can open, modify, test and debug that before running the job for real. Seriously. SQL Server has some immensely kick-ass and über-capable tooling compared to pretty much every other database out there. To even suggest it doesn't support UTF8 is ludicrous.
- Pxtl 13y agoThe various command-line and text-friendly bulk/bcp tools are the pain-point here. SSIS is insanely powerful and performant, but it's also insanely cumbersome and script-unfriendly. Microsoft has finally started embracing the power of plain text and scriptable tools in their web-stack and .NET, but SSIS represents a holdover from their heavyweight GUI-and-wizard days.
- josteink 13y agoFair enough. I don't use those tools too often, so I tend to forget they're around as well. That said, if you are doing repeatable jobs (and not just one-off imports) you can still create a SSIS package, and then run the package from your script using the package-runner and appropriate config-data.
- Dewie 13y agoIT is so Anglophile that programs can become slower if you deviate from ASCII... But of course being so incredibly anglocentric is not an issue, at least that seems to be the consensus of the participants when I read discussions on the Web where all the people who are discussing it write English with such a proficiency that I can't tell who are and aren't native speakers of the language.
- cube13 13y ago> IT is so Anglophile that programs can become slower if you deviate from ASCII... Well, switching from a simple, 7-bit character set where all you need to do is parse and display byte by byte to a potentially multi-byte processing and a display lookup table that's several orders of magnitude larger... it's pretty easy to see why things can become slower.
- jasonjei 13y agoI'm Chinese American, and I don't agree with your statement that string libraries are Anglophile. How would you encapsulate the 10,000 commonly used Chinese characters? It's just the reality of having a lot of characters in a language. Not much else you can do to speed up processing. How would you design string storage to be faster for a language like Chinese? English happens to be the lingua franca of Engineering. It's not about brown nosing English-speaking countries, but about getting the widest range of audience.
- Dewie 13y agoI won't pretend to have a solution. I guess you would like to have some compression scheme, since I'm guessing it would save space over having 2-4 bytes (however many there are in the Chinese language) for every character. You won't gain much compared to having an English-centric encoding scheme, I guess, in the case of a language with a large amount of characters. But it's still funny to me how even the computer who speaks in 1's and 0's favours English-centered notation. > English happens to be the lingua franca of Engineering. It's not about brown nosing English-speaking countries, but about getting the widest range of audience. Pragmatism ũber alles, chants the American. I guess I'm not impressed by the support of non-English languages in IT. I have for that matter met engineering students who don't seem to speak a lick of English, maybe even people studying CS/CE.
- wehadfun 13y agoI admire and appreciate your concern for something that is missunderstood and ignored. However this webpage took way to long to say what is so great about utf 8.
- jasonjei 13y agoHonestly, I think it is platform politics. *nix systems seem to prefer UTF-8, while UTF-16 is the default on Windows. Space and memory are cheap, so either encoding seems fine. The bottom line is that UTF-8 is awkward to use on Windows, while UTF-16/wchar_t is awkward to use on Linux, simply because the core APIs make them so (there is no _wfopen function in glibc).
- hdevalence 13y agoThe other problem with UTF-16 is that it's much easier to pretend that 1 element = 1 character than with UTF-8.
- MichaelGG 13y agoIt's not really politics. Microsoft made the choice for fixed-sized chars back when it was thought that 16 bits was enough for everyone. MS was at the forefront of internationalizing things, and probably still are. (Multilanguage support in Windows and Office is quite top class.) Unfortunately, we need more than 16 bits of codepoints, so 16-bit chars is a waste and a bad decision with that insight. It seems unlikely that a fresh platform with no legacy requirements would choose a 16-bit encoding. Think of all the XML in Java and .NET - all of it nearly always ASCII, using up double the RAM for zero benefit. It sucks. Was UTF-8 even around when Microsoft decided on 16-bit widechar? Other platforms seem to have lucked out by not worrying as much as standardizing on a single charset and UTF8 came in and solved the problems.
- Someone 13y ago"Was UTF-8 even around when Microsoft decided on 16-bit widechar?" No, Thompson's placemat is from September 1992 and NT 3.1 from July 1993, but development on NT started in November 1989 (http://en.wikipedia.org/wiki/Windows_NT#Development http://en.wikipedia.org/wiki/Windows_NT#Development)
- elwell 13y agoI can only imagine what kind of frustration drove someone to make this site.
- angersock 13y agoI'm more than imagining it right now. >:(
- SomeCallMeTim 13y agoThe same frustration made me write a custom string library for the Playground SDK, years ago. std::string is missing a lot of functionality one tends to need when dealing with strings (such as iterating over UTF-8 characters or fast conversion to UTF-16, but also things like search-and-replace). And it makes me sad that I can't use that string library any more (legally) because of the license PlayFirst insisted on using (no redistribution). As far as I'm concerned, though, there IS no good string library available for use anywhere. I've looked at all of the ones I could find, and they're all broken in some fundamental way. I guess solving the "string problem" isn't sexy enough for someone to release a library that actually hits all the pain points.
- angersock 13y agoWell, looks like that is what it is. Shoot me an email with your pain points, and maybe something will come of it. :)
- klmr 13y agoYou might like ogonek[1].It’s currently still not implementing regular expressions and thus strongly limited in its capabilities (and C++11 regex are so badly designed that they cannot be extended meaningfully to handle this case), but it has hands down the best API for working with text in C++. It makes using a wrong encoding a compile-time error and offers effortless ways of dealing with actual Unicode entities (code points, grapheme clusters) rather than bytes. [1] https://github.com/rmartinho/ogonek https://github.com/rmartinho/ogonek
- SomeCallMeTim 13y ago
- BadassFractal 13y agoWould be lovely if MS Office could export CSV to UTF-8, but nope.
- huhtenberg 13y agoExport in UTF-16, open in Notepad and re-save as UTF-8.
- frik 13y agoYes, I had issues with that in Excel 2010 too. The only well known Microsoft application that can handle UTF-8 is notepad.exe (Win 7).
- optimiz3 13y agoMost of the post talks about how Windows made a poor design decision in choosing 16bit characters. No debate there. However, advocating "just make windows use UTF8" ignores the monumental engineering challenge and legacy back-compat issues. In Windows most APIs have FunctionA and FunctionW versions, with FunctionA meaning legacy ASCII/ANSI and FunctionW meaning Unicode. You couldn't really fix this without adding a 3rd version that was truly UTF-8 without breaking lots of apps in subtle ways. Likely it would also only be available to Windows 9 compatible apps if such a feature shipped. No dev wanting to make money is going to ship software that only targets Windows 9, so the entire ask is tough to sell. Still no debate on the theoretical merits of UTF-8 though.
- angersock 13y agoNothing worth doing is easy. Anyways, the FunctionA/FunctionW is usually hidden behind a macro anyways (for better or worse). This could simply be yet another compiler option.
- gamacodre 13y agoWe "solved" (worked around? hacked?) this by creating a set of FunctionU macros and in some cases stubs that wrap all of the Windows entry points we use with incoming and outgoing converters. It's ugly under the hood and a bit slower than it needs to be, but the payoff of the app being able to consistently "think" in UTF-8 has been worth it. Of course, we had to ditch resource-based string storage anyway for other cross-platform reasons, and were never particularly invested in the "Windows way" of doing things, so it wasn't a big shock to our developers when we made this change.
- nabla9 13y agoUTF-8 is usually good enough in disk. I would like to have at least two options in memory: utf-8 and vector of displayed characters (there's many combinations in use in existing modern languages with no single-character representations in UTF-<anything>).
- Guvante 13y agoDo you need a vector of displayed characters? Usually all you care about is the rendered size, which your rendering engine should be able to tell you. No need to be able to pick out those characters in most situations.
- nabla9 13y agoYes. If I want to work with language and do some stringology, that's what I want. I might want to swap some characters, find length of words etc. To have vector of characters (as what humans consider characters) is valuable.
- twic 13y agoYes. A really good string type, that actually modelled a sequence of characters, not bytes, codepoints, interspersed glyphs and modifiers, or what have you, would be very useful at times. The acid test for this sort of 'humane string' type would be whether you could splice together any two substrings from any two input strings and get something that could be validly displayed. UTF-8 bytes fail because you can get fractional codepoints. Codepoints in >=20-bit integers fail because you can get modifier characters which don't attach to anything. A similar test would be whether you can reverse any input string by reversing the sequence of units. For example, reversing "amm͊z" should yield "zm͊ma", which it doesn't in unicode, because "m͊" is made with a combining mark, and doesn't have a composed form. For extra fun, i suspect that reversing the string "œ" should yield "eo". It should also be simple to do things like search for particular characters in a modifier-insensitive way. For example, i should be able to count that "sš" contains two copies of the letter 's' without having to do any deciphering. I suspect i should also be able to count that the string "ß" contains two copies of the letter 's', but i'm not nearly as sure about that. I think i essentially want a string that looks like: List<Pair<Character, List<Modifier>>> But i'm not sure. And i'm even less sure about how i'd encode it efficiently.
- huhtenberg 13y agoNow, all the advice in the Windows section - don't do this, don't do that, only and always do third - is lovely, but if you happen to care about app's performance, you will have to carry wstrings around. Take a simple example of an app that generates a bunch of logs that need to be displayed to the user. If you are to follow article's recommendations, you'd have these logs generated and stored in UTF8. Then, only when they are about to be displayed on the screen you'd convert them to UTF16. Now, say, you have a custom control that renders log entries. Furthermore, let's imagine a user who sits there and hits PgUp, PgDown, PgUp, PgDown repeatedly. On every keypress the app will run a bunch of strings through MultiByteToWideChar() to do the conversion (and whatever else fluff that comes with any boost/stl wrappers), feed the result to DrawText() and then discard wstrings, triggering a bunch of heap operation along the way. And you'd better hope latter doesn't cause heap wobble across a defrag threshold. Is your code as sublime as it gets? Check. Does it look like it's written by over-enlightened purists? You bet. Just look at this "advice" from the page - ::SetWindowTextW(widen("string litteral").c_str()) This marvel passes a constant string to widen() to get another constant string to pass to an API call. Just because the code is more kosher without that goddamn awful L prefix. Extra CPU cycles? Bah. A couple of KB added to the .exe due to inlining? Who cares. But would you just look at how zen the code is. -- tl;dr - keeping as much text as possible in UTF8 in a Windows app is a good idea, but just make sure not to take it to the extremes.
- guard-of-terra 13y ago"if you happen to care about app's performance, you will have to carry wstrings around" If those strings are for the user to read, he's reading a million times slower than you handle the most ornate reencoding. Sounds like a premature optimization.
- josteink 13y agoLooking at the .NET parts of the manifesto, I just have to roll my eyes: Both C# and Java offer a 16 bit char type, which is less than a Unicode character, congratulations. The .NET indexer str[i] works in units of the internal representation, hence a leaky abstraction once again. Substring methods will happily return an invalid string, cutting a non-BMP character in parts. While theoretically true, for most practical purposes, this reeks of a USA/American/English bias and lack of real world experience. You know what? I want to know that the text "ØÆÅ" is three characters long. I dont want to know that it's a 6-byte array once encoded to UTF8. Anywhere in my code telling me this is 6 characters is lying, not to mention a violation of numerous business-requirements. When I work with text I want to work with text and never the byte-stream it will eventually be encoded to. I want to work on top of an abstraction which lets me treat text as text. Yes, their are cases where the abstraction will leak. But those cases are very far and few in between. And in all cases where it doesn't, it offers me numerous advantages over the PHPesque, amateurish and incorrect approach of treating everything as a dumb byte-array. It's not. It's text in my program. It's text rendered on your screen. It's just a byte-array when we send it over the wire, so stop trying to pretend text isn't text. This manifesto is wildly misguided.
- derefr 13y agoI fundamentally agree, but I don't think even you understand the implications of what you're saying. Though you say "characters", You're probably still thinking about Unicode codepoints here, not characters. A real character-oriented API would require an arbitrary number of bytes to store each character (a character being an arbitrary vector of codepoints), and would require a font to be associated to each region of codepoints--because it's the font's decision of how to render a sequence of codepoints, so it's the font that determines how many characters you get. (For an edge-case example: http://symbolset.com/ http://symbolset.com/) ...and saying that, it'd still be a good idea.
- chipsy 13y agoWhen I learned of the difficulty mapping between code points and characters, I realized that Unicode is a standard nobody will ever (knowingly) implement correctly. Even if everyone has access to a font API, there'll probably be bugs in the fonts for all eternity.
- duaneb 13y agoThank god for emoji.
- pjscott 13y agoThey're very useful if you want a test case that requires multiple bytes in UTF-8 and multiple words in UTF-16.
- duaneb 13y agoSeriously, I wasn't being sarcastic! Emoji have been the single largest driving factor in proper unicode adoption I've seen—they're the first non-BMP characters to see wide-spread use.
- jahewson 13y ago> there is a silent agreement that UTF-8 is the most correct encoding for Unicode on the planet Earth But what about other planets? Is there a Unicode Astral Plane which may encode poorly in the future?
- taejo 13y agoThere is are 13 Unicode astral planes (though only two of them have characters assigned so far), and it they do indeed encode poorly in some environments: the planes other than plane 0, the basic multilingual plane, are informally known as "the astral planes".
- vorg 13y agoThere's 16 astral planes (U+1xxxx to U+10xxxx), of which 3 have characters assigned... * plane 1 is the supplementary multilingual plane * plane 2 is the supplementary ideographic plane * plane E is the supplementary special-purpose plane * planes F and 10 are private-use planes Perhaps you wrote from memory.
- taejo 13y agoThanks for providing the correct details. I was indeed writing from memory.
- mattfenwick 13y agoI think it's unfortunate that it doesn't have more concrete examples. I think having more of those would really help strengthen their case, clarify their points, and make their arguments tangible and understandable to a much wider audience. One instance where I really wish for examples: they mention characters, code points, code units, grapheme clusters, user-perceived characters, fonts, encoding schemes, multi-byte patterns, BE vs LE, BOM, .... while I kind of get some of these, I certainly don't understand all of them in detail, and so there's no way that I'll grasp the subtleties of their complicated interactions. Examples, even of simple things such as what actually gets saved to disk when I write out a string using UTF-8 encoding vs. UTF-16 -- especially when using higher codepoints, would be hugely beneficial for me.
- vorg 13y agoI don't like the way UTF-8 was clipped to only 1 million codepoints in 2003 to match the UTF-16 limit. The original 2.1 billion codepoint capacity of the original 1993 UTF-8 proposal would've been far better. Go Lang uses \Uffffffff as syntax to represent runes, giving the same upper limit as the original UTF-8 proposal, so I wonder if it supports, or one day will support, the extended 5- and 6-byte sequences. In fact, UTF-16 doesn't really have the 1 million character limit: by using the two private-use planes (F and 10) as 2nd-tier surrogates, we can encode all 4-byte sequences of UCS-32, and all those in the original UTF-8 proposal. I suspect the reason is more political than technical. unicode.org (http://www.unicode.org/faq/utf_bom.html#utf16-6 http://www.unicode.org/faq/utf_bom.html#utf16-6) says "Both Unicode and ISO 10646 have policies in place that formally limit future code assignment to the integer range that can be expressed with current UTF-16 (0 to 1,114,111). Even if other encoding forms (i.e. other UTFs) can represent larger intergers, these policies mean that all encoding forms will always represent the same set of characters. Over a million possible codes is far more than enough for the goal of Unicode of encoding characters, not glyphs. Unicode is not designed to encode arbitrary data."
- kzrdude 13y agoWhat use would it have to have so much extra codepoint space?
- vorg 13y ago2 planes (130,000) of private-use codepoints aren't enough, and because the top 2 planes of Unicode are designated private use, UTF-16 gives developers the option of extending them to 2.1 billion if they need it. I've wanted extra private-use space for generating Unihan characters by formula in the same way the 10,000 Korean Hangul ones are generated from 24 Jamo. I'm sure many other developers come across other scenarios where 130,000 isn't enough for private use. I'm simply saying that UTF-8 shouldn't be crippled in the Unicode/ISO spec to 21 bits, but be extended to 31 bits as originally designed because the technical reason given (i.e. because UTF-16 is only 21 bits) isn't actually true. The extra space should be assigned as more private use characters. (Except of course the last two codepoints in each extra plane would be nonchars as at present, and probably also the entire last 2 planes if the 2nd-tier "high surrogates" finish at the end of a plane.)
- puppetmaster3 13y agoIn Javascript, it's UTF16. Also Java. Can't speak for other of the top.
- chj 13y agoWindows is a horrible environment for UTF8 unless MS provides a special locale for it. At present state, you can choose to use utf8 internally in your app, but when you need to cooperate with other programs (over sockets or files), it's going to be confusing. Some will be sending you ANSI bytes and you take it as UTF8.
- GnarfGnarf 13y agoInteresting. I came to the same conclusion myself a few years ago when converting a Windows app to Unicode. I store all strings as UTF-8, which enabled me to continue using strncpy, char[] etc. I convert to wchar_t only when I need to pass the string to Win32. I can even change from narrow to widechar dynamically. I use a global switch which tells me whether I am running in Unicode or not, and call the 'A' or 'W' version of the Win32 function, after converting to wchar_t if necessary.