8 ms·
UTF-8 good, UTF-16 bad
- wooster 16y agoBetter discussion here, IMO: http://research.swtch.com/2010/03/utf-8-bits-bytes-and-benefits.html http://research.swtch.com/2010/03/utf-8-bits-bytes-and-benef...
- burgerbrain 16y agoI honestly had no idea there were parts of the development community that actually used and preferred UTF-16...
- btilly 16y agoAnyone who programs in Java is using UTF-16. And in my experience very few Java programmers understand that UTF-16 is a variable length encoding.
- machrider 16y agoPython, too (it can be compiled to use UTF-32, but normally uses UTF-16 to mean "Unicode").
- Locke1689 16y agoActually, it's just stored internally as UCS-2. I'm not sure why that really matters, though. You shouldn't care how your Unicode code points are stored (i.e., the size of the integer) as long as they encode to a UTF encoding correctly. Edit: Ah, guys it's actually UTF-16, the configure flag is just named ucs2. False alarm.
- fedd 16y agoif you want to deal with characters with high numbers you should know code points stuff. for example, the String.length() would return a number of two-bytes chars, not real four bytes characters, which may confuse someone //edit: this is about Java
- Locke1689 16y agoThat's actually my point. Python supports Unicode code points and UTF. If you get the output encoding in UTF-8 it would actually be variable length chars. What's important is your coding output, not the internal code point representation.
- btilly 16y agoExactly. A Java char is not synonymous with a Unicode code point. But the majority of the time they are synonymous, older documentation claimed that they were the same, and this is the meme that many Java programmers (in my experience) have.
- fedd 16y agoyes. i write my java-based matrix to be code-points aware so that no-one in Japan and China using it would face any problems.
- fhars 16y agoYou should very much care about that, because if your tool stores text as UCS-2, it means that it doesn't support unicode at all, UCS-2 stopped being a valid encoding a long time ago.
- Locke1689 16y agoAs the parent noted, it can be compiled for UTF-32 support. Just recompile if you need the extra characters. Edit: Also, turns out it's UTF-16. The configure flag is named ucs2.
- pilif 16y agoI doubt that they are encoded in UCS-2 as that character set isn't able to encode every (or even just the majority) of unicode code points. You are right though (and this is why I upvoted you back to 1) that you shouldn't care. In fact, you not knowing the internal encoding the proof of that. In python (I'm talking python 3 here which has done this right), you don't care how a string is stored internally. The only place where you care about this is when your strings interact with the outside world (i/o). Then your strings need to be converted into bytes and thus the internal representation must be encoded using some kind of encoding. This is what the .decode and .encode methods are used for. Have a look at http://diveintopython3.org/strings.html http://diveintopython3.org/strings.html which manages to say this better (and with more words) than I ever would be able to.
- eklitzke 16y agoIn Python 2.x are encoded in UCS-2, not UTF-16, at least by default (I'm not sure about Python 3.x, I assume it's the same though). If you want to support every single possible Unicode codepoint, you can tell Python to do so at compile time (via ./configure flag). In practice the characters that aren't in UCS-2 tend to be characters that don't exist in modern languages, e.g. the characterset for Linear B, Domino tiles, and Cuneiform, so they're not supported since they're not of practical use to most people. There's a fairly good list at http://en.wikipedia.org/wiki/Plane_(Unicode) http://en.wikipedia.org/wiki/Plane_(Unicode) . In this list, Python by default doesn't support things not in the BMP.
- Locke1689 16y agoNo, the Python internals support surrogates so you can support characters outside the BMP. This makes it (basically) UTF-16.
- sedachv 16y agoThings outside of the BMP aren't just dead languages anymore. You have to be able to support characters outside the BMP if you want to sell your software in China: http://en.wikipedia.org/wiki/GB_18030 http://en.wikipedia.org/wiki/GB_18030
- pieter 16y agoIt leaks through in some places. For example, len(u'\U0001D310') (from the Tai Xuan Jing Symbols) returns 1 on 32-bit wide pythons, and returns 2 on the default 16-bit wide builds.
- Locke1689 16y agoNope, that's the correct behavior. Run len on the UTF encode and you'll get the expected result.
- pilif 16y agoUTF-16 behaves to UCS-2 as UTF-8 does to ASCII. Meaning: They share the character set. UTF-16 extends UCS-2 by using some reserved characters to indicate that what is following should be interpreted according to UTF-16 rules. So just like UTF-8. Meaning: Every UCS-2 document is also an UTF-16 document, but not the reverse (just like every ASCII document is also an UTF-8 document). But as I said below: It doesn't matter and could even be a totally proprietary character set as long as pythons string operations work on that character set and as long as there's a way to decode input data into that set and encode output data from that set.
- ot 16y agoNo, it is really UCS-2: >>> unichr(0x10000) ------------------------------------------------------------ Traceback (most recent call last): File "<ipython console>", line 1, in <module> ValueError: unichr() arg not in range(0x10000) (narrow Python build) If you want to support codepoints greater than 0x10000 you have to recompile with the option UTF32. I think it must be a constant-lenght encoding to allow s[i] to be constant time.
- Locke1689 16y agoGuido has a different opinion: http://mail.python.org/pipermail/python-dev/2008-July/080895.html http://mail.python.org/pipermail/python-dev/2008-July/080895...
- ot 16y agoYou are completely right, I'm sorry about my previous comment. The strange thing is that I couldn't find any reference to surrogate pairs in the Python documentation, so I was assuming that the elements of an unicode strings were complete codepoints. Instead this is not the case: >>> list(u'\U00010000') [u'\ud800', u'\udc00'] If I had Python compiled with the UTF32 option, this would return a single element, so Python is leaking an implementation detail that can change across builds. That's really really bad...
- Locke1689 16y agoNo, that's the correct behavior. list only incidentally returns a single character in ASCII strings -- it's not required to. You shouldn't be using list on raw unicode strings. u'\U00010000'.encode('utf-8') should produce the same result on every Python version.
- ot 16y ago> You shouldn't be using list on raw unicode strings. Why? I am using list only to show what are the values of s[0] and s[1]. What I am saying is that it returns the list of characters of the underlying representation, so a list of wide chars (possibly surrogate) if compiled with UTF16 or a list of 32bit characters if compiled with UTF16. Are you suggesting that all the string processing (including iteration) should be done on a str encoded in UTF8 instead of using the native unicode type?
- Confusion 16y agoToo few programmers know that UTF-8 is a variable length encoding: I've heard plenty assert that in UTF-8, every character takes two bytes, while claiming simultaneously they could encode every possible character in it. A bit broader: too few programmers understand the difference between a character set and a character encoding.
- fedd 16y ago> too few programmers understand the difference between a character set and a character encoding why then they have the same names??? :))) ps/ http://www.grauw.nl/blog/entry/254 http://www.grauw.nl/blog/entry/254 - is this article ok? first in google by "charset encoding difference"
- qntm 16y agoUnicode is a character set, and the only character set really worth speaking of. The Unicode character set includes almost every character in every writing system on Earth. A string is a piece of text, i.e. an ordered sequence of characters all taken from the same character set. A character encoding is a mapping/function/algorithm/set of rules which can be used to convert a string into a sequence of bytes and back again. A character set may have multiple encodings. UTF-8 and UTF-16 are two possible encodings of the Unicode character set.
- Confusion 16y agoThat article starts out OK and then suddenly tries to argue that you can use the terms interchangeably. You can not and you will drown in confusion if you try to. Just imagine that tomorrow, the Chinese introduce their own character set next to Unicode, but use UTF-8 to minimize the number of bytes it takes to represent their language (which makes sense, because the frequency of characters drops off pretty fast and some characters are much more common than others, so you'd like to represent those with one byte). The fact that the HTTP RFC speaks of 'charset=utf-8' is explained by this part of the spec: Note: This use of the term "character set" is more commonly referred to as a "character encoding." However, since HTTP and MIME share the same registry, it is important that the terminology also be shared. Why does MIME use the 'wrong' terminology? Perhaps because the registry is old and the difference between set and encoding was less obvious and relevant back then. Perhaps it was simply a mistake; a detail meant to be corrected. Perhaps the person that drew it up was inept. Who knows. It doesn't matter, it is still wrong. And don't get me started on the use of character set in MySql...
- fedd 16y agoi guess utf-8 requires more computations while processing strings. if you work with just 2 bytes chars, it (might) work faster // upvoted all replies, you're right
- edsrzf 16y agoThis is a common misconception about UTF-8 vs. UTF-16. You're missing two important facts. 1. Most UTF-8 string operations can operate on a byte at a time. You just have realize that functions like strlen will be telling you byte length instead of character length, and this usually doesn't even matter. (It's still important to know.) 2. UTF-16 is still a variable-width encoding. It was originally intended to be fixed-width, but then the Unicode character set grew too large to be represented in 16 bits.
- program 16y agoUTF-16 was never intended to be a fixed-width encoding and has been created in order to support characters outside the BMP which aren't covered by UCS-2.
- edsrzf 16y agoOkay, you're right. I shortened what I was trying to say too much. UTF-16 evolved from UCS-2, so its roots are in a 16-bit, fixed-width encoding.
- fnl 16y agoIf I have no Surrogate Range CPs in a string, it is far easier to work with UTF-16 than UTF-8 at the byte level, because all chars are constant size. For UTF-8 that only applies to ASCII. And SRs characters are extraordinarily rare, while non-ASCII chars are extremely common. So my programs ensure at the entry points the string is UCS-2 compatible, and then all subsequent string manipulations are far less complex to handle than with UTF-8.
- lysium 16y agoBut UTF-16 is variable encoding, too?
- sid0 16y agoWhatever environments jumped on Unicode early, before it was realized that 2 bytes wouldn't be enough, all chose to use UCS-2 for obvious reasons. In particular, that includes Windows and Java.
- derleth 16y ago> all chose to use UCS-2 for obvious reasons Probably because they figured they could just ignore endianness issues and that ASCII compatibility would be Somebody Else's Problem. There were always problems with UCS-2. UTF-8 would have had a number of advantages over it even if Unicode had never grown beyond the BMP (Basic Multilingual Plane, the first and lowest-numbered 16-bit code space).
- fedd 16y ago> endianness issues what are the issues? > ASCII compatibility would be Somebody Else's Problem for many of those outside "A" in ASCII (euphemism for America :) there were already a ton of problems, so endianness was the least (i personally never hit this problem) // disclaimer: i'm not that serious about predominance of Latin script, this is sorta irony
- barrkel 16y agoEndian: there's little-endian UTF-16LE and big-endian UTF-16BE, mirror images of one another.
- fedd 16y agoi thought the word 'issue' meant 'a problem'...
- barrkel 16y agoDepending on the level of abstraction you're living at - and that depends on the overall goal, performance constraints, environmental integration, OS / machine heterogeneity etc. - it may or may not be a problem. It's easy to dismiss if you have all the time in the world and a deep stack of abstractions. If you're doing deep packet analysis on UTF-16 text in a router, things may be different.
- program 16y agoStrings are big-endian UTF-16 by default even in Cocoa (stored in an array of unsigned shorts). Worst of all GCC define the wchar_t as a 4 byte int unless you specify -fshort-wchar.
- __david__ 16y agoAs far as I know, wchar_t is meant to be an internal only representation, so it's good that it is 32 bits--that way you are in one codepoint per word territory. It's a mistake to think you can just overlay some unicode binary data with a wchar_t pointer--you need to convert into and out of wchat_t from utf8/utf-16/whatever. Otherwise you aren't handling codepoints above 16 bits correctly.
- fauigerzigerk 16y agoUnfortunately the size of wchar_t is platform/compiler dependent.
- getsat 16y agoThe Windows NT development team made the decision to standardise on UTF-16. Every release of Windows since the original NT uses UTF-16 internally for all its "wide character" API calls (e.g., wcslen() and FindWindowW()).
- bzbarsky 16y agoI don't know about "preferring", but anyone manipulating strings in JavaScript is effectively using UTF-16 (or more precisely is using arrays of 16-bit integers which a web browser will interpret as UTF-16-encoded Unicode if you tell it that the array contains text). As a consequence at least Gecko and Webkit both use UTF-16 for their string classes, though there has been talk of trying to switch Gecko over to UTF-8. The problem then would be implementing the JS string APIs on top of UTF-8 strings efficiently.
- fedd 16y agoand i find it fair that American characters require 2 bytes in Java as everybody else, not 1 as in utf-8! :)
- brownleej 16y agoYou seem to be confusing "fair" with "equal". Treating everybody the same is not necessarily fair. It seems fair to me to have the most common characters be shortest. I don't have any evidence, but I would guess that Latin characters[1] are used most commonly. [1] "Latin characters" is the proper term, not "American characters"
- fedd 16y agothanks! but i just tried to kid which i can't control sometimes. anyway, i think that even if Latin chars weren't the most used in the world, it would be fair to keep them the primary charset for use in programming and markup languages, as no-one now complains that the international language of medicine is Latin, not, say, Chinese :) as computers started to be massively developed in America.
- prodigal_erik 16y agoI keep hoping a string API will catch on in which combining marks are mostly treated as indivisible. Handling text one codepoint at a time is as bad an idea as handling US-ASCII one bit at a time--almost everything it lets you do is an elaborate way to misinterpret or corrupt your data.
- fedd 16y agodid you mean the situation when for example "ä" can be transmitted as 00e4 alone or as 0061 "a" + 0308 "umlaut"?
- barrkel 16y agoIt's not so simple: it depends on what you're doing with the text. If you're not trying to do analysis with it, encoded text is more or less a program written in a DSL that, when interpreted by a font renderer, draws symbols in some graphical context. Depending on the analysis you want to do, you need varying amounts of knowledge. Perhaps you only need to know about word boundaries; perhaps you're trying to look things up in a normalized dictionary; maybe even decompose a word into phonemes to try and pronounce it. These require different levels of analysis, and one size won't fit all.
- fedd 16y agolook what i found and now plan to use http://download.oracle.com/javase/6/docs/api/java/text/Normalizer.html http://download.oracle.com/javase/6/docs/api/java/text/Norma... update: for particular purposes consider using Collator class, it makes collation keys (byte arrays) out of strings applying locale, case sensitiveness and unicode decomposition. (at least so says the doc, http://download.oracle.com/javase/6/docs/api/java/text/Collator.html http://download.oracle.com/javase/6/docs/api/java/text/Colla... )
- chalst 16y agoBroad backwards compatibility with ASCII is a strong reason to prefer UTF-8 in most applications, however I find the issues with UTF-16 are overstated and the advantages of it ignored. "UTF-16 has no practical advantages over UTF-8" - UTF-16 is better - i.e., quite a bit more compact, since, e.g., Chinese characters always take 3 characters in UTF-8, for asian languages. "A related drawback is the loss of self-synchronization at the byte level." - Maybe this is a problem, maybe not. Maybe the failure of UTF-8 to be self-synchronising at the 4-bit level is a problem is some circumstances. I don't mean to be flippant, but the wider point is that with UTF-16, you really need to commit to 16-bit char width. "The encoding is inflexible" - I think the author has confused the fixed-width UCS-2 and the variable-width UTF-16. "We lose backwards compatibility with code treating NUL as a string terminator." - Not true. NUL is a 16-bit character in UTF-16. Use a C compiler that supports 16-bit char width.
- yuhong 16y ago>"We lose backwards compatibility with code treating NUL as a string terminator." - Not true. NUL is a 16-bit character in UTF-16. Use a C compiler that supports 16-bit char width. Yea, it really should be "We lose backwards compatibility with code treating a zero 8-bit byte as a string terminator." >"UTF-16 has no practical advantages over UTF-8" - UTF-16 is better - i.e., quite a bit more compact, since, e.g., Chinese characters always take 3 characters in UTF-8, for asian languages. For pure CJK text files, yes.
- udoprog 16y ago"...Unicode has been standardising CJK character sets" - CJK characters sets are complex, and will always face opposition, besides, this is less related to how the characters are encoded, and more how to choose and composite your code points. "I think the author has confused the fixed-length UCS-2 and the variable length UTF-16." - No, in the same sentence you are quoting he mentions that utf-16 is limited to 0x110000 codepoints, contra utf-8 that is specified to expand up to 6 bytes. "Not true. NUL is a 16-bit character in UTF-16. Use a C compiler that supports 16-bit char width." - I won't be converting my source code into utf-16 any time soon. Besides, C is not a good example where this is an actual problem, the "unicode strings" will still just be represented as binary blobs - different depending on which encoding you choose. It's more of a problem in e.g. Python, where the native string is an array of characters, not just a null terminated blob. I wouldn't encode my source code in utf-8, however, if my editor my default happened to support utf-8, this wouldn't be an issue unless I enter an non-ascii compatible code point. I see this as a big plus.
- natmaster 16y agoCan someone link me to where python uses UTF-16? It was my understanding it defaulted to UTF-8. http://www.python.org/dev/peps/pep-3120/ http://www.python.org/dev/peps/pep-3120/
- lysium 16y agoThat PEP only refers to the encoding of the source (code) file, not to the encoding on how strings are stored in Python. Edit: Your question might be answered at SO: http://stackoverflow.com/questions/1838170/what-is-internal-representation-of-string-in-python-3-x#1838285 http://stackoverflow.com/questions/1838170/what-is-internal-...
- fedd 16y agosorry for the quastion as the reply, i would also ask for some info about the unicode issues in jython, as i lack experience with all python stuff. does it have problems or everything is transparent?
- masklinn 16y ago> It was my understanding it defaulted to UTF-8. For external IO, internally it's in UTF-16 (actually UCS-2, mostly unaware of surrogates) or in UCS-4 (via a compile-time switch). See http://docs.python.org/c-api/unicode.html#Py_UNICODE http://docs.python.org/c-api/unicode.html#Py_UNICODE for the Py_UNICODE API.
- fnl 16y agocheck PEP-261
- adobriyan 16y ago> The Go designers knew what they were doing, so they chose UTF-8. The authors of Go and authors of UTF-8 are more or less the same people, so the choice was no-brainer.
- thristian 16y agoThe article, at the end, claims that "ASCII was developed from telegraph codes by a committee." It turns out the story is much, much more complicated and interesting than that: http://www.wps.com/projects/codes/ http://www.wps.com/projects/codes/
- fedd 16y agomay i dare make a conclusion with my observations? :) utf-8 is good for network interchange and is de facto becoming standard. utf-16 is not bad for internal storage of strings in memory or in database. not nesserily bad. maybe even better for some reasons
- zbowling 16y agoUCS2, even though being an outdated predecessor to UTF-16, has some unique qualities that make it useful for things like databases or other storage mediums that you are not mixing with a lot of low code point characters (like you do with XML and HTML markups). One being that's fair for all languages with respect to size so when you may be storing your standard Chinese, Korean, Japanese characters. When UTF-16 made UC2 variable length, a few of the nice things were lost, but when dealing a lot of the higher code point characters mostly, UTF-16 may save you space.
- wildmXranat 16y agoCorrect me if I'm wrong, but I think Excel still outputs UTF-16 in some cases. I remember parsing generated .txt/.csv files and there were issues with it and it's endian order.
- mikecaron 16y agoGood read! I always HATE it when things complain that my visual studio .c/.h files are BINARY! WTF!