8 ms·
Anyone who programs in Java is using UTF-16. And in my experience very few Java programmers understand that UTF-16 is a variable length encoding.
by btilly 16y ago
Anyone who programs in Java is using UTF-16. And in my experience very few Java programmers understand that UTF-16 is a variable length encoding.
- machrider 16y agoPython, too (it can be compiled to use UTF-32, but normally uses UTF-16 to mean "Unicode").
- Locke1689 16y agoActually, it's just stored internally as UCS-2. I'm not sure why that really matters, though. You shouldn't care how your Unicode code points are stored (i.e., the size of the integer) as long as they encode to a UTF encoding correctly. Edit: Ah, guys it's actually UTF-16, the configure flag is just named ucs2. False alarm.
- fedd 16y agoif you want to deal with characters with high numbers you should know code points stuff. for example, the String.length() would return a number of two-bytes chars, not real four bytes characters, which may confuse someone //edit: this is about Java
- Locke1689 16y agoThat's actually my point. Python supports Unicode code points and UTF. If you get the output encoding in UTF-8 it would actually be variable length chars. What's important is your coding output, not the internal code point representation.
- btilly 16y agoExactly. A Java char is not synonymous with a Unicode code point. But the majority of the time they are synonymous, older documentation claimed that they were the same, and this is the meme that many Java programmers (in my experience) have.
- fedd 16y agoyes. i write my java-based matrix to be code-points aware so that no-one in Japan and China using it would face any problems.
- fhars 16y agoYou should very much care about that, because if your tool stores text as UCS-2, it means that it doesn't support unicode at all, UCS-2 stopped being a valid encoding a long time ago.
- Locke1689 16y agoAs the parent noted, it can be compiled for UTF-32 support. Just recompile if you need the extra characters. Edit: Also, turns out it's UTF-16. The configure flag is named ucs2.
- pilif 16y agoI doubt that they are encoded in UCS-2 as that character set isn't able to encode every (or even just the majority) of unicode code points. You are right though (and this is why I upvoted you back to 1) that you shouldn't care. In fact, you not knowing the internal encoding the proof of that. In python (I'm talking python 3 here which has done this right), you don't care how a string is stored internally. The only place where you care about this is when your strings interact with the outside world (i/o). Then your strings need to be converted into bytes and thus the internal representation must be encoded using some kind of encoding. This is what the .decode and .encode methods are used for. Have a look at http://diveintopython3.org/strings.html http://diveintopython3.org/strings.html which manages to say this better (and with more words) than I ever would be able to.
- eklitzke 16y agoIn Python 2.x are encoded in UCS-2, not UTF-16, at least by default (I'm not sure about Python 3.x, I assume it's the same though). If you want to support every single possible Unicode codepoint, you can tell Python to do so at compile time (via ./configure flag). In practice the characters that aren't in UCS-2 tend to be characters that don't exist in modern languages, e.g. the characterset for Linear B, Domino tiles, and Cuneiform, so they're not supported since they're not of practical use to most people. There's a fairly good list at http://en.wikipedia.org/wiki/Plane_(Unicode) http://en.wikipedia.org/wiki/Plane_(Unicode) . In this list, Python by default doesn't support things not in the BMP.
- Locke1689 16y agoNo, the Python internals support surrogates so you can support characters outside the BMP. This makes it (basically) UTF-16.
- sedachv 16y agoThings outside of the BMP aren't just dead languages anymore. You have to be able to support characters outside the BMP if you want to sell your software in China: http://en.wikipedia.org/wiki/GB_18030 http://en.wikipedia.org/wiki/GB_18030
- pieter 16y agoIt leaks through in some places. For example, len(u'\U0001D310') (from the Tai Xuan Jing Symbols) returns 1 on 32-bit wide pythons, and returns 2 on the default 16-bit wide builds.
- Locke1689 16y agoNope, that's the correct behavior. Run len on the UTF encode and you'll get the expected result.
- pilif 16y agoUTF-16 behaves to UCS-2 as UTF-8 does to ASCII. Meaning: They share the character set. UTF-16 extends UCS-2 by using some reserved characters to indicate that what is following should be interpreted according to UTF-16 rules. So just like UTF-8. Meaning: Every UCS-2 document is also an UTF-16 document, but not the reverse (just like every ASCII document is also an UTF-8 document). But as I said below: It doesn't matter and could even be a totally proprietary character set as long as pythons string operations work on that character set and as long as there's a way to decode input data into that set and encode output data from that set.
- ot 16y agoNo, it is really UCS-2: >>> unichr(0x10000) ------------------------------------------------------------ Traceback (most recent call last): File "<ipython console>", line 1, in <module> ValueError: unichr() arg not in range(0x10000) (narrow Python build) If you want to support codepoints greater than 0x10000 you have to recompile with the option UTF32. I think it must be a constant-lenght encoding to allow s[i] to be constant time.
- Locke1689 16y agoGuido has a different opinion: http://mail.python.org/pipermail/python-dev/2008-July/080895.html http://mail.python.org/pipermail/python-dev/2008-July/080895...
- ot 16y agoYou are completely right, I'm sorry about my previous comment. The strange thing is that I couldn't find any reference to surrogate pairs in the Python documentation, so I was assuming that the elements of an unicode strings were complete codepoints. Instead this is not the case: >>> list(u'\U00010000') [u'\ud800', u'\udc00'] If I had Python compiled with the UTF32 option, this would return a single element, so Python is leaking an implementation detail that can change across builds. That's really really bad...
- Locke1689 16y agoNo, that's the correct behavior. list only incidentally returns a single character in ASCII strings -- it's not required to. You shouldn't be using list on raw unicode strings. u'\U00010000'.encode('utf-8') should produce the same result on every Python version.
- ot 16y ago> You shouldn't be using list on raw unicode strings. Why? I am using list only to show what are the values of s[0] and s[1]. What I am saying is that it returns the list of characters of the underlying representation, so a list of wide chars (possibly surrogate) if compiled with UTF16 or a list of 32bit characters if compiled with UTF16. Are you suggesting that all the string processing (including iteration) should be done on a str encoded in UTF8 instead of using the native unicode type?
- Confusion 16y agoToo few programmers know that UTF-8 is a variable length encoding: I've heard plenty assert that in UTF-8, every character takes two bytes, while claiming simultaneously they could encode every possible character in it. A bit broader: too few programmers understand the difference between a character set and a character encoding.
- fedd 16y ago> too few programmers understand the difference between a character set and a character encoding why then they have the same names??? :))) ps/ http://www.grauw.nl/blog/entry/254 http://www.grauw.nl/blog/entry/254 - is this article ok? first in google by "charset encoding difference"
- qntm 16y agoUnicode is a character set, and the only character set really worth speaking of. The Unicode character set includes almost every character in every writing system on Earth. A string is a piece of text, i.e. an ordered sequence of characters all taken from the same character set. A character encoding is a mapping/function/algorithm/set of rules which can be used to convert a string into a sequence of bytes and back again. A character set may have multiple encodings. UTF-8 and UTF-16 are two possible encodings of the Unicode character set.
- Confusion 16y agoThat article starts out OK and then suddenly tries to argue that you can use the terms interchangeably. You can not and you will drown in confusion if you try to. Just imagine that tomorrow, the Chinese introduce their own character set next to Unicode, but use UTF-8 to minimize the number of bytes it takes to represent their language (which makes sense, because the frequency of characters drops off pretty fast and some characters are much more common than others, so you'd like to represent those with one byte). The fact that the HTTP RFC speaks of 'charset=utf-8' is explained by this part of the spec: Note: This use of the term "character set" is more commonly referred to as a "character encoding." However, since HTTP and MIME share the same registry, it is important that the terminology also be shared. Why does MIME use the 'wrong' terminology? Perhaps because the registry is old and the difference between set and encoding was less obvious and relevant back then. Perhaps it was simply a mistake; a detail meant to be corrected. Perhaps the person that drew it up was inept. Who knows. It doesn't matter, it is still wrong. And don't get me started on the use of character set in MySql...