4 ms·

Wow. Considering PHP is used primarily for generating HTML, UTF-8 is the de-facto standard encoding for unicode on the web, and PHP files are usually ASCII or U

Wow. Considering PHP is used primarily for generating HTML, UTF-8 is the de-facto standard encoding for unicode on the web, and PHP files are usually ASCII or UTF-8, going for UTF-16 seems like a phenomenally bad idea. I can see how UTF-16 looked promising back when it was still UCS-2 and there were no surrogate pairs. These days though, UTF-8 and maybe UTF-32 seem to be the realistic choices when working from scratch; UTF-32's advantage in some areas is probably too weak to make it a real contender unless your strings are literally linked lists, not codepoint arrays. (i.e. you don't care that it uses 2-4 times as much memory or storage)

Carlfish 17y ago
One advantage of UTF-16 is that unlike UTF-8, very few characters you encounter in Real Life invoke surrogate pairs. So for the cost of a one-bit flag per string you can assume two bytes per character in your string operations for the overwhelmingly general case.
- pmjordan 17y ago
  Let's face it though, the main string operation in web apps (or most software) is concatenation. I strongly doubt there is any point in converting back and forth between UTF-16 and UTF-8 just for that tiny advantage in addressing individual code points - and even then, not quite 100% of the time. For search-and/or-replace (or anything that can be done with regexes), I'm pretty sure that UTF-16 has no advantage over UTF-8, as you can make the state machine operate on the bytes directly.
- jerf 17y ago
  "So for the cost of a one-bit flag per string you can assume two bytes per character in your string operations for the overwhelmingly general case." Thank you for that clear and concise explanation of the dangers of using UTF-16. Yes, I know that wasn't your intention, but it was the end result. One of the most dangerous library failures you can have is a function that works 99.99% of the time. Or in this case, 100% of the time on the input the English-speaking developer provides but distinctly less than 100% in the field. In this specific case, you can't actually optimize anything because all your optimizations are bugs. You can't just divide by two for character count; that's not an optimization, it's a bug. You can't just multiply by two for a substring operation, because you might chop a character in half, that's a bug. And so on. You'd need a separate type that indicates you've scanned the string to verify it never has split chars and now you might as well be on UCS-2, and that has its own dangers w.r.t. working 99.99% of the time. Much better to use UTF-8, where the dangers are much more apparent, all you have to do is leave the base ASCII case and you're testing UTF-8. Even I, an English-speaking developer, manage to test that case (once I know it exists, anyhow). There's still ways you can screw up but you're off to a much better start.
  - nradov 17y ago
    Right. From what I have seen, the majority of Java code out there is broken in exactly that way.
nradov 17y ago
Java's use of UTF-16 is now generally considered a design flaw. It takes more memory than UTF-8, but still has the complexity of variable-length character encoding. UTF-16 was a reasonable choice for Java at the time the decision was made but you would think the PHP developers would learn from that mistake.
- mmastrac 17y ago
  [citation needed!] As a Java developer, I'm very happy with the compromise that UTF-16 strikes. For day-to-day unicode use, UTF-16 covers all the bases. In the rare cases I need to step outside of UTF-16 to the higher planes, it would work transparently for me up to the point where I start slicing strings naively. To be honest, I've never actually had to use anything outside of the Basic Multilingual Plane. That's not from lack of breadth in my day-to-day job either - I wrote the first implementation of our web crawler/search engine at DotSpots and had to deal with fetching and indexing pages in languages from most of the BMP (mainly English, Russian and Chinese). Anyways, I strongly disagree with your assertion that Java was "generally considered" to have "made a mistake" by using UTF-16. It's made my life far easier and the memory costs aren't an issue on today's machines. We don't live in an ASCII world most of the time and storing strings internally as UTF-8 completely ignores this fact. [edit] Just so it's clear: UTF-8 only wins when you are representing pure ASCII text. For everything else, it either breaks even or loses. For most Chinese text, UTF-8 is 50% larger: For characters <= U+0080 (ie: ABC), you win by one byte. For characters > U+0080 but <= U+07FF (ie: Ȁɐ), you break even at two bytes. Everything higher than U+07FF in the BMP, < U+10000 (ie: 丂且⬄☃), you lose (by one byte). For characters >= U+10000 you break even again at four bytes. If you really want to optimize for ASCII text in Java, there's always byte[] and you're free to wrap a CharSequence around it.
  - nradov 17y ago
    Have you actually worked with code points and surrogate pairs for the java.lang.String class? It's a huge hassle and doing it correctly requires at least triple the amount of code as doing it wrong. The problem is that earlier versions of Java used UCS-2 and so a single real-world character was always represented by a single Java char value. But then the Java maintainers later changed the string representation to UTF-16 so now a single real-world character could take up either one or multiple Java char values. A lot of new code is still being written which implicitly, and incorrectly, assumes that 1 character == 1 char. I even intentionally do it wrong sometimes myself because dealing with code points leads to such code bloat.
wvenable 17y ago
PHP developers don't like to reinvent the wheel. They were trying to use existing libraries that stored strings as UTF-16. Unfortunately since PHP developers don't like to reinvent the wheel, PHP is also based on a large number of 3rd party libraries that aren't unicode aware.
ibsulon 17y ago
I just don't understand this. A file is very very very simple to convert once. Tell your developers, "If you don't encode in UTF-16, you will have a performance penalty. Set your file encodings as UTF-16 too. You weren't doing complex internationalization work before, it's really not that big a deal." I worked with someone who had been on the ICU project, and he argued that UTF-16 is the best compromise for most cases. If you're working primarily in the western character set, UTF-8 is attractive, but that comes at the expense of others. And frankly, if you don't roll it yourself, what are you going to use other than ICU?
- drm237 17y ago
  I would guess file encoding is not at all the limiting factor since often times opcode caches mean that the file is only read once anyway. The problem is that you get input for all sorts of other areas like forms, databases, web services, etc. most of which aren't UTF-16.
- wingo 17y ago
  > And frankly, if you don't roll it yourself, what are you going to use other than ICU? GNU libunistring: http://www.gnu.org/software/libunistring/ http://www.gnu.org/software/libunistring/
  - ibsulon 17y ago
    For PHP at least, there is a license match issue.
chipsy 17y ago
I recall doing a quick assessment of the UTF encodings recently and coming to a similar conclusion: UTF-8 is nearly ideal for transmission and storage and is fairly robust for manipulation, albeit it can be the least straightforward to implement(not that app developers actually have to implement it). UTF-32 is probably most useful as an internal optimization for tasks that can really benefit from a straight scan/cut/paste over even-sized memory cells. You wouldn't store it in your database, but you might want to make use of it in a document editor, for example, to speed up search+replace type operations. UTF-16 is still substantially more heavyweight than UTF-8, but it can't be optimized into straight memory cells like UTF-32 without breaking the spec. So - unless your needs are extremely specific and you discover a sweet spot in UTF-16 after extensive profiling - it's just not a likely candidate.

Carlfish 17y ago

One advantage of UTF-16 is that unlike UTF-8, very few characters you encounter in Real Life invoke surrogate pairs. So for the cost of a one-bit flag per string you can assume two bytes per character in your string operations for the overwhelmingly general case.

pmjordan 17y ago
Let's face it though, the main string operation in web apps (or most software) is concatenation. I strongly doubt there is any point in converting back and forth between UTF-16 and UTF-8 just for that tiny advantage in addressing individual code points - and even then, not quite 100% of the time. For search-and/or-replace (or anything that can be done with regexes), I'm pretty sure that UTF-16 has no advantage over UTF-8, as you can make the state machine operate on the bytes directly.
jerf 17y ago
"So for the cost of a one-bit flag per string you can assume two bytes per character in your string operations for the overwhelmingly general case." Thank you for that clear and concise explanation of the dangers of using UTF-16. Yes, I know that wasn't your intention, but it was the end result. One of the most dangerous library failures you can have is a function that works 99.99% of the time. Or in this case, 100% of the time on the input the English-speaking developer provides but distinctly less than 100% in the field. In this specific case, you can't actually optimize anything because all your optimizations are bugs. You can't just divide by two for character count; that's not an optimization, it's a bug. You can't just multiply by two for a substring operation, because you might chop a character in half, that's a bug. And so on. You'd need a separate type that indicates you've scanned the string to verify it never has split chars and now you might as well be on UCS-2, and that has its own dangers w.r.t. working 99.99% of the time. Much better to use UTF-8, where the dangers are much more apparent, all you have to do is leave the base ASCII case and you're testing UTF-8. Even I, an English-speaking developer, manage to test that case (once I know it exists, anyhow). There's still ways you can screw up but you're off to a much better start.
- nradov 17y ago
  Right. From what I have seen, the majority of Java code out there is broken in exactly that way.

nradov 17y ago

Java's use of UTF-16 is now generally considered a design flaw. It takes more memory than UTF-8, but still has the complexity of variable-length character encoding. UTF-16 was a reasonable choice for Java at the time the decision was made but you would think the PHP developers would learn from that mistake.

mmastrac 17y ago
[citation needed!] As a Java developer, I'm very happy with the compromise that UTF-16 strikes. For day-to-day unicode use, UTF-16 covers all the bases. In the rare cases I need to step outside of UTF-16 to the higher planes, it would work transparently for me up to the point where I start slicing strings naively. To be honest, I've never actually had to use anything outside of the Basic Multilingual Plane. That's not from lack of breadth in my day-to-day job either - I wrote the first implementation of our web crawler/search engine at DotSpots and had to deal with fetching and indexing pages in languages from most of the BMP (mainly English, Russian and Chinese). Anyways, I strongly disagree with your assertion that Java was "generally considered" to have "made a mistake" by using UTF-16. It's made my life far easier and the memory costs aren't an issue on today's machines. We don't live in an ASCII world most of the time and storing strings internally as UTF-8 completely ignores this fact. [edit] Just so it's clear: UTF-8 only wins when you are representing pure ASCII text. For everything else, it either breaks even or loses. For most Chinese text, UTF-8 is 50% larger: For characters <= U+0080 (ie: ABC), you win by one byte. For characters > U+0080 but <= U+07FF (ie: Ȁɐ), you break even at two bytes. Everything higher than U+07FF in the BMP, < U+10000 (ie: 丂且⬄☃), you lose (by one byte). For characters >= U+10000 you break even again at four bytes. If you really want to optimize for ASCII text in Java, there's always byte[] and you're free to wrap a CharSequence around it.
- nradov 17y ago
  Have you actually worked with code points and surrogate pairs for the java.lang.String class? It's a huge hassle and doing it correctly requires at least triple the amount of code as doing it wrong. The problem is that earlier versions of Java used UCS-2 and so a single real-world character was always represented by a single Java char value. But then the Java maintainers later changed the string representation to UTF-16 so now a single real-world character could take up either one or multiple Java char values. A lot of new code is still being written which implicitly, and incorrectly, assumes that 1 character == 1 char. I even intentionally do it wrong sometimes myself because dealing with code points leads to such code bloat.

wvenable 17y ago

PHP developers don't like to reinvent the wheel. They were trying to use existing libraries that stored strings as UTF-16. Unfortunately since PHP developers don't like to reinvent the wheel, PHP is also based on a large number of 3rd party libraries that aren't unicode aware.

ibsulon 17y ago

I just don't understand this. A file is very very very simple to convert once. Tell your developers, "If you don't encode in UTF-16, you will have a performance penalty. Set your file encodings as UTF-16 too. You weren't doing complex internationalization work before, it's really not that big a deal." I worked with someone who had been on the ICU project, and he argued that UTF-16 is the best compromise for most cases. If you're working primarily in the western character set, UTF-8 is attractive, but that comes at the expense of others. And frankly, if you don't roll it yourself, what are you going to use other than ICU?

drm237 17y ago
I would guess file encoding is not at all the limiting factor since often times opcode caches mean that the file is only read once anyway. The problem is that you get input for all sorts of other areas like forms, databases, web services, etc. most of which aren't UTF-16.
wingo 17y ago
> And frankly, if you don't roll it yourself, what are you going to use other than ICU? GNU libunistring: http://www.gnu.org/software/libunistring/ http://www.gnu.org/software/libunistring/
- ibsulon 17y ago
  For PHP at least, there is a license match issue.

chipsy 17y ago

I recall doing a quick assessment of the UTF encodings recently and coming to a similar conclusion: UTF-8 is nearly ideal for transmission and storage and is fairly robust for manipulation, albeit it can be the least straightforward to implement(not that app developers actually have to implement it). UTF-32 is probably most useful as an internal optimization for tasks that can really benefit from a straight scan/cut/paste over even-sized memory cells. You wouldn't store it in your database, but you might want to make use of it in a document editor, for example, to speed up search+replace type operations. UTF-16 is still substantially more heavyweight than UTF-8, but it can't be optimized into straight memory cells like UTF-32 without breaking the spec. So - unless your needs are extremely specific and you discover a sweet spot in UTF-16 after extensive profiling - it's just not a likely candidate.