3 ms·

> you are treating text as binary Nope; it's treated differently by different functions. Like ars wrote, with UTF-8 a lot of properties of old ASCII strings h

by dexen 15y ago

> you are treating text as binary Nope; it's treated differently by different functions. Like ars wrote, with UTF-8 a lot of properties of old ASCII strings holds. A lot of PHP functions did not need any changes to handle UTF-8 just fine, for example explode() and join(). Other functions do handle actual characters -- when they must. In contrast, using UTF-16 or UTF-32 would require treating strings as opaque binary blobs in some cases, because those encodings lack certain properties of UTF-8. For example, the old ASCII-age strpos() is reliable with UTF-8, but unreliable with UTF-16. > Also, "trust the client" as your correctness mechanism? Not very safe. Never the case here. Standard library is trusted, and the average developer team is not much more secure than maintainers of such libraries. I know my code is not better than the standard library, anyway, so I'll rather rely on it to handle correctness. You use two kinds of functions: some care about properties of characters -- like mb_strtoupper(); others just work on strings -- like explode(). Only the former kind matters in aspect of correctness. When such function tries to operate on the mis-encoded character, it follows the Unicode standard: replaces the character with U+FFFD, a.k.a. `REPLACEMENT CHARACTER'. Result is valid UTF-8, with one character replaced with the marker. Any subsequent processing will go just fine, unless you want to stop and warn the user. The condition is easy to detect: there is one unique byte sequence for the REPLACEMENT CHARACTER, and it's clear where in the string the problem was. UTF-8 was designed to be self-synchronizing in case of such errors. > This isn't "handling unicode", it's "doing nothing" That ``doing nothing'' was my original point; thanks for re-iterating? You -- the developer -- do nothing, in particular you don't insert boilerplate code everywhere (and what if you forgot one rarely used execution path?). The functions that must handle individual characters perform the mundane work. Unicode is handled where it has to be -- and only there. > (...) your xterm assumes everything is UTF-8, (...) Nothing's assumed there; the process is instructed to do so via configuration. On POSIX platforms, that's environment variables: $LC_ALL and related. The main point of such configuration is that you can adjust whole system and make programs speak common language to one another. And they do -- following your instruction. Unless explicitly instructed to do otherwise -- when interfacing with remote systems and a different encoding has been agreed upon. > The second you throw in some other source, things are going to break, and you will need a plan for how to handle things. When you extend software to interface another software, you either configure both to use the same protocol (here: encoding), or implement protocol (here: encoding) negotiation. You -- the developer -- configure or implement it once, as part of creating the interface between them.

jrockway 15y ago
Honestly, this post leads me to believe you don't really understand how characters work, and have just gotten lucky so far. Do you really think it's acceptable to accept input, process it, and end up with REPLACEMENT CHARACTER because your string handling got fucked up? Sure, it's valid UTF-8, but it's the Wrong Answer. Plain and simple: by not handling Unicode, you are not handling Unicode. Echoing back UTF-8 when you get UTF-8 input is handling binary, not handling text. Yes, I know this works for your particular application, but it not correct in general. It won't work at all if you ever have any data sources or sinks that are not UTF-8, and it won't work if you actually have to treat the data as text ("match all characters that are whitespace"). Yes, some PHP functions take binary blobs that contain UTF-8 octets and return correct-looking answers. That's all they do, though; that's handling UTF-8, not handling Unicode. In Perl, characters are abstracted away from their character encoding. Perl strings are not UTF-8 or Latin-1 or anything; they are character strings. That makes producing correct results much easier; if you try to write a Perl character to a file, it will say, "hey, you can't expose characters to the outside world, you need to pick a character encoding, encode, and output that". Your PHP method just assumes everything is UTF-8, which is simply not a safe assumption to make. (Consider the case of binary data: if you start treating binary data like it's UTF-8 text, you break the binary data.) Also, regarding LANG, yes, that will change your terminal's encoding. So if you take one of your UTF-8 blobs and print it when the LANG is en_US.UTF-8, it will show up. But change the locale to en_US.ISO-8859-1, and it doesn't work anymore. That's because you can't just print binary blobs to the terminal and expect it to interpret that as text.
- ars 15y ago
  > Echoing back UTF-8 when you get UTF-8 input is handling binary, not handling text. Nope. UTF-8 is text. We don't live in an ascii world anymore, text requires all 8 bits and a variable number of bytes per character. > It won't work at all if you ever have any data sources or sinks that are not UTF-8 So don't do that. (Or convert as necessary.) > and it won't work if you actually have to treat the data as text ("match all characters that are whitespace"). Wrong. Actually it works just fine. > Yes, some PHP functions take binary blobs that contain UTF-8 octets and return correct-looking answers. That's all they do, though; that's handling UTF-8, not handling Unicode. So? Since that's completely sufficient to do the job why do you want more? > In Perl, characters are abstracted away from their character encoding. Why? Why would I want to do that? Is this complexity for the sake of complexity? > Your PHP method just assumes everything is UTF-8, which is simply not a safe assumption to make. Yes, actually it is a safe assumption for the simple reason that the only types of strings I handle are UTF-8. In todays world there is no reason to use anything else. All other character sets are obsolete and should never be used. And if you do need to handle one then convert as necessary, or define your app to handle only that character set. Programs that need to handle multiple character sets in the same applicate are extremely rare. There is no need for extreme complexity just to handle the rare case. > Consider the case of binary data: if you start treating binary data like it's UTF-8 text, you break the binary data. So don't do that. > But change the locale to en_US.ISO-8859-1, and it doesn't work anymore. So don't do that.

by dexen 15y ago

jrockway 15y ago

Honestly, this post leads me to believe you don't really understand how characters work, and have just gotten lucky so far. Do you really think it's acceptable to accept input, process it, and end up with REPLACEMENT CHARACTER because your string handling got fucked up? Sure, it's valid UTF-8, but it's the Wrong Answer. Plain and simple: by not handling Unicode, you are not handling Unicode. Echoing back UTF-8 when you get UTF-8 input is handling binary, not handling text. Yes, I know this works for your particular application, but it not correct in general. It won't work at all if you ever have any data sources or sinks that are not UTF-8, and it won't work if you actually have to treat the data as text ("match all characters that are whitespace"). Yes, some PHP functions take binary blobs that contain UTF-8 octets and return correct-looking answers. That's all they do, though; that's handling UTF-8, not handling Unicode. In Perl, characters are abstracted away from their character encoding. Perl strings are not UTF-8 or Latin-1 or anything; they are character strings. That makes producing correct results much easier; if you try to write a Perl character to a file, it will say, "hey, you can't expose characters to the outside world, you need to pick a character encoding, encode, and output that". Your PHP method just assumes everything is UTF-8, which is simply not a safe assumption to make. (Consider the case of binary data: if you start treating binary data like it's UTF-8 text, you break the binary data.) Also, regarding LANG, yes, that will change your terminal's encoding. So if you take one of your UTF-8 blobs and print it when the LANG is en_US.UTF-8, it will show up. But change the locale to en_US.ISO-8859-1, and it doesn't work anymore. That's because you can't just print binary blobs to the terminal and expect it to interpret that as text.

ars 15y ago
> Echoing back UTF-8 when you get UTF-8 input is handling binary, not handling text. Nope. UTF-8 is text. We don't live in an ascii world anymore, text requires all 8 bits and a variable number of bytes per character. > It won't work at all if you ever have any data sources or sinks that are not UTF-8 So don't do that. (Or convert as necessary.) > and it won't work if you actually have to treat the data as text ("match all characters that are whitespace"). Wrong. Actually it works just fine. > Yes, some PHP functions take binary blobs that contain UTF-8 octets and return correct-looking answers. That's all they do, though; that's handling UTF-8, not handling Unicode. So? Since that's completely sufficient to do the job why do you want more? > In Perl, characters are abstracted away from their character encoding. Why? Why would I want to do that? Is this complexity for the sake of complexity? > Your PHP method just assumes everything is UTF-8, which is simply not a safe assumption to make. Yes, actually it is a safe assumption for the simple reason that the only types of strings I handle are UTF-8. In todays world there is no reason to use anything else. All other character sets are obsolete and should never be used. And if you do need to handle one then convert as necessary, or define your app to handle only that character set. Programs that need to handle multiple character sets in the same applicate are extremely rare. There is no need for extreme complexity just to handle the rare case. > Consider the case of binary data: if you start treating binary data like it's UTF-8 text, you break the binary data. So don't do that. > But change the locale to en_US.ISO-8859-1, and it doesn't work anymore. So don't do that.