7 ms·
No, this is not the proper way to handle UTF-8. PHP is binary-clean, meaning that binary is left untouched. This has nothing to do with UTF-8 except that if t
by jrockway 15y ago
No, this is not the proper way to handle UTF-8. PHP is binary-clean, meaning that binary is left untouched. This has nothing to do with UTF-8 except that if the user submits a form encoded in Latin-1 and you want to produce a UTF-8-encoded result, you're fucked.
- ars 15y agoExcept you aren't going to get a Latin-1 form because you told the browser to send you UTF-8 forms. You can use: <FORM accept-charset="UTF-8"> Or just leave it out since the browser will submit all forms using the character set of the page. So yes, UTF-8 in PHP is pretty easy - and I speak from experience, not theory.
- jrockway 15y agoOK, but realize that you are treating text as binary. This isn't "handling unicode", it's "doing nothing". (Also, "trust the client" as your correctness mechanism? Not very safe.) So far, you've gotten lucky because your browser wants and provides UTF-8-encoded text, your xterm assumes everything is UTF-8, your filesystem metadata is UTF-8-encoded text, constants in your source code are UTF-8-encoded opaque blobs, and your database results are returned as UTF-8-encoded text. The second you throw in some other source, things are going to break, and you will need a plan for how to handle things. Otherwise, you inadvertently end up with garbage responses or a database where half the data is UTF-8 and the other half is Latin-1. Not good.
- ars 15y agoSo don't do that? Why would I include some other bad source if it's going to mess up my data? Also a nice property of UTF-8 is that it can be recovered midstream. So if the client sends me bad encodings, he'll get back bad encodings, but it won't harm anything else. For example a bad encoding followed by a double quote can't do anything to the double quote.
- jrockway 15y agoThe problem is that if you accept, say, some latin-1 data, stash it away in the database unknowingly, and later treat it as if it were a UTF-8 string, that is problem. (In some cases, it's actually a security problem because of the memory corruption that could result from treating latin-1 as utf8. But security aside, your app will display garbage because your database is garbage, and that's bad.)
- ars 15y agoThis is like the 5th time I've told you this: So don't do that. Don't accept some latin-1 data. Accept only UTF-8, and if you have to accept latin-1 you have two choices: Convert it to UTF-8, or use only latin-1. > In some cases, it's actually a security problem because of the memory corruption that could result from treating latin-1 as utf8. What are you talking about? There is no memory corruption. UTF-16 can have serious issues, but UTF-8 is quite resilient.
- berntb 15y ago>> Don't accept some latin-1 data. Accept only UTF-8 I think I get it. You are a web programmer and hence always get encoding specified and are never fscked by something else? Or you've never had the problem of recognizing whitespace (as was the jrockway example) or sorting? (English isn't my first language, but I'm not good with UTF-8 anyway. I tried to understand why you guys are talking past each other.)
- ars 15y agoWe're talking about PHP so obviously we are talking about web programming. And if you specify one encoding and get a different one then exactly how do you plan to handle that? Obviously you need to know what encoding you get. Recognizing whitespace works just fine in PHP - you can use ordinary string function and they will work with UTF-8 for that. Sorting is more complicated, but I typically sort in the database which handles UTF-8 sorting correctly for me. You should learn more about UTF-8. There is no reason to use any other character set.
- dexen 15y ago> you are treating text as binary Nope; it's treated differently by different functions. Like ars wrote, with UTF-8 a lot of properties of old ASCII strings holds. A lot of PHP functions did not need any changes to handle UTF-8 just fine, for example explode() and join(). Other functions do handle actual characters -- when they must. In contrast, using UTF-16 or UTF-32 would require treating strings as opaque binary blobs in some cases, because those encodings lack certain properties of UTF-8. For example, the old ASCII-age strpos() is reliable with UTF-8, but unreliable with UTF-16. > Also, "trust the client" as your correctness mechanism? Not very safe. Never the case here. Standard library is trusted, and the average developer team is not much more secure than maintainers of such libraries. I know my code is not better than the standard library, anyway, so I'll rather rely on it to handle correctness. You use two kinds of functions: some care about properties of characters -- like mb_strtoupper(); others just work on strings -- like explode(). Only the former kind matters in aspect of correctness. When such function tries to operate on the mis-encoded character, it follows the Unicode standard: replaces the character with U+FFFD, a.k.a. `REPLACEMENT CHARACTER'. Result is valid UTF-8, with one character replaced with the marker. Any subsequent processing will go just fine, unless you want to stop and warn the user. The condition is easy to detect: there is one unique byte sequence for the REPLACEMENT CHARACTER, and it's clear where in the string the problem was. UTF-8 was designed to be self-synchronizing in case of such errors. > This isn't "handling unicode", it's "doing nothing" That ``doing nothing'' was my original point; thanks for re-iterating? You -- the developer -- do nothing, in particular you don't insert boilerplate code everywhere (and what if you forgot one rarely used execution path?). The functions that must handle individual characters perform the mundane work. Unicode is handled where it has to be -- and only there. > (...) your xterm assumes everything is UTF-8, (...) Nothing's assumed there; the process is instructed to do so via configuration. On POSIX platforms, that's environment variables: $LC_ALL and related. The main point of such configuration is that you can adjust whole system and make programs speak common language to one another. And they do -- following your instruction. Unless explicitly instructed to do otherwise -- when interfacing with remote systems and a different encoding has been agreed upon. > The second you throw in some other source, things are going to break, and you will need a plan for how to handle things. When you extend software to interface another software, you either configure both to use the same protocol (here: encoding), or implement protocol (here: encoding) negotiation. You -- the developer -- configure or implement it once, as part of creating the interface between them.