11 ms·
After reading this lengthy how-to, I am tempted to say that PHP may have upper hand when it comes to reliably dealing with UTF-encoded data from different sourc
by dexen 15y ago
After reading this lengthy how-to, I am tempted to say that PHP may have upper hand when it comes to reliably dealing with UTF-encoded data from different sources (including source code) and sinks. At least as long as I default to UTF-8 and convert only when explicitly requested by user or remote system to do so.
PHP has no separate Unicode type, the built-in string type works pretty well for UTF-8. I'd liken its use to the concept of duck-typing (as seen, for example, in Python): if it walks like a duck^W^W UTF-8, swims like a duck^W^W UTF-8, quacks like a duck^W^W UTF-8, it can be handled (consumed and produced) like a duck^W^W UTF-8.
Note in Python Unicode is a separate data type, so the above comparison is related to duck-typing only.
- pornel 15y agoIt's not pretty in PHP either. Many of the Perl issues have their PHP counterparts. • You need to be aware that most built-in string functions won't work and use mb_ equivalents where available (and not all are available). • You need to set internal encoding to UTF-8 for some extensions in php.ini. • You need to normalize all input yourself (thankfully PHP5.3 has Normalizer class. In 5.2 it was a nightmare) • PCRE library doesn't support all uncode ranges (e.g. {InFoo} is unsupported). All gotchas of regexes apply to PHP/PCRE as well. The list goes on… "Binary-safe" strings and UTF-8 cleverness will let you roundtrip characters safely in most cases, but full, correct Unicode support is really hard.
- ars 15y ago> You need to be aware that most built-in string functions won't work Actually the non-mb string functions will work perfectly fine for almost everything. For example splitting on space or comma, or joining strings will work just fine with UTF-8. The only things that don't work are those that split by character position. Even search works fine. > You need to set internal encoding to UTF-8 for some extensions in php.ini. ini_set('default_charset', 'UTF-8'); mb_internal_encoding("UTF-8"); > You need to normalize all input yourself Only if you need to compare against it (for example to see if the username was taken). Most of the time you just store the input as is. For PCRE make sure to add the /u option.
- dexen 15y agoI'd just stick to mb_ereg_...() functions. They follow the standard setlocale(LC_...). They support the standard POSIX regex syntax. They support Unicode always when the platform does (depends only on standard C library) -- and I've seen PHP's PCRE distributed without Unicode support. Last but not least, PCRE suffers from http://swtch.com/~rsc/regexp/regexp1.html http://swtch.com/~rsc/regexp/regexp1.html setlocale(LC_ALL, 'pl_PL.utf-8'); # if not set explicitly defaults to system's $LC_ALL, thus fits most of the time mb_internal_encoding('UTF-8'); mb_ereg_match('[[:alpha:]]', 'Ł') -> true mb_ereg_match('[[:alpha:]]', '♙') -> false # that's a white chess pawn character, U+2659. mb_ereg_match('[[:graph:]]', '♙') -> true
- ars 15y agoI don't like setlocale because usually I'm writing a general application, usable for any language.
- dexen 15y agoI like setlocale() as a central knob that other functions obey. Works a-OK for me on several different websites. Normally my website's code starts with platform defaults, reads configuration early on, and, if need be, issues setlocale(). Should an error be triggered before the config is applied, it would be handled with default locales; I believe that's the only noticeable downside. Hopefully a rare one ;-) setlocale() can be switched at run-time, even multiple times if you want. You could imagine running most of the code with user's preferred locale to output UI or report any errors, switching to another locale to format and send message to a foreign customer, and then back to normal settings to continue with outputing UI, all in one pass of PHP.
- berntb 15y agoSo both PHP and Python have AI in their regexp libs and understand what you mean with [a-z]? :-) Strange that the libs are named PCRE (where "P" stands for "Perl"), then? :-)
- deleted 15y ago[deleted]
- jrockway 15y agoNo, this is not the proper way to handle UTF-8. PHP is binary-clean, meaning that binary is left untouched. This has nothing to do with UTF-8 except that if the user submits a form encoded in Latin-1 and you want to produce a UTF-8-encoded result, you're fucked.
- ars 15y agoExcept you aren't going to get a Latin-1 form because you told the browser to send you UTF-8 forms. You can use: <FORM accept-charset="UTF-8"> Or just leave it out since the browser will submit all forms using the character set of the page. So yes, UTF-8 in PHP is pretty easy - and I speak from experience, not theory.
- jrockway 15y agoOK, but realize that you are treating text as binary. This isn't "handling unicode", it's "doing nothing". (Also, "trust the client" as your correctness mechanism? Not very safe.) So far, you've gotten lucky because your browser wants and provides UTF-8-encoded text, your xterm assumes everything is UTF-8, your filesystem metadata is UTF-8-encoded text, constants in your source code are UTF-8-encoded opaque blobs, and your database results are returned as UTF-8-encoded text. The second you throw in some other source, things are going to break, and you will need a plan for how to handle things. Otherwise, you inadvertently end up with garbage responses or a database where half the data is UTF-8 and the other half is Latin-1. Not good.
- ars 15y agoSo don't do that? Why would I include some other bad source if it's going to mess up my data? Also a nice property of UTF-8 is that it can be recovered midstream. So if the client sends me bad encodings, he'll get back bad encodings, but it won't harm anything else. For example a bad encoding followed by a double quote can't do anything to the double quote.
- jrockway 15y ago