2 ms·
Honestly, this post leads me to believe you don't really understand how characters work, and have just gotten lucky so far. Do you really think it's acceptable
by jrockway 15y ago
Honestly, this post leads me to believe you don't really understand how characters work, and have just gotten lucky so far. Do you really think it's acceptable to accept input, process it, and end up with REPLACEMENT CHARACTER because your string handling got fucked up? Sure, it's valid UTF-8, but it's the Wrong Answer.
Plain and simple: by not handling Unicode, you are not handling Unicode. Echoing back UTF-8 when you get UTF-8 input is handling binary, not handling text. Yes, I know this works for your particular application, but it not correct in general. It won't work at all if you ever have any data sources or sinks that are not UTF-8, and it won't work if you actually have to treat the data as text ("match all characters that are whitespace").
Yes, some PHP functions take binary blobs that contain UTF-8 octets and return correct-looking answers. That's all they do, though; that's handling UTF-8, not handling Unicode.
In Perl, characters are abstracted away from their character encoding. Perl strings are not UTF-8 or Latin-1 or anything; they are character strings. That makes producing correct results much easier; if you try to write a Perl character to a file, it will say, "hey, you can't expose characters to the outside world, you need to pick a character encoding, encode, and output that". Your PHP method just assumes everything is UTF-8, which is simply not a safe assumption to make.
(Consider the case of binary data: if you start treating binary data like it's UTF-8 text, you break the binary data.)
Also, regarding LANG, yes, that will change your terminal's encoding. So if you take one of your UTF-8 blobs and print it when the LANG is en_US.UTF-8, it will show up. But change the locale to en_US.ISO-8859-1, and it doesn't work anymore. That's because you can't just print binary blobs to the terminal and expect it to interpret that as text.
- ars 15y ago> Echoing back UTF-8 when you get UTF-8 input is handling binary, not handling text. Nope. UTF-8 is text. We don't live in an ascii world anymore, text requires all 8 bits and a variable number of bytes per character. > It won't work at all if you ever have any data sources or sinks that are not UTF-8 So don't do that. (Or convert as necessary.) > and it won't work if you actually have to treat the data as text ("match all characters that are whitespace"). Wrong. Actually it works just fine. > Yes, some PHP functions take binary blobs that contain UTF-8 octets and return correct-looking answers. That's all they do, though; that's handling UTF-8, not handling Unicode. So? Since that's completely sufficient to do the job why do you want more? > In Perl, characters are abstracted away from their character encoding. Why? Why would I want to do that? Is this complexity for the sake of complexity? > Your PHP method just assumes everything is UTF-8, which is simply not a safe assumption to make. Yes, actually it is a safe assumption for the simple reason that the only types of strings I handle are UTF-8. In todays world there is no reason to use anything else. All other character sets are obsolete and should never be used. And if you do need to handle one then convert as necessary, or define your app to handle only that character set. Programs that need to handle multiple character sets in the same applicate are extremely rare. There is no need for extreme complexity just to handle the rare case. > Consider the case of binary data: if you start treating binary data like it's UTF-8 text, you break the binary data. So don't do that. > But change the locale to en_US.ISO-8859-1, and it doesn't work anymore. So don't do that.
- dexen 15y ago> Do you really think it's acceptable to accept input, process it, and end up with REPLACEMENT CHARACTER because your string handling got fucked up? Sure, it's valid UTF-8, but it's the Wrong Answer. There just isn't One True Way about validating user input; please don't get religious there. For sake of an example, if an user stated his birth year as 1905, do you accept it? The answer is, ``depends on who pays the bill'', in other words, on business rules. Sometimes it makes sense to accept and store data that's unprobable or flat out wrong -- perhaps no entity cares, ever; perhaps the user knows better than us, developers, this time. Sometimes data must fit strict criteria -- say in processing legal documents -- or else you raise big red error message and rollback transaction. And raising an error on that REPLACEMENT CHARACTER is trivial. Do so when you need. However, don't let rare cases dominate your application with boilerplate code. That's time/effort/money down the drain. That REPLACEMENT CHARACTER ``is valid UTF-8'' was only to point out there won't be any undefined behavior. Thus no weakening of security, which was your point. > Your PHP method just assumes everything is UTF-8, which is simply not a safe assumption to make. That was exclaimed twice in above posts [1][2]. As a developer I pick and choose all the data sources and sinks the app interacts with. It's either an UTF-8-ony entity (source code, the files created by my software, etc.), or my code negotiates some common encoding at the start of communication (the browser, the database, etc.) and perform sconversion if needed. Once, in the implementation of interface to the source/sink. Conversion to UTF-8 is simple, takes O(n) time and O(1) memory. If there is no metadata describing content's encoding, no amount of abstraction will help, ever. You may report an error or try to go on using defaults; again depends on needs of users/owners/whatever. I don't know how you arrived at binary data getting mixed up with textual. The only shared matter is the `string' datatype itself; all else is separate. The execution paths are different: very, very different classes read, mangle and write back user-uploaded binnary files (to re-scale images etc.) and different ones handle text submitted by user, fetched from DB etc. Even the name of uploaded file is held in different variable than the content. Normally most of the files have metadata associated stating the type of the content, either in some header or implied by format. Again, the developer chooses which files are handled by what code, if at all. > In Perl, characters are abstracted away from their character encoding. Good in theory, but usage -- as shown in OP -- is elaborate. Error-prone, one could say. Abstracting stuff away is means, not an end in itself. The elaborate OP suggests the means may hamper achieving the end in some cases. What are you gonna do? > That makes producing correct results much easier; Let's see if others agree with you: http://stackoverflow.com/questions/6162484/why-does-modern-perl-avoid-utf-8-by-default/#6163129 http://stackoverflow.com/questions/6162484/why-does-modern-p... ---- [1] > and convert only when explicitly requested by user or remote system to do so. [2] > When you extend software to interface another software, you either configure both to use the same protocol (here: encoding), or implement protocol (here: encoding) negotiation. EDIT: on a website, what is worse: mojibake (garbled characters) or an error message? May depend on audience: AFAIK some internet users ``understand'' mojibake in some way (may for example attempt to adjust browser's encoding settings and actually get correct text, yay!), but most hate cryptic error messages. God forbid an error message tell them they sent ``illegal character'' [3] ;-) Any distration (modal dialog, workflow interruption) during user registration, order submission etc. may end up scaring away a good deal of potential customers. May be easier for site's owner to actually manually correct the profile/order/whatever, or ask the user to correct it later on. [3] http://forums.thedailywtf.com/forums/p/24359/253469.aspx#253469 http://forums.thedailywtf.com/forums/p/24359/253469.aspx#253...