> Do you really think it's acceptable to accept input, process it, and end up with REPLACEMENT CHARACTER because your string handling got fucked up? Sure, it's

> Do you really think it's acceptable to accept input, process it, and end up with REPLACEMENT CHARACTER because your string handling got fucked up? Sure, it's valid UTF-8, but it's the Wrong Answer. There just isn't One True Way about validating user input; please don't get religious there. For sake of an example, if an user stated his birth year as 1905, do you accept it? The answer is, ``depends on who pays the bill'', in other words, on business rules. Sometimes it makes sense to accept and store data that's unprobable or flat out wrong -- perhaps no entity cares, ever; perhaps the user knows better than us, developers, this time. Sometimes data must fit strict criteria -- say in processing legal documents -- or else you raise big red error message and rollback transaction. And raising an error on that REPLACEMENT CHARACTER is trivial. Do so when you need. However, don't let rare cases dominate your application with boilerplate code. That's time/effort/money down the drain. That REPLACEMENT CHARACTER ``is valid UTF-8'' was only to point out there won't be any undefined behavior. Thus no weakening of security, which was your point. > Your PHP method just assumes everything is UTF-8, which is simply not a safe assumption to make. That was exclaimed twice in above posts [1][2]. As a developer I pick and choose all the data sources and sinks the app interacts with. It's either an UTF-8-ony entity (source code, the files created by my software, etc.), or my code negotiates some common encoding at the start of communication (the browser, the database, etc.) and perform sconversion if needed. Once, in the implementation of interface to the source/sink. Conversion to UTF-8 is simple, takes O(n) time and O(1) memory. If there is no metadata describing content's encoding, no amount of abstraction will help, ever. You may report an error or try to go on using defaults; again depends on needs of users/owners/whatever. I don't know how you arrived at binary data getting mixed up with textual. The only shared matter is the `string' datatype itself; all else is separate. The execution paths are different: very, very different classes read, mangle and write back user-uploaded binnary files (to re-scale images etc.) and different ones handle text submitted by user, fetched from DB etc. Even the name of uploaded file is held in different variable than the content. Normally most of the files have metadata associated stating the type of the content, either in some header or implied by format. Again, the developer chooses which files are handled by what code, if at all. > In Perl, characters are abstracted away from their character encoding. Good in theory, but usage -- as shown in OP -- is elaborate. Error-prone, one could say. Abstracting stuff away is means, not an end in itself. The elaborate OP suggests the means may hamper achieving the end in some cases. What are you gonna do? > That makes producing correct results much easier; Let's see if others agree with you: http://stackoverflow.com/questions/6162484/why-does-modern-perl-avoid-utf-8-by-default/#6163129 http://stackoverflow.com/questions/6162484/why-does-modern-p... ---- [1] > and convert only when explicitly requested by user or remote system to do so. [2] > When you extend software to interface another software, you either configure both to use the same protocol (here: encoding), or implement protocol (here: encoding) negotiation. EDIT: on a website, what is worse: mojibake (garbled characters) or an error message? May depend on audience: AFAIK some internet users ``understand'' mojibake in some way (may for example attempt to adjust browser's encoding settings and actually get correct text, yay!), but most hate cryptic error messages. God forbid an error message tell them they sent ``illegal character'' [3] ;-) Any distration (modal dialog, workflow interruption) during user registration, order submission etc. may end up scaring away a good deal of potential customers. May be easier for site's owner to actually manually correct the profile/order/whatever, or ask the user to correct it later on. [3] http://forums.thedailywtf.com/forums/p/24359/253469.aspx#253469 http://forums.thedailywtf.com/forums/p/24359/253469.aspx#253...