2 ms·
So don't do that? Why would I include some other bad source if it's going to mess up my data? Also a nice property of UTF-8 is that it can be recovered midstr
by ars 15y ago
So don't do that?
Why would I include some other bad source if it's going to mess up my data?
Also a nice property of UTF-8 is that it can be recovered midstream. So if the client sends me bad encodings, he'll get back bad encodings, but it won't harm anything else.
For example a bad encoding followed by a double quote can't do anything to the double quote.
- jrockway 15y agoThe problem is that if you accept, say, some latin-1 data, stash it away in the database unknowingly, and later treat it as if it were a UTF-8 string, that is problem. (In some cases, it's actually a security problem because of the memory corruption that could result from treating latin-1 as utf8. But security aside, your app will display garbage because your database is garbage, and that's bad.)
- ars 15y agoThis is like the 5th time I've told you this: So don't do that. Don't accept some latin-1 data. Accept only UTF-8, and if you have to accept latin-1 you have two choices: Convert it to UTF-8, or use only latin-1. > In some cases, it's actually a security problem because of the memory corruption that could result from treating latin-1 as utf8. What are you talking about? There is no memory corruption. UTF-16 can have serious issues, but UTF-8 is quite resilient.
- berntb 15y ago>> Don't accept some latin-1 data. Accept only UTF-8 I think I get it. You are a web programmer and hence always get encoding specified and are never fscked by something else? Or you've never had the problem of recognizing whitespace (as was the jrockway example) or sorting? (English isn't my first language, but I'm not good with UTF-8 anyway. I tried to understand why you guys are talking past each other.)
- ars 15y agoWe're talking about PHP so obviously we are talking about web programming. And if you specify one encoding and get a different one then exactly how do you plan to handle that? Obviously you need to know what encoding you get. Recognizing whitespace works just fine in PHP - you can use ordinary string function and they will work with UTF-8 for that. Sorting is more complicated, but I typically sort in the database which handles UTF-8 sorting correctly for me. You should learn more about UTF-8. There is no reason to use any other character set.
- dexen 15y ago> There is no reason to use any other character set. coughencodingcough ;-) There is a good reason to use another in general: http://man.cat-v.org/plan_9/6/utf http://man.cat-v.org/plan_9/6/utf http://man.cat-v.org/plan_9/2/rune http://man.cat-v.org/plan_9/2/rune Plan 9 uses UTF-8 for IPC and most simple string handling, but programs that do more involved string processing (text editors, regular expressions stuff etc), use Runes instead. Runes are similar to UTF-32, one Rune directly represents one character. Most of standard library string functions come in two varieties: using UTF-8 and Runes.
- ars 15y agoActually UTF-8 is both an encoding and a character set. The character set happens to be "all of them". UTF-32 is a very bad idea. You have issues with nulls and byte order. And on top of that it's basically useless since it solves no problems at all, while wasting space.
- dexen 15y agoWith UTF-8 certain common operations take long. Individual character access is O(n); accessing all characters in a string is O(n log n). Even just cutting a string is O(n). For some applications that'd be too slow. Implementation of regex and other features is complex. Rune is not UTF-32, but a similar constant-width format. It is not multibyte, but rather single-int-per-character. Gotta stress this point: never interpreted as bytes. Resides only in process memory, not in files, nor for IPC. Held in arrays of of 32-bit ints, not bytes, thus no problem with nulls. There is a trailing zero int, quite like in good old strings. Individual Rune character access (again, an int, not byte) is O(1); various string processing functions are simple to implement for Runes; character ranges are easy to compare, etc., strings have `predictable' length, etc. etc. As an internal-only format, it may be improved at will. Indeed, once it was -- from 16 bit short to 32 bit int, to support newer Unicode character set ;-) The change was internal to library and header files; programs were just re-compiled.