5 ms·

>> Don't accept some latin-1 data. Accept only UTF-8 I think I get it. You are a web programmer and hence always get encoding specified and are never fscked by

by berntb 15y ago

>> Don't accept some latin-1 data. Accept only UTF-8 I think I get it. You are a web programmer and hence always get encoding specified and are never fscked by something else? Or you've never had the problem of recognizing whitespace (as was the jrockway example) or sorting? (English isn't my first language, but I'm not good with UTF-8 anyway. I tried to understand why you guys are talking past each other.)

ars 15y ago
We're talking about PHP so obviously we are talking about web programming. And if you specify one encoding and get a different one then exactly how do you plan to handle that? Obviously you need to know what encoding you get. Recognizing whitespace works just fine in PHP - you can use ordinary string function and they will work with UTF-8 for that. Sorting is more complicated, but I typically sort in the database which handles UTF-8 sorting correctly for me. You should learn more about UTF-8. There is no reason to use any other character set.
- dexen 15y ago
  > There is no reason to use any other character set. coughencodingcough ;-) There is a good reason to use another in general: http://man.cat-v.org/plan_9/6/utf http://man.cat-v.org/plan_9/6/utf http://man.cat-v.org/plan_9/2/rune http://man.cat-v.org/plan_9/2/rune Plan 9 uses UTF-8 for IPC and most simple string handling, but programs that do more involved string processing (text editors, regular expressions stuff etc), use Runes instead. Runes are similar to UTF-32, one Rune directly represents one character. Most of standard library string functions come in two varieties: using UTF-8 and Runes.
  - ars 15y ago
    Actually UTF-8 is both an encoding and a character set. The character set happens to be "all of them". UTF-32 is a very bad idea. You have issues with nulls and byte order. And on top of that it's basically useless since it solves no problems at all, while wasting space.
    - dexen 15y ago
      With UTF-8 certain common operations take long. Individual character access is O(n); accessing all characters in a string is O(n log n). Even just cutting a string is O(n). For some applications that'd be too slow. Implementation of regex and other features is complex. Rune is not UTF-32, but a similar constant-width format. It is not multibyte, but rather single-int-per-character. Gotta stress this point: never interpreted as bytes. Resides only in process memory, not in files, nor for IPC. Held in arrays of of 32-bit ints, not bytes, thus no problem with nulls. There is a trailing zero int, quite like in good old strings. Individual Rune character access (again, an int, not byte) is O(1); various string processing functions are simple to implement for Runes; character ranges are easy to compare, etc., strings have `predictable' length, etc. etc. As an internal-only format, it may be improved at will. Indeed, once it was -- from 16 bit short to 32 bit int, to support newer Unicode character set ;-) The change was internal to library and header files; programs were just re-compiled.
    - jrockway 15y ago
      No, the character set is "Unicode". UTF-8, UTF-16, UCS-4, etc. are the character encoding. A character encoding is an algorithm for reading/writing characters from/to octet streams. A character set is ... a set of characters. (Unicode has additional semantics; collation, etc.) As to why you'd waste space with a 32-bit character representation: algorithmic efficiency. UTF-8 is efficient in storage, but it's not efficient for string operations: splitting a string is O(n), finding the nth character is O(n), determining the length of the string is O(n), etc. When each character takes the same amount of memory, though, these operations become constant time. Finally, nulls don't cause problems because these are character arrays, not null-terminated octet arrays ("c strings"). Most big C apps don't use null-terminated octet arrays anymore; they use something like bstrings, or std::string, or their own homegrown implementation (SVPV in perl, etc.).
- berntb 15y ago
  >>We're talking about PHP so obviously we are talking about web programming. Uploaded files etc are also so simple for web programmers?
  - ars 15y ago
    Either you know what the character set is or you don't. If you know then convert it into something sane and PHP functions work on it just fine. If you don't then you are stuck in any case, and PHP doesn't hurt anything since it's binary safe. And presumable since you don't know what it is you avoid using any text functions, and treat it as binary.
dexen 15y ago
Sometimes software gets data with no encoding specified, or with wrong encoding. May be result of a bug, transmission glitch or attempt at hacking the software. As jrockway wrote, you can't trust data from user. The software must handle every input in a predictable way. For wrong encoding, it may stop and signal error, or may continue processing the data -- depends on owner's wishes. But must not do anything stupid, no undefined behavior. Where doing something stupid could be giving user information he may not access (like SQL injections; IIRC wrongly-encoded data was successfully used for SQL injections by fooling code that should escape it), damaging data stored in system or saving invalid data if that would break consistency or any work later on. Two things to be done: 1) instruct browser and other systems that work with your software to send and read UTF-8 2) handle situations when something else is sent instead With UTF-8, the 2) is mostly done by PHP's mb_...() functions, IF configured correctly.
- ars 15y ago
  > For wrong encoding, it may stop and signal error You can't tell though. If you get data with the wrong encoding it's almost impossible to tell. At a minimum you'll need to know what language, and from that you may be able to check. But in that case what if the language is wrong?

by berntb 15y ago

ars 15y ago

We're talking about PHP so obviously we are talking about web programming. And if you specify one encoding and get a different one then exactly how do you plan to handle that? Obviously you need to know what encoding you get. Recognizing whitespace works just fine in PHP - you can use ordinary string function and they will work with UTF-8 for that. Sorting is more complicated, but I typically sort in the database which handles UTF-8 sorting correctly for me. You should learn more about UTF-8. There is no reason to use any other character set.

dexen 15y ago
> There is no reason to use any other character set. coughencodingcough ;-) There is a good reason to use another in general: http://man.cat-v.org/plan_9/6/utf http://man.cat-v.org/plan_9/6/utf http://man.cat-v.org/plan_9/2/rune http://man.cat-v.org/plan_9/2/rune Plan 9 uses UTF-8 for IPC and most simple string handling, but programs that do more involved string processing (text editors, regular expressions stuff etc), use Runes instead. Runes are similar to UTF-32, one Rune directly represents one character. Most of standard library string functions come in two varieties: using UTF-8 and Runes.
- ars 15y ago
  Actually UTF-8 is both an encoding and a character set. The character set happens to be "all of them". UTF-32 is a very bad idea. You have issues with nulls and byte order. And on top of that it's basically useless since it solves no problems at all, while wasting space.
  - dexen 15y ago
    With UTF-8 certain common operations take long. Individual character access is O(n); accessing all characters in a string is O(n log n). Even just cutting a string is O(n). For some applications that'd be too slow. Implementation of regex and other features is complex. Rune is not UTF-32, but a similar constant-width format. It is not multibyte, but rather single-int-per-character. Gotta stress this point: never interpreted as bytes. Resides only in process memory, not in files, nor for IPC. Held in arrays of of 32-bit ints, not bytes, thus no problem with nulls. There is a trailing zero int, quite like in good old strings. Individual Rune character access (again, an int, not byte) is O(1); various string processing functions are simple to implement for Runes; character ranges are easy to compare, etc., strings have `predictable' length, etc. etc. As an internal-only format, it may be improved at will. Indeed, once it was -- from 16 bit short to 32 bit int, to support newer Unicode character set ;-) The change was internal to library and header files; programs were just re-compiled.
  - jrockway 15y ago
    No, the character set is "Unicode". UTF-8, UTF-16, UCS-4, etc. are the character encoding. A character encoding is an algorithm for reading/writing characters from/to octet streams. A character set is ... a set of characters. (Unicode has additional semantics; collation, etc.) As to why you'd waste space with a 32-bit character representation: algorithmic efficiency. UTF-8 is efficient in storage, but it's not efficient for string operations: splitting a string is O(n), finding the nth character is O(n), determining the length of the string is O(n), etc. When each character takes the same amount of memory, though, these operations become constant time. Finally, nulls don't cause problems because these are character arrays, not null-terminated octet arrays ("c strings"). Most big C apps don't use null-terminated octet arrays anymore; they use something like bstrings, or std::string, or their own homegrown implementation (SVPV in perl, etc.).
berntb 15y ago
>>We're talking about PHP so obviously we are talking about web programming. Uploaded files etc are also so simple for web programmers?
- ars 15y ago
  Either you know what the character set is or you don't. If you know then convert it into something sane and PHP functions work on it just fine. If you don't then you are stuck in any case, and PHP doesn't hurt anything since it's binary safe. And presumable since you don't know what it is you avoid using any text functions, and treat it as binary.

dexen 15y ago

Sometimes software gets data with no encoding specified, or with wrong encoding. May be result of a bug, transmission glitch or attempt at hacking the software. As jrockway wrote, you can't trust data from user. The software must handle every input in a predictable way. For wrong encoding, it may stop and signal error, or may continue processing the data -- depends on owner's wishes. But must not do anything stupid, no undefined behavior. Where doing something stupid could be giving user information he may not access (like SQL injections; IIRC wrongly-encoded data was successfully used for SQL injections by fooling code that should escape it), damaging data stored in system or saving invalid data if that would break consistency or any work later on. Two things to be done: 1) instruct browser and other systems that work with your software to send and read UTF-8 2) handle situations when something else is sent instead With UTF-8, the 2) is mostly done by PHP's mb_...() functions, IF configured correctly.

ars 15y ago
> For wrong encoding, it may stop and signal error You can't tell though. If you get data with the wrong encoding it's almost impossible to tell. At a minimum you'll need to know what language, and from that you may be able to check. But in that case what if the language is wrong?