3 ms·

> There is no reason to use any other character set. coughencodingcough ;-) There is a good reason to use another in general: http://man.cat-v.org/plan_9/6/u

by dexen 15y ago

> There is no reason to use any other character set. coughencodingcough ;-) There is a good reason to use another in general: http://man.cat-v.org/plan_9/6/utf http://man.cat-v.org/plan_9/6/utf http://man.cat-v.org/plan_9/2/rune http://man.cat-v.org/plan_9/2/rune Plan 9 uses UTF-8 for IPC and most simple string handling, but programs that do more involved string processing (text editors, regular expressions stuff etc), use Runes instead. Runes are similar to UTF-32, one Rune directly represents one character. Most of standard library string functions come in two varieties: using UTF-8 and Runes.

ars 15y ago
Actually UTF-8 is both an encoding and a character set. The character set happens to be "all of them". UTF-32 is a very bad idea. You have issues with nulls and byte order. And on top of that it's basically useless since it solves no problems at all, while wasting space.
- dexen 15y ago
  With UTF-8 certain common operations take long. Individual character access is O(n); accessing all characters in a string is O(n log n). Even just cutting a string is O(n). For some applications that'd be too slow. Implementation of regex and other features is complex. Rune is not UTF-32, but a similar constant-width format. It is not multibyte, but rather single-int-per-character. Gotta stress this point: never interpreted as bytes. Resides only in process memory, not in files, nor for IPC. Held in arrays of of 32-bit ints, not bytes, thus no problem with nulls. There is a trailing zero int, quite like in good old strings. Individual Rune character access (again, an int, not byte) is O(1); various string processing functions are simple to implement for Runes; character ranges are easy to compare, etc., strings have `predictable' length, etc. etc. As an internal-only format, it may be improved at will. Indeed, once it was -- from 16 bit short to 32 bit int, to support newer Unicode character set ;-) The change was internal to library and header files; programs were just re-compiled.
- jrockway 15y ago
  No, the character set is "Unicode". UTF-8, UTF-16, UCS-4, etc. are the character encoding. A character encoding is an algorithm for reading/writing characters from/to octet streams. A character set is ... a set of characters. (Unicode has additional semantics; collation, etc.) As to why you'd waste space with a 32-bit character representation: algorithmic efficiency. UTF-8 is efficient in storage, but it's not efficient for string operations: splitting a string is O(n), finding the nth character is O(n), determining the length of the string is O(n), etc. When each character takes the same amount of memory, though, these operations become constant time. Finally, nulls don't cause problems because these are character arrays, not null-terminated octet arrays ("c strings"). Most big C apps don't use null-terminated octet arrays anymore; they use something like bstrings, or std::string, or their own homegrown implementation (SVPV in perl, etc.).

by dexen 15y ago

ars 15y ago

Actually UTF-8 is both an encoding and a character set. The character set happens to be "all of them". UTF-32 is a very bad idea. You have issues with nulls and byte order. And on top of that it's basically useless since it solves no problems at all, while wasting space.

dexen 15y ago
With UTF-8 certain common operations take long. Individual character access is O(n); accessing all characters in a string is O(n log n). Even just cutting a string is O(n). For some applications that'd be too slow. Implementation of regex and other features is complex. Rune is not UTF-32, but a similar constant-width format. It is not multibyte, but rather single-int-per-character. Gotta stress this point: never interpreted as bytes. Resides only in process memory, not in files, nor for IPC. Held in arrays of of 32-bit ints, not bytes, thus no problem with nulls. There is a trailing zero int, quite like in good old strings. Individual Rune character access (again, an int, not byte) is O(1); various string processing functions are simple to implement for Runes; character ranges are easy to compare, etc., strings have `predictable' length, etc. etc. As an internal-only format, it may be improved at will. Indeed, once it was -- from 16 bit short to 32 bit int, to support newer Unicode character set ;-) The change was internal to library and header files; programs were just re-compiled.
jrockway 15y ago
No, the character set is "Unicode". UTF-8, UTF-16, UCS-4, etc. are the character encoding. A character encoding is an algorithm for reading/writing characters from/to octet streams. A character set is ... a set of characters. (Unicode has additional semantics; collation, etc.) As to why you'd waste space with a 32-bit character representation: algorithmic efficiency. UTF-8 is efficient in storage, but it's not efficient for string operations: splitting a string is O(n), finding the nth character is O(n), determining the length of the string is O(n), etc. When each character takes the same amount of memory, though, these operations become constant time. Finally, nulls don't cause problems because these are character arrays, not null-terminated octet arrays ("c strings"). Most big C apps don't use null-terminated octet arrays anymore; they use something like bstrings, or std::string, or their own homegrown implementation (SVPV in perl, etc.).