2 ms·
The python nightmare is purely performance due to its use of the UCS-2 or 4 encoding (wchar_t without surrogate pairs). UTF-8 was ruled out for its inability fo
by rurban 8y ago
The python nightmare is purely performance due to its use of the UCS-2 or 4 encoding (wchar_t without surrogate pairs). UTF-8 was ruled out for its inability for direct indexing, but this proved wrong. 4x longer string buffers trash the cache much more than the needed CPU cycles for utf-8 decoding.
perl's problem is that they didn't make room for the original encoding, it's only latin-1 or utf-8 from everything else. so the context needs to store it, like IO, process. and for performance they silently loose the utf-8 flag for 128—255 codepoints.
but all this is still better than the libc/POSIX design failure to use ENV vars, where the encoding is global, not localized. every single string API needs to do a getenv, and its interpretation is action at a distance.
- avian 8y ago> it's only latin-1 or utf-8 There’s no other way, is it? You basically have to have the complete string library re-implemented for each supported multi-byte encoding.
- rurban 8y agoNo. Foreign encodings are internally converted to utf-8 via the Encoding library and IO layers. The string library is only done for latin-1 and utf-8. The only trouble is that you have to remember the original encoding, if you need that information. It's stored nowhere. E.g. to write to a file in the original encoding you have to open it with the correct IO layer. perl6 got that fixed.