3 ms·

This depends very much on the sort of work one does, I think. If there are only strings made of characters, it can be difficult to form certain sequences of by

by to3m 14y ago

This depends very much on the sort of work one does, I think. If there are only strings made of characters, it can be difficult to form certain sequences of bytes. For example, (uint8_t[]){0x80,0x80} isn't valid UTF-8; one could create the string "\x80\x80", but this is actually four bytes in size. Not great for efficiency, no good for interoperability. To create every possible array of bytes, arrays of bytes must be a data type too. And these arrays of bytes must be able to turn into strings, and vice versa, because the data read from and written to sockets and files and streams and what-have-you is most naturally of array-of-byte type rather than string. Perhaps mandating UTF-8 as the string encoding, leaving alternative encodings up to the programmer, would be reasonable these days, I suppose - if a little rude. Another advantage to having arrays of bytes in addition to strings of chars is that arrays of bytes support efficient random access. (Javascript looks like an example of this playing itself out, from the little I've used it so far. Websockets has alternative text/binary frames, which appear to be a later addition, and there's this ArrayBuffer thing which I believe was prompted by WebGL's requirement for efficient arrays of unadorned bytes.)

snogglethorpe 14y ago
> Perhaps mandating UTF-8 as the string encoding, leaving alternative encodings up to the programmer, would be reasonable these days, I suppose - if a little rude. It really works quite well, especially "these days" (maybe not so much in the '90s), and has the—huge—advantage that it's very simple and lightweight, and avoids the very tricky issue of re-encoding (Guaranteed To Make Your Life Hell). The python3/java/microsoft/whatever solution is to add an enormously heavyweight and complex layer that attempts to interpret and re-encode the contents of strings, with often dubious benefits. My experience suggest that the latter method is in many cases a classic case of over-engineering, where for a wide swathe of applications, the downsides—complexity and cost—exceed the upsides. But because these languages (or systems, in the case of MS) require this at a fundamental level, all applications pay the costs, even those that don't gain any benefit. [Given the encoding jungle of the '80s/'90s, it's to some degree understandable that MS/Java went the way they did (though UTF-8 was around back then, and would have at least made a much better canonical representation). That python3 dove into that swamp is a bit more mystifying given the rather clear signs that the future will essentially be Unicode encoded with UTF-8...]

by to3m 14y ago

snogglethorpe 14y ago

> Perhaps mandating UTF-8 as the string encoding, leaving alternative encodings up to the programmer, would be reasonable these days, I suppose - if a little rude. It really works quite well, especially "these days" (maybe not so much in the '90s), and has the—huge—advantage that it's very simple and lightweight, and avoids the very tricky issue of re-encoding (Guaranteed To Make Your Life Hell). The python3/java/microsoft/whatever solution is to add an enormously heavyweight and complex layer that attempts to interpret and re-encode the contents of strings, with often dubious benefits. My experience suggest that the latter method is in many cases a classic case of over-engineering, where for a wide swathe of applications, the downsides—complexity and cost—exceed the upsides. But because these languages (or systems, in the case of MS) require this at a fundamental level, all applications pay the costs, even those that don't gain any benefit. [Given the encoding jungle of the '80s/'90s, it's to some degree understandable that MS/Java went the way they did (though UTF-8 was around back then, and would have at least made a much better canonical representation). That python3 dove into that swamp is a bit more mystifying given the rather clear signs that the future will essentially be Unicode encoded with UTF-8...]