3 ms·
> What sort of properties would you consider a "textual string" to have that a "binary string" would lack? Being a unicode stream correctly reified using a spe
by masklinn 9y ago
> What sort of properties would you consider a "textual string" to have that a "binary string" would lack?
Being a unicode stream correctly reified using a specified (or specific) encoding.
So:
* encoding possibly (the ABI could also specify it)
* in-memory length probably (could be a stream but that would require some sort of stream/iterator support at the ABI level), though you'd probably use the length in code units and match that to the encoding for in-memory length (if the encoding is not hard-coded)
* validated... in the sense that the stream could be garbage which may or may match the encoding above? No, this would be intrinsic.
* normalised, no
* length in codepoints, no
* length in grapheme clusters, no
> Do you also need actual random access to individual codepoints/"full characters"?
At the ABI level? Absolutely not.
- mjevans 9y agoI agree with everything EXCEPT for the validation being intrinsic. If the validation is /required/ after every operation then a lot of processing must happen on the underlying storage after EVERY operation to ensure it is still valid. If the validation is deferred (such as until programmer request or possibly when emitted out of program) then those checks are also combined in to one final operation; hence the reason for tracking 'is this validated'. Similar logic also applies to normalization (which opens options for optimizations in comparisons among other things).
- masklinn 9y ago> If the validation is /required/ after every operation The only thing that is required is that the data be valid, anything else would be UB. I don't know of any properly implemented text processing instruction which would make valid text invalid, and thus there are almost no operations which would require validation. The only point at which you'd need validation (aside from bytes to text) is when you're trying to play tricky bugger and do text processing via non-text instruction for performance gains. > a lot of processing must happen on the underlying storage after EVERY operation to ensure it is still valid. Only if your storage is insane and does not understand the concept of text e.g. mysql or files. And then yes, I would certainly want whatever garbage these output to be checked every damn time before somebody tells me "yeah it's text just trust me". In fact that's more or less how every language with an actual concept of text separate from bytes/arbitrary data treats files. > If the validation is deferred If the validation is deferred you need to check at runtime before each text-processing instruction if it's working on valid text or not, and if that is put in the hand of developers (to avoid paying the cost for every instruction)… we know how "just check them" null pointers end up.