3 ms·

What sort of properties would you consider a "textual string" to have that a "binary string" would lack? Offhand I'd assume you mean some of the following, but

What sort of properties would you consider a "textual string" to have that a "binary string" would lack? Offhand I'd assume you mean some of the following, but I'm curious if /my/ assumption about your needs is correct and complete. (Also, note that some of these may be implied by values elsewhere in the specification, it isn't strictly a data structure.) * flag:isValidated * targetEncoding * flag:isNormalized * targetNormalization * lengthMemory (not including the implicit size of the type's metadata) * lengthCodepoints (distinct encoding elements) * length(Some word meaning complete display elements, but not the actual width, height, etc.) Do you also need actual random access to individual codepoints/"full characters"? Or is the assumption that if such a scan is in progress the entire sequence or major parts of it are being fully-scanned? What if, within a given function call*, the underlying storage could be referenced with zero-copying and substrings made? (On exiting that context they'd be copied if retained.) It might be useful to have library iterators that split on say, a single given character, segments up to a maximum memory size of N (and scanning backwards for the last "full character" or at least complete rune). Of course, I believe that, by /default/ a program should perform /binary/ interaction with files including standard in/out/err. If there is a process for converting such binary input/output it should assume and produce data in UTF-8 (in NFC normalization) (when converting to/from "text" types) as a default, which should be easy to globally over-ride or set specifically for a given open file (and change mid use). Of course I don't normally need to care about the encoding of data directly; libraries with parsers and writers typically do that as well as other encoding/decoding for that format for me. Everything else tends to be copy / compare and do not change, or making filenames (mostly append bits of "should all be UTF-8"). Actual, hard core, manipulation of encoded streams seems far more like a composition editing issue (A specific editor where default input dialogs are insufficient) or something that underlying libraries need to worry about.

hsivonen 9y ago
In practice, use cases don't justify strings precalculating their number of code points or grapheme clusters. Pixel width isn't a property of a string alone, since it depends on font and the container to render it in (for line breaks). So a string should know its number of code units. Validity information should be part of the type system and not runtime flags. I.e. you have a different type for arbitrary bytes and for valid UTF-8. (See IDNA for why normalization guarantee in infra over time is problematic.)
- mjevans 9y ago
  Having thought about it you're probably right. That properly belongs on a different kind of wrapper around this data structure or ones like it. Maybe in that /specific/ and /rare/ use case it even makes sense to store the underlying data in a wide data structure; but my supposition is that handling it as a series of fragments (which would have a length in memory and in code-points sizes) would be the answer. If addressing in such a way replacement and other editing operations seem likely.
masklinn 9y ago
> What sort of properties would you consider a "textual string" to have that a "binary string" would lack? Being a unicode stream correctly reified using a specified (or specific) encoding. So: * encoding possibly (the ABI could also specify it) * in-memory length probably (could be a stream but that would require some sort of stream/iterator support at the ABI level), though you'd probably use the length in code units and match that to the encoding for in-memory length (if the encoding is not hard-coded) * validated... in the sense that the stream could be garbage which may or may match the encoding above? No, this would be intrinsic. * normalised, no * length in codepoints, no * length in grapheme clusters, no > Do you also need actual random access to individual codepoints/"full characters"? At the ABI level? Absolutely not.
- mjevans 9y ago
  I agree with everything EXCEPT for the validation being intrinsic. If the validation is /required/ after every operation then a lot of processing must happen on the underlying storage after EVERY operation to ensure it is still valid. If the validation is deferred (such as until programmer request or possibly when emitted out of program) then those checks are also combined in to one final operation; hence the reason for tracking 'is this validated'. Similar logic also applies to normalization (which opens options for optimizations in comparisons among other things).
  - masklinn 9y ago
    > If the validation is /required/ after every operation The only thing that is required is that the data be valid, anything else would be UB. I don't know of any properly implemented text processing instruction which would make valid text invalid, and thus there are almost no operations which would require validation. The only point at which you'd need validation (aside from bytes to text) is when you're trying to play tricky bugger and do text processing via non-text instruction for performance gains. > a lot of processing must happen on the underlying storage after EVERY operation to ensure it is still valid. Only if your storage is insane and does not understand the concept of text e.g. mysql or files. And then yes, I would certainly want whatever garbage these output to be checked every damn time before somebody tells me "yeah it's text just trust me". In fact that's more or less how every language with an actual concept of text separate from bytes/arbitrary data treats files. > If the validation is deferred If the validation is deferred you need to check at runtime before each text-processing instruction if it's working on valid text or not, and if that is put in the hand of developers (to avoid paying the cost for every instruction)… we know how "just check them" null pointers end up.

hsivonen 9y ago

In practice, use cases don't justify strings precalculating their number of code points or grapheme clusters. Pixel width isn't a property of a string alone, since it depends on font and the container to render it in (for line breaks). So a string should know its number of code units. Validity information should be part of the type system and not runtime flags. I.e. you have a different type for arbitrary bytes and for valid UTF-8. (See IDNA for why normalization guarantee in infra over time is problematic.)

mjevans 9y ago
Having thought about it you're probably right. That properly belongs on a different kind of wrapper around this data structure or ones like it. Maybe in that /specific/ and /rare/ use case it even makes sense to store the underlying data in a wide data structure; but my supposition is that handling it as a series of fragments (which would have a length in memory and in code-points sizes) would be the answer. If addressing in such a way replacement and other editing operations seem likely.

masklinn 9y ago

> What sort of properties would you consider a "textual string" to have that a "binary string" would lack? Being a unicode stream correctly reified using a specified (or specific) encoding. So: * encoding possibly (the ABI could also specify it) * in-memory length probably (could be a stream but that would require some sort of stream/iterator support at the ABI level), though you'd probably use the length in code units and match that to the encoding for in-memory length (if the encoding is not hard-coded) * validated... in the sense that the stream could be garbage which may or may match the encoding above? No, this would be intrinsic. * normalised, no * length in codepoints, no * length in grapheme clusters, no > Do you also need actual random access to individual codepoints/"full characters"? At the ABI level? Absolutely not.

mjevans 9y ago
I agree with everything EXCEPT for the validation being intrinsic. If the validation is /required/ after every operation then a lot of processing must happen on the underlying storage after EVERY operation to ensure it is still valid. If the validation is deferred (such as until programmer request or possibly when emitted out of program) then those checks are also combined in to one final operation; hence the reason for tracking 'is this validated'. Similar logic also applies to normalization (which opens options for optimizations in comparisons among other things).
- masklinn 9y ago
  > If the validation is /required/ after every operation The only thing that is required is that the data be valid, anything else would be UB. I don't know of any properly implemented text processing instruction which would make valid text invalid, and thus there are almost no operations which would require validation. The only point at which you'd need validation (aside from bytes to text) is when you're trying to play tricky bugger and do text processing via non-text instruction for performance gains. > a lot of processing must happen on the underlying storage after EVERY operation to ensure it is still valid. Only if your storage is insane and does not understand the concept of text e.g. mysql or files. And then yes, I would certainly want whatever garbage these output to be checked every damn time before somebody tells me "yeah it's text just trust me". In fact that's more or less how every language with an actual concept of text separate from bytes/arbitrary data treats files. > If the validation is deferred If the validation is deferred you need to check at runtime before each text-processing instruction if it's working on valid text or not, and if that is put in the hand of developers (to avoid paying the cost for every instruction)… we know how "just check them" null pointers end up.