3 ms·

My favorite bit of learning with Rust strings was the WTF-8 format: https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/ Implemented: https:

My favorite bit of learning with Rust strings was the WTF-8 format: https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/ Implemented: https://github.com/rust-lang/rust/blob/master/src/libstd/sys/common/wtf8.rs https://github.com/rust-lang/rust/blob/master/src/libstd/sys...

kibwen 11y ago
WTF-8 is great, and lest anyone think that it's intended to be a joke it's worth mentioning that it's actually a very useful thing: "WTF-8 (Wobbly Transformation Format − 8-bit) is a superset of UTF-8 that encodes surrogate code points if they are not in a pair. It represents, in a way compatible with UTF-8, text from systems such as JavaScript and Windows that use UTF-16 internally but don’t enforce the well-formedness invariant that surrogates must be paired." Though note that it's only suitable for internal usage (as in, you wouldn't expose an API that uses WTF-8), so it's not something that you need to actually think about when learning Rust. :P
- derefr 11y ago
  I don't really get the point of it; I'd think that, properly, what you'd want is: 1. a plain "buffer of unparsed octets" type (like Erlang's "binary" type), to hold onto your weird invalid UTF-16 (or whatever other weird invalid stuff you like); and 2. a strict, validated-on-construction "sequence of codepoints" type. Going from #1 to #2 wouldn't be a direct cast; it'd instead be a stream-parser function that could be hooked with a callback, which would be handed parsed-and-valid, and invalid-and-left-unparsed, "segments": maybe_parsed_segment() = valid_string_of_codepoints() | binary() ...and would then respond with a valid segment, which could just be "" to discard the input segment. The default callback would presumably respond to a binary() of size N by emitting a "�" string of length N. That's pretty much what we have today, but both sturdier and more extensible.

kibwen 11y ago

WTF-8 is great, and lest anyone think that it's intended to be a joke it's worth mentioning that it's actually a very useful thing: "WTF-8 (Wobbly Transformation Format − 8-bit) is a superset of UTF-8 that encodes surrogate code points if they are not in a pair. It represents, in a way compatible with UTF-8, text from systems such as JavaScript and Windows that use UTF-16 internally but don’t enforce the well-formedness invariant that surrogates must be paired." Though note that it's only suitable for internal usage (as in, you wouldn't expose an API that uses WTF-8), so it's not something that you need to actually think about when learning Rust. :P

derefr 11y ago
I don't really get the point of it; I'd think that, properly, what you'd want is: 1. a plain "buffer of unparsed octets" type (like Erlang's "binary" type), to hold onto your weird invalid UTF-16 (or whatever other weird invalid stuff you like); and 2. a strict, validated-on-construction "sequence of codepoints" type. Going from #1 to #2 wouldn't be a direct cast; it'd instead be a stream-parser function that could be hooked with a callback, which would be handed parsed-and-valid, and invalid-and-left-unparsed, "segments": maybe_parsed_segment() = valid_string_of_codepoints() | binary() ...and would then respond with a valid segment, which could just be "" to discard the input segment. The default callback would presumably respond to a binary() of size N by emitting a "�" string of length N. That's pretty much what we have today, but both sturdier and more extensible.