3 ms·
> Both rust and go decided to go with Unicode support that is largely based around utf-8 with free transmutations to bytes and almost free conversions from byte
by Sohcahtoa82 7y ago
> Both rust and go decided to go with Unicode support that is largely based around utf-8 with free transmutations to bytes and almost free conversions from bytes. This could have been Python 3 but that path was dismissed without a lot of evidence.
Do you mean that if you have bytes, but you want to send them to a function that expects a string, then it would automatically interpret the bytes as UTF-8?
If so, that violates the "Explicit is better than implicit" part of the Zen of Python. Encoding/Decoding bytes to/from strings shouldn't happen automatically because doing so means you have to make an assumption about the encoding.
- masklinn 7y ago> Do you mean that if you have bytes, but you want to send them to a function that expects a string, then it would automatically interpret the bytes as UTF-8? No, the types are separate and not implicitly converted P2-style, however "unicode strings" are guaranteed to be proper UTF8 so encoding to UTF-8 is completely free, and decoding from UTF8 just requires validating. Python's maintainers rejected this approach because "it doesn't provide non-amortised O(1) access to codepoints", and while Python 3 broke a lot of things they sadly refused to break this one completely useless thing, only to have to come up with PEP 393 a few years later.
- Sohcahtoa82 7y agoAh, that makes sense. Thank you for the clarification.
- Lev1a 7y agoTo add to your earlier dialog partner, here are the doc pages for the relevant Rust functions/methods, embedded with runnable examples: https://doc.rust-lang.org/std/string/struct.String.html#method.as_bytes https://doc.rust-lang.org/std/string/struct.String.html#meth... https://doc.rust-lang.org/std/string/struct.String.html#method.from_utf8 https://doc.rust-lang.org/std/string/struct.String.html#meth... https://doc.rust-lang.org/std/primitive.str.html#method.as_bytes https://doc.rust-lang.org/std/primitive.str.html#method.as_b... https://doc.rust-lang.org/std/str/fn.from_utf8.html https://doc.rust-lang.org/std/str/fn.from_utf8.html Also, as explained in those docs, if and when you are absolutely sure that the Vec or slice of bytes is valid UTF-8, you could use the following "unsafe" methods to not incur the overhead of validation (warnings in the docs): https://doc.rust-lang.org/std/string/struct.String.html#method.from_utf8_unchecked https://doc.rust-lang.org/std/string/struct.String.html#meth... https://doc.rust-lang.org/std/str/fn.from_utf8_unchecked.html https://doc.rust-lang.org/std/str/fn.from_utf8_unchecked.htm...