4 ms·

For what it’s worth this makes the same mistake that Python 2 did: string and bytes are not the same type and shouldn’t be treated as such.

School-Cotton 2y ago
What is your definition of "string"? If it's "human-readable text", then fine, a string is not the same thing as an arbitrary byte array. But lots of languages don't enforce that definition.
- bobbylarrybobby 2y ago
  Well that's the very thing: not enforcing that distinction is the very mistake in question.
michaelsbradley 2y ago
Given the nature of it (pretty.c) and the stated intention of being "backwards-compatible with C and all of its libraries", what would make more sense than sticking with C's multibyte strings? https://en.cppreference.com/w/c/string/multibyte https://en.cppreference.com/w/c/string/multibyte
- IgorPartola 2y ago
  Right but pretty.c doesn’t seem to explicitly support those.
  - michaelsbradley 2y ago
    How so? it’s just char*
    - IgorPartola 2y ago
      What if len() of a char* vs a Unicode string?
      - michaelsbradley 2y ago
        char* is just raw bytes. At the language level C historically hasn't offered much support for working with specific character sets and their encodings. With C17 and C23 we get u"...", U"...", u8"...", type char8_t, and similar, but there's still little/no built-in tooling for text processing. For text processing work with char* whose bytes are some encoding/s of Unicode, e.g. UTF-8, then you an use a C library such as libunistring or ICU. However the bytes of a char* could instead be an encoding of a non-Unicode character set, e.g. GB2312 encoded as EUC-CN. So char* is character set and encoding agnostic. And C-the-language doesn't even try to offer you tools for working with different sets and encodings. Instead, you can use a library or write your own code for that purpose. A number of languages make the same decision, keeping the string type set/encoding agnostic, with libraries taking up the slack. In Nim, for example, the string type is essentially raw bytes (string literals in .nim sources are UTF-8). If you're doing Unicode text processing then you'd use facilities from the std/unicode module https://nim-lang.org/docs/unicode.html https://nim-lang.org/docs/unicode.html Same story with Zig https://ziglang.org/documentation/0.8.0/std/#std;unicode https://ziglang.org/documentation/0.8.0/std/#std;unicode Lua too, and you'll probably use a 3rd party library such as luaut8 for working with Unicode/UTF-8 https://github.com/starwing/luautf8 https://github.com/starwing/luautf8 Returning to the matter of pretty.c, since it's just sugar for C, it makes sense (to me) that the string type is just an alias for the set/encoding agnostic char*. It's up to the programmer to know and decide what the bytes represent and choose a library accordingly.
AlotOfReading 2y ago
What do you consider the type of shell text, i.e. what's in argv and what you get from subprocess output? It's not well-formed utf8 strings because any random garbage can be in there, yet tools like awk and grep are ubiquitous. I'd argue that strings and bytes are the same general type, but it's sometimes useful to give well-formed utf8 bytes a different type internally. Rust gets this mostly correct with OsString and String.
- ryandrake 2y ago
  The way I understand it: Bytes are just bytes, until you provide an encoding. Then they can be can be converted to a string, if validly encoded. Taking an array of characters and just treating it or casting it as a string is usually a bad idea. The thing I think Rust maybe goofed, or at least made a little complicated, is their weird distinction between a String and a str (and a &str). As a newbie learning the language, I have no idea which one to use, and usually just pick one, try to compile, then if it fails, pick the other one. I'm sure there was a great reason to have two types for the same thing, that I will understand when I know the language better.
  - steveklabnik 2y ago
    I wrote a blog post that may help you! https://steveklabnik.com/writing/when-should-i-use-string-vs-str/ https://steveklabnik.com/writing/when-should-i-use-string-vs... If you want to understand more deeply, the Rust Programming Langauge, chapter 4, uses String and &String and &str to talk about ownership and borrowing. Here’s a link to the start of that chapter: https://doc.rust-lang.org/stable/book/ch04-00-understanding-ownership.html https://doc.rust-lang.org/stable/book/ch04-00-understanding-...
    - ryandrake 2y ago
      How timely and helpful, thanks! Your blog post is practical and clearly explains what to do, when, which is helpful. What's confusing is why Rust has the two types and why the language designers decided it was a good idea to have to convert back and forth between them depending on whether it was going in a struct or being passed as an argument. I suppose the "why" is probably better found in the Rust docs. As a long-time C++ user, it seems like std::string vs const char* all over again, and we somehow didn't find a better way.
samatman 2y ago
I don't agree. This doctrine presumes all of the following: - String data will be properly encoded - There is one encoding of strings (UTF-8 usually) - Validation must occur when string data is created - Truncating a logical codepoint is never acceptable - You may not do string things to "invalid" bytes - Proper encoding is the beginning and the end of validation None of these things are consistently true. It's a useful practice to wrap validated byte sequences in a type which can only be created by validation, and once you're doing that, `Utf8String` and `EmailAddress` are basically the same thing, there's no reason to privilege the encoding in the type system.
- IgorPartola 2y ago
  I mean other languages make it work.

School-Cotton 2y ago

What is your definition of "string"? If it's "human-readable text", then fine, a string is not the same thing as an arbitrary byte array. But lots of languages don't enforce that definition.

bobbylarrybobby 2y ago
Well that's the very thing: not enforcing that distinction is the very mistake in question.

michaelsbradley 2y ago

Given the nature of it (pretty.c) and the stated intention of being "backwards-compatible with C and all of its libraries", what would make more sense than sticking with C's multibyte strings? https://en.cppreference.com/w/c/string/multibyte https://en.cppreference.com/w/c/string/multibyte

IgorPartola 2y ago
Right but pretty.c doesn’t seem to explicitly support those.
- michaelsbradley 2y ago
  How so? it’s just char*
  - IgorPartola 2y ago
    What if len() of a char* vs a Unicode string?
    - michaelsbradley 2y ago
      char* is just raw bytes. At the language level C historically hasn't offered much support for working with specific character sets and their encodings. With C17 and C23 we get u"...", U"...", u8"...", type char8_t, and similar, but there's still little/no built-in tooling for text processing. For text processing work with char* whose bytes are some encoding/s of Unicode, e.g. UTF-8, then you an use a C library such as libunistring or ICU. However the bytes of a char* could instead be an encoding of a non-Unicode character set, e.g. GB2312 encoded as EUC-CN. So char* is character set and encoding agnostic. And C-the-language doesn't even try to offer you tools for working with different sets and encodings. Instead, you can use a library or write your own code for that purpose. A number of languages make the same decision, keeping the string type set/encoding agnostic, with libraries taking up the slack. In Nim, for example, the string type is essentially raw bytes (string literals in .nim sources are UTF-8). If you're doing Unicode text processing then you'd use facilities from the std/unicode module https://nim-lang.org/docs/unicode.html https://nim-lang.org/docs/unicode.html Same story with Zig https://ziglang.org/documentation/0.8.0/std/#std;unicode https://ziglang.org/documentation/0.8.0/std/#std;unicode Lua too, and you'll probably use a 3rd party library such as luaut8 for working with Unicode/UTF-8 https://github.com/starwing/luautf8 https://github.com/starwing/luautf8 Returning to the matter of pretty.c, since it's just sugar for C, it makes sense (to me) that the string type is just an alias for the set/encoding agnostic char*. It's up to the programmer to know and decide what the bytes represent and choose a library accordingly.

AlotOfReading 2y ago

What do you consider the type of shell text, i.e. what's in argv and what you get from subprocess output? It's not well-formed utf8 strings because any random garbage can be in there, yet tools like awk and grep are ubiquitous. I'd argue that strings and bytes are the same general type, but it's sometimes useful to give well-formed utf8 bytes a different type internally. Rust gets this mostly correct with OsString and String.

ryandrake 2y ago
The way I understand it: Bytes are just bytes, until you provide an encoding. Then they can be can be converted to a string, if validly encoded. Taking an array of characters and just treating it or casting it as a string is usually a bad idea. The thing I think Rust maybe goofed, or at least made a little complicated, is their weird distinction between a String and a str (and a &str). As a newbie learning the language, I have no idea which one to use, and usually just pick one, try to compile, then if it fails, pick the other one. I'm sure there was a great reason to have two types for the same thing, that I will understand when I know the language better.
- steveklabnik 2y ago
  I wrote a blog post that may help you! https://steveklabnik.com/writing/when-should-i-use-string-vs-str/ https://steveklabnik.com/writing/when-should-i-use-string-vs... If you want to understand more deeply, the Rust Programming Langauge, chapter 4, uses String and &String and &str to talk about ownership and borrowing. Here’s a link to the start of that chapter: https://doc.rust-lang.org/stable/book/ch04-00-understanding-ownership.html https://doc.rust-lang.org/stable/book/ch04-00-understanding-...
  - ryandrake 2y ago
    How timely and helpful, thanks! Your blog post is practical and clearly explains what to do, when, which is helpful. What's confusing is why Rust has the two types and why the language designers decided it was a good idea to have to convert back and forth between them depending on whether it was going in a struct or being passed as an argument. I suppose the "why" is probably better found in the Rust docs. As a long-time C++ user, it seems like std::string vs const char* all over again, and we somehow didn't find a better way.

samatman 2y ago

I don't agree. This doctrine presumes all of the following: - String data will be properly encoded - There is one encoding of strings (UTF-8 usually) - Validation must occur when string data is created - Truncating a logical codepoint is never acceptable - You may not do string things to "invalid" bytes - Proper encoding is the beginning and the end of validation None of these things are consistently true. It's a useful practice to wrap validated byte sequences in a type which can only be created by validation, and once you're doing that, `Utf8String` and `EmailAddress` are basically the same thing, there's no reason to privilege the encoding in the type system.

IgorPartola 2y ago
I mean other languages make it work.