3 ms·

Note that while strings are generally opaque bytes in Zig, there are standard library functions[0] to work with them as Unicode codepoints and such. But Unicode

Note that while strings are generally opaque bytes in Zig, there are standard library functions[0] to work with them as Unicode codepoints and such. But Unicode is quite large and complicated, and things like grapheme clusters aren't in the standard library (yet?). I also believe that that module is planned to be rewritten. So it's more the case that Zig plans to someday support Unicode at the standard library level, but not as a specific type. Then again, I'm not sure exactly what a type would be for Unicode. Basically just a flag that the array of bytes has been checked and is valid? It's nice to let the type system help you enforce your application boundaries, but I think you could wrap it pretty easily in a struct of your own if you needed that. [0] https://github.com/ziglang/zig/blob/master/lib/std/unicode.zig https://github.com/ziglang/zig/blob/master/lib/std/unicode.z...

moogly 5y ago
Looks like this is an attempt: https://github.com/jecolon/zigstr https://github.com/jecolon/zigstr
tialaramex 5y ago
> Basically just a flag that the array of bytes has been checked and is valid? In something like Zig I think it's OK that it's merely a stated assumption that these bytes are UTF-8, not actually checked - so long as people take that seriously. It's nice in code that maybe isn't very concerned with such things to be able to know that any "string" is actually text we can display, output to a console, something like that. Not for example a TCP/IP packet we haven't decoded yet, or the first 32 bytes of a JPEG image. Programmers who aren't writing firmware for a vacuum cleaner or washing machine, probably want string literals like "this" in their code, and you'd naturally want those to have a type, for which a string type is the obvious fit. I believe in 2021 this type should obviously be UTF-8. "It's just some bytes" is far from useless, but it isn't much of a string type. I was sceptical at first about Rust's choice to build in str (immutable UTF-8 string slices), because that seems like a relatively high level concept. But unlike std::string::String, str isn't that tricky after all. See, the only place such things would come from (not having std::string::String to make new ones) is the program source, and our compiler is necessarily already reading the program source, so it follows that the compiler must know how the source is encoded etc and thus it does actually know exactly what those literal strings are. If your literal wasn't Unicode, that wasn't a Rust program and it won't compile.
- coldtea 5y ago
  >In something like Zig I think it's OK that it's merely a stated assumption that these bytes are UTF-8, not actually checked - so long as people take that seriously. And also as long as all supported operations on those bytes still result in utf-8 strings (there could be unsafe operations too, like splitting at a random byte offset, but there should be a clearly marked safe set that wont corrupt a valid utf-8 byte sequence).
mamcx 5y ago
> Then again, I'm not sure exactly what a type would be for Unicode. I have this internally in my lang AST Rust Utf8 I think that is the type: struct City { name:Utf8 } For a scripting lang like mine I surface it as `Str` but in a system language I think is better to say `Utf8`
coldtea 5y ago
>*It's nice to let the type system help you enforce your application boundaries, but I think you could wrap it pretty easily in a struct of your own if you needed that. Well, as you said "Unicode is quite large and complicated". And in 2021 it's not just anglosaxon software users anymore (if they ever were), good unicode lib if not type is a must.

moogly 5y ago

Looks like this is an attempt: https://github.com/jecolon/zigstr https://github.com/jecolon/zigstr

tialaramex 5y ago

> Basically just a flag that the array of bytes has been checked and is valid? In something like Zig I think it's OK that it's merely a stated assumption that these bytes are UTF-8, not actually checked - so long as people take that seriously. It's nice in code that maybe isn't very concerned with such things to be able to know that any "string" is actually text we can display, output to a console, something like that. Not for example a TCP/IP packet we haven't decoded yet, or the first 32 bytes of a JPEG image. Programmers who aren't writing firmware for a vacuum cleaner or washing machine, probably want string literals like "this" in their code, and you'd naturally want those to have a type, for which a string type is the obvious fit. I believe in 2021 this type should obviously be UTF-8. "It's just some bytes" is far from useless, but it isn't much of a string type. I was sceptical at first about Rust's choice to build in str (immutable UTF-8 string slices), because that seems like a relatively high level concept. But unlike std::string::String, str isn't that tricky after all. See, the only place such things would come from (not having std::string::String to make new ones) is the program source, and our compiler is necessarily already reading the program source, so it follows that the compiler must know how the source is encoded etc and thus it does actually know exactly what those literal strings are. If your literal wasn't Unicode, that wasn't a Rust program and it won't compile.

coldtea 5y ago
>In something like Zig I think it's OK that it's merely a stated assumption that these bytes are UTF-8, not actually checked - so long as people take that seriously. And also as long as all supported operations on those bytes still result in utf-8 strings (there could be unsafe operations too, like splitting at a random byte offset, but there should be a clearly marked safe set that wont corrupt a valid utf-8 byte sequence).

mamcx 5y ago

> Then again, I'm not sure exactly what a type would be for Unicode. I have this internally in my lang AST Rust Utf8 I think that is the type: struct City { name:Utf8 } For a scripting lang like mine I surface it as `Str` but in a system language I think is better to say `Utf8`

coldtea 5y ago

>*It's nice to let the type system help you enforce your application boundaries, but I think you could wrap it pretty easily in a struct of your own if you needed that. Well, as you said "Unicode is quite large and complicated". And in 2021 it's not just anglosaxon software users anymore (if they ever were), good unicode lib if not type is a must.