4 ms·

I've been toying with Zig over the summer, and I'm really impressed by the build system, general ergonomics and the explicitness of the language. The only thin

by moogly 5y ago

I've been toying with Zig over the summer, and I'm really impressed by the build system, general ergonomics and the explicitness of the language. The only thing I've found being a pain point is C's biggest pain point: the lack of a proper string type. Even though I've worked for years in the past with C and C++, I still get tripped up, and in Zig I keep on being confused whether I should use raw byte arrays, or sentinel-terminated arrays for certain string manipulations. Maybe I'm not familiar with common idioms in Zig that might help avoid this C-style tracking of string/buffer lengths malarkey that I tend to end up with where a slice is not appropriate. Maybe a dynamic-length string library is the way forward. I've always tried as much as possible to treat strings as just opaque data and never look into them, which tends to work well, but in some domains you really need to look at and massage the characters/codepoints/grapheme clusters, and the lack of a first-citizen UTF-8-aware string type is, I think, a bit unfortunate in this day and age. I understand having one of those could make C interop a bit gnarlier (I think Odin's approach of having two separate string types -- one of which is just meant to be used for interop -- should be workable). Oh, and the ecosystem of libraries is still young. I needed to format timestamps as human readable date/time to a file, and had to step down to C to do something that basic[1], but it'll get there in time, I'm sure. [1]: "basic" from the perspective of the end-user of the language. Having written a (not even fully featured) date/time library in C++, I consider date and time to be one of the hardest domains I've ever had to work in.

10000truths 5y ago
> I've always tried as much as possible to treat strings as just opaque data and never look into them, which tends to work well, but in some domains you really need to look at and massage the characters/codepoints/grapheme clusters, and the lack of a first-citizen UTF-8-aware string type is, I think, a bit unfortunate in this day and age. You don't need a string type for that, you just need routines that handle UTF-8 strings, like utfcpp (https://github.com/nemtrif/utfcpp https://github.com/nemtrif/utfcpp).
- moogly 5y ago
  Sure, you're not wrong. For my purposes that'd work, but I'm of the opinion that delegating that to an optional library will make a lot of developers lazy and not even bother thinking about Unicode/i18n/l10n issues at all. I've seen so much code, in a multitude of languages that is oblivious to it, but even if it's reified in a type or even having some form of language support, I guess you can argue that people can still ignore it.
  - mikepurvis 5y ago
    That was basically the state of affairs with strings in Python 2, and although the transition was awful, I'm really, really appreciative of Python 3 now having clearly-separated str and bytes types.
  - moogly 5y ago
    In addition, I should add there are synergy effects in having the community (and the major libraries) settling on a common way, and you'll only really get that if there is a built-in. However, if the philosophy here is to be unopinionated and optimize for flexibility over everything else, I feel I can't really fault them for making that choice.
- kristoff_it 5y ago
  Agreed, and Zig also has a lib for that as well: https://github.com/jecolon/ziglyph/ https://github.com/jecolon/ziglyph/
  - moogly 5y ago
    Ooh, haven't come across this. Thanks!
  - coldtea 5y ago
    If it's not a single official library, then this will just end up in a future mess... Like in any language with any lib that is not self-contained (cuts across all aspects of a program and can be used anywhere) and has many different implementations and no sanctioned one.
losvedir 5y ago
Note that while strings are generally opaque bytes in Zig, there are standard library functions[0] to work with them as Unicode codepoints and such. But Unicode is quite large and complicated, and things like grapheme clusters aren't in the standard library (yet?). I also believe that that module is planned to be rewritten. So it's more the case that Zig plans to someday support Unicode at the standard library level, but not as a specific type. Then again, I'm not sure exactly what a type would be for Unicode. Basically just a flag that the array of bytes has been checked and is valid? It's nice to let the type system help you enforce your application boundaries, but I think you could wrap it pretty easily in a struct of your own if you needed that. [0] https://github.com/ziglang/zig/blob/master/lib/std/unicode.zig https://github.com/ziglang/zig/blob/master/lib/std/unicode.z...
- moogly 5y ago
  Looks like this is an attempt: https://github.com/jecolon/zigstr https://github.com/jecolon/zigstr
- tialaramex 5y ago
  > Basically just a flag that the array of bytes has been checked and is valid? In something like Zig I think it's OK that it's merely a stated assumption that these bytes are UTF-8, not actually checked - so long as people take that seriously. It's nice in code that maybe isn't very concerned with such things to be able to know that any "string" is actually text we can display, output to a console, something like that. Not for example a TCP/IP packet we haven't decoded yet, or the first 32 bytes of a JPEG image. Programmers who aren't writing firmware for a vacuum cleaner or washing machine, probably want string literals like "this" in their code, and you'd naturally want those to have a type, for which a string type is the obvious fit. I believe in 2021 this type should obviously be UTF-8. "It's just some bytes" is far from useless, but it isn't much of a string type. I was sceptical at first about Rust's choice to build in str (immutable UTF-8 string slices), because that seems like a relatively high level concept. But unlike std::string::String, str isn't that tricky after all. See, the only place such things would come from (not having std::string::String to make new ones) is the program source, and our compiler is necessarily already reading the program source, so it follows that the compiler must know how the source is encoded etc and thus it does actually know exactly what those literal strings are. If your literal wasn't Unicode, that wasn't a Rust program and it won't compile.
AndyKelley 5y ago
Out of curiosity, what are you working on that requires a proper string type (rather than encoded UTF-8 bytes)?
- moogly 5y ago
  I wouldn't use the word "require", and perhaps you didn't mean it that strongly. Don't get me wrong: I could get by without and achieve what I needed to, but to be honest it didn't feel great. I was doing some pretty gnarly high-level filesystem work (transforming an unstructured tree of files/folders into a more structured tree following certain naming and categorization conventions -- a lot of parsing/building/concatenating paths). I have been working with higher-level languages for the past decade or so, but have had a wanderlust to get away from garbage collected langauges and go back to my roots a bit, so I've been comparing some of the newer options in the "improved C"/"C++ replacement" space. It may very well be that I'm not the target demographic for Zig, and that's totally fair, but to me it felt pretty jarring to go back to working with C-style character (byte) arrays and all the (well-known) issues with which they are fraught. I didn't feel like Zig was an improvement over C in the string handling area (as opposed to all other areas), whereas languages like Go and Rust are. Other than that, I felt Zig was a joy to use.
  - nyanpasu64 5y ago
    Keep in mind that filesystem paths aren't strings. On Linux, they are raw bytes without any fixed encoding (but usually UTF-8 on UTF-8-based locales), and on Windows, they are sequences of 16-bit codepoints which are expected to be UTF-16 but not validated. Rust's OsStr is my favorite approach so far. It stores Linux's raw bytes as-is, and stores Windows's possibly-valid UTF-16 as WTF-8. This makes path management "just work", with the ability to operate normally on invalid UTF-8 or UTF-16 paths, and zero-copy conversion from UTF-8/ASCII strings to OsStr (though converting OsStr into UTF-16 requires parsing). (Qt's QString-based file dialogs on Linux fail to convert invalid UTF-8 paths like those in https://github.com/petrosagg/wtfiles https://github.com/petrosagg/wtfiles into QString, causing Qt-based apps to open/save the wrong paths.) However there are difficulties in printing an OsStr. For example, a file dialog that shows filenames as raw bytes can't show non-Latin/Unicode characters in a human-readable form, and a file dialog that shows filenames as Unicode strings can't handle invalid Unicode filenames. GTK3 file dialogs show filenames as Unicode strings, and when encountering files with invalid Unicode names, instead displays "file�name.txt (invalid encoding)". Worse yet, how should a file dialog allow users to rename files? If it's based around byte arrays, the user can't enter Unicode characters directly, and if it's based around Unicode (or a locale-specific text encoding), it can't display existing files with invalid Unicode/etc. in the name (probably not an issue if it allows the user to rename to a valid name), nor allow users to enter invalid Unicode (which is not an issue IMO).
Lvl999Noob 5y ago
For strings, I believe Rust's approach of a dynamic `String` and a view `str` is the right approach. For a minimal std, maybe Zig can have a standard `str` type which is used for all functions that don't need to grow the string and the `String` type is provided by any 3rd party library which is expected to provide a conversion function from `String` to `str`. These strings types could or could not have a requirement of being Utf-8. Maybe there is a builtin wrapper to convert them. So you can any dynamic byte array which can be converted to `str` (just a pointer and length, no validation) which can be later converted to `utf8` (str but validated). A `String` which guarantees Utf-8 could directly convert to `utf8` to save validation costs.

by moogly 5y ago

10000truths 5y ago

> I've always tried as much as possible to treat strings as just opaque data and never look into them, which tends to work well, but in some domains you really need to look at and massage the characters/codepoints/grapheme clusters, and the lack of a first-citizen UTF-8-aware string type is, I think, a bit unfortunate in this day and age. You don't need a string type for that, you just need routines that handle UTF-8 strings, like utfcpp (https://github.com/nemtrif/utfcpp https://github.com/nemtrif/utfcpp).

moogly 5y ago
Sure, you're not wrong. For my purposes that'd work, but I'm of the opinion that delegating that to an optional library will make a lot of developers lazy and not even bother thinking about Unicode/i18n/l10n issues at all. I've seen so much code, in a multitude of languages that is oblivious to it, but even if it's reified in a type or even having some form of language support, I guess you can argue that people can still ignore it.
- mikepurvis 5y ago
  That was basically the state of affairs with strings in Python 2, and although the transition was awful, I'm really, really appreciative of Python 3 now having clearly-separated str and bytes types.
- moogly 5y ago
  In addition, I should add there are synergy effects in having the community (and the major libraries) settling on a common way, and you'll only really get that if there is a built-in. However, if the philosophy here is to be unopinionated and optimize for flexibility over everything else, I feel I can't really fault them for making that choice.
kristoff_it 5y ago
Agreed, and Zig also has a lib for that as well: https://github.com/jecolon/ziglyph/ https://github.com/jecolon/ziglyph/
- moogly 5y ago
  Ooh, haven't come across this. Thanks!
- coldtea 5y ago
  If it's not a single official library, then this will just end up in a future mess... Like in any language with any lib that is not self-contained (cuts across all aspects of a program and can be used anywhere) and has many different implementations and no sanctioned one.

losvedir 5y ago

Note that while strings are generally opaque bytes in Zig, there are standard library functions[0] to work with them as Unicode codepoints and such. But Unicode is quite large and complicated, and things like grapheme clusters aren't in the standard library (yet?). I also believe that that module is planned to be rewritten. So it's more the case that Zig plans to someday support Unicode at the standard library level, but not as a specific type. Then again, I'm not sure exactly what a type would be for Unicode. Basically just a flag that the array of bytes has been checked and is valid? It's nice to let the type system help you enforce your application boundaries, but I think you could wrap it pretty easily in a struct of your own if you needed that. [0] https://github.com/ziglang/zig/blob/master/lib/std/unicode.zig https://github.com/ziglang/zig/blob/master/lib/std/unicode.z...

moogly 5y ago
Looks like this is an attempt: https://github.com/jecolon/zigstr https://github.com/jecolon/zigstr
tialaramex 5y ago
> Basically just a flag that the array of bytes has been checked and is valid? In something like Zig I think it's OK that it's merely a stated assumption that these bytes are UTF-8, not actually checked - so long as people take that seriously. It's nice in code that maybe isn't very concerned with such things to be able to know that any "string" is actually text we can display, output to a console, something like that. Not for example a TCP/IP packet we haven't decoded yet, or the first 32 bytes of a JPEG image. Programmers who aren't writing firmware for a vacuum cleaner or washing machine, probably want string literals like "this" in their code, and you'd naturally want those to have a type, for which a string type is the obvious fit. I believe in 2021 this type should obviously be UTF-8. "It's just some bytes" is far from useless, but it isn't much of a string type. I was sceptical at first about Rust's choice to build in str (immutable UTF-8 string slices), because that seems like a relatively high level concept. But unlike std::string::String, str isn't that tricky after all. See, the only place such things would come from (not having std::string::String to make new ones) is the program source, and our compiler is necessarily already reading the program source, so it follows that the compiler must know how the source is encoded etc and thus it does actually know exactly what those literal strings are. If your literal wasn't Unicode, that wasn't a Rust program and it won't compile.

AndyKelley 5y ago

Out of curiosity, what are you working on that requires a proper string type (rather than encoded UTF-8 bytes)?

moogly 5y ago
I wouldn't use the word "require", and perhaps you didn't mean it that strongly. Don't get me wrong: I could get by without and achieve what I needed to, but to be honest it didn't feel great. I was doing some pretty gnarly high-level filesystem work (transforming an unstructured tree of files/folders into a more structured tree following certain naming and categorization conventions -- a lot of parsing/building/concatenating paths). I have been working with higher-level languages for the past decade or so, but have had a wanderlust to get away from garbage collected langauges and go back to my roots a bit, so I've been comparing some of the newer options in the "improved C"/"C++ replacement" space. It may very well be that I'm not the target demographic for Zig, and that's totally fair, but to me it felt pretty jarring to go back to working with C-style character (byte) arrays and all the (well-known) issues with which they are fraught. I didn't feel like Zig was an improvement over C in the string handling area (as opposed to all other areas), whereas languages like Go and Rust are. Other than that, I felt Zig was a joy to use.
- nyanpasu64 5y ago
  Keep in mind that filesystem paths aren't strings. On Linux, they are raw bytes without any fixed encoding (but usually UTF-8 on UTF-8-based locales), and on Windows, they are sequences of 16-bit codepoints which are expected to be UTF-16 but not validated. Rust's OsStr is my favorite approach so far. It stores Linux's raw bytes as-is, and stores Windows's possibly-valid UTF-16 as WTF-8. This makes path management "just work", with the ability to operate normally on invalid UTF-8 or UTF-16 paths, and zero-copy conversion from UTF-8/ASCII strings to OsStr (though converting OsStr into UTF-16 requires parsing). (Qt's QString-based file dialogs on Linux fail to convert invalid UTF-8 paths like those in https://github.com/petrosagg/wtfiles https://github.com/petrosagg/wtfiles into QString, causing Qt-based apps to open/save the wrong paths.) However there are difficulties in printing an OsStr. For example, a file dialog that shows filenames as raw bytes can't show non-Latin/Unicode characters in a human-readable form, and a file dialog that shows filenames as Unicode strings can't handle invalid Unicode filenames. GTK3 file dialogs show filenames as Unicode strings, and when encountering files with invalid Unicode names, instead displays "file�name.txt (invalid encoding)". Worse yet, how should a file dialog allow users to rename files? If it's based around byte arrays, the user can't enter Unicode characters directly, and if it's based around Unicode (or a locale-specific text encoding), it can't display existing files with invalid Unicode/etc. in the name (probably not an issue if it allows the user to rename to a valid name), nor allow users to enter invalid Unicode (which is not an issue IMO).

Lvl999Noob 5y ago

For strings, I believe Rust's approach of a dynamic `String` and a view `str` is the right approach. For a minimal std, maybe Zig can have a standard `str` type which is used for all functions that don't need to grow the string and the `String` type is provided by any 3rd party library which is expected to provide a conversion function from `String` to `str`. These strings types could or could not have a requirement of being Utf-8. Maybe there is a builtin wrapper to convert them. So you can any dynamic byte array which can be converted to `str` (just a pointer and length, no validation) which can be later converted to `utf8` (str but validated). A `String` which guarantees Utf-8 could directly convert to `utf8` to save validation costs.