4 ms·

Out of curiosity, what are you working on that requires a proper string type (rather than encoded UTF-8 bytes)?

moogly 5y ago
I wouldn't use the word "require", and perhaps you didn't mean it that strongly. Don't get me wrong: I could get by without and achieve what I needed to, but to be honest it didn't feel great. I was doing some pretty gnarly high-level filesystem work (transforming an unstructured tree of files/folders into a more structured tree following certain naming and categorization conventions -- a lot of parsing/building/concatenating paths). I have been working with higher-level languages for the past decade or so, but have had a wanderlust to get away from garbage collected langauges and go back to my roots a bit, so I've been comparing some of the newer options in the "improved C"/"C++ replacement" space. It may very well be that I'm not the target demographic for Zig, and that's totally fair, but to me it felt pretty jarring to go back to working with C-style character (byte) arrays and all the (well-known) issues with which they are fraught. I didn't feel like Zig was an improvement over C in the string handling area (as opposed to all other areas), whereas languages like Go and Rust are. Other than that, I felt Zig was a joy to use.
- nyanpasu64 5y ago
  Keep in mind that filesystem paths aren't strings. On Linux, they are raw bytes without any fixed encoding (but usually UTF-8 on UTF-8-based locales), and on Windows, they are sequences of 16-bit codepoints which are expected to be UTF-16 but not validated. Rust's OsStr is my favorite approach so far. It stores Linux's raw bytes as-is, and stores Windows's possibly-valid UTF-16 as WTF-8. This makes path management "just work", with the ability to operate normally on invalid UTF-8 or UTF-16 paths, and zero-copy conversion from UTF-8/ASCII strings to OsStr (though converting OsStr into UTF-16 requires parsing). (Qt's QString-based file dialogs on Linux fail to convert invalid UTF-8 paths like those in https://github.com/petrosagg/wtfiles https://github.com/petrosagg/wtfiles into QString, causing Qt-based apps to open/save the wrong paths.) However there are difficulties in printing an OsStr. For example, a file dialog that shows filenames as raw bytes can't show non-Latin/Unicode characters in a human-readable form, and a file dialog that shows filenames as Unicode strings can't handle invalid Unicode filenames. GTK3 file dialogs show filenames as Unicode strings, and when encountering files with invalid Unicode names, instead displays "file�name.txt (invalid encoding)". Worse yet, how should a file dialog allow users to rename files? If it's based around byte arrays, the user can't enter Unicode characters directly, and if it's based around Unicode (or a locale-specific text encoding), it can't display existing files with invalid Unicode/etc. in the name (probably not an issue if it allows the user to rename to a valid name), nor allow users to enter invalid Unicode (which is not an issue IMO).
  - dnautics 5y ago
    MacOS filesystem is utf-8 strings with valid Unicode glyphs that are pinned to a certain revision of Unicode.
    - masklinn 5y ago
      > MacOS filesystem is utf-8 strings APFS is utf8, HFS+ is utf16. > with valid Unicode glyphs That doesn’t really mean anything. Apple’s fs do guarantee the paths are correct, as in, valid whatever encoding this has nothing to do with glyphs. APFS also does not perform any normalisation while HFS+ uses a custom variant if NFD. While HFS+’s normalisation has its issues and critics, APFS’s lack of normalisation is probably worse: https://eclecticlight.co/2017/04/06/apfs-is-currently-unusable-with-most-non-english-languages/ https://eclecticlight.co/2017/04/06/apfs-is-currently-unusab...
  - roca 5y ago
    The difficulty of printing OsStr is nothing to do with Rust, it's really just the difficulty of printing Linux file names in any context given you don't know what the non-UTF8 bytes mean.
    - int_19h 5y ago
      It's also a good rationale for why filenames should be valid Unicode - because they're data displayed to the end user.

I wouldn't use the word "require", and perhaps you didn't mean it that strongly. Don't get me wrong: I could get by without and achieve what I needed to, but to be honest it didn't feel great. I was doing some pretty gnarly high-level filesystem work (transforming an unstructured tree of files/folders into a more structured tree following certain naming and categorization conventions -- a lot of parsing/building/concatenating paths). I have been working with higher-level languages for the past decade or so, but have had a wanderlust to get away from garbage collected langauges and go back to my roots a bit, so I've been comparing some of the newer options in the "improved C"/"C++ replacement" space. It may very well be that I'm not the target demographic for Zig, and that's totally fair, but to me it felt pretty jarring to go back to working with C-style character (byte) arrays and all the (well-known) issues with which they are fraught. I didn't feel like Zig was an improvement over C in the string handling area (as opposed to all other areas), whereas languages like Go and Rust are. Other than that, I felt Zig was a joy to use.

nyanpasu64 5y ago
Keep in mind that filesystem paths aren't strings. On Linux, they are raw bytes without any fixed encoding (but usually UTF-8 on UTF-8-based locales), and on Windows, they are sequences of 16-bit codepoints which are expected to be UTF-16 but not validated. Rust's OsStr is my favorite approach so far. It stores Linux's raw bytes as-is, and stores Windows's possibly-valid UTF-16 as WTF-8. This makes path management "just work", with the ability to operate normally on invalid UTF-8 or UTF-16 paths, and zero-copy conversion from UTF-8/ASCII strings to OsStr (though converting OsStr into UTF-16 requires parsing). (Qt's QString-based file dialogs on Linux fail to convert invalid UTF-8 paths like those in https://github.com/petrosagg/wtfiles https://github.com/petrosagg/wtfiles into QString, causing Qt-based apps to open/save the wrong paths.) However there are difficulties in printing an OsStr. For example, a file dialog that shows filenames as raw bytes can't show non-Latin/Unicode characters in a human-readable form, and a file dialog that shows filenames as Unicode strings can't handle invalid Unicode filenames. GTK3 file dialogs show filenames as Unicode strings, and when encountering files with invalid Unicode names, instead displays "file�name.txt (invalid encoding)". Worse yet, how should a file dialog allow users to rename files? If it's based around byte arrays, the user can't enter Unicode characters directly, and if it's based around Unicode (or a locale-specific text encoding), it can't display existing files with invalid Unicode/etc. in the name (probably not an issue if it allows the user to rename to a valid name), nor allow users to enter invalid Unicode (which is not an issue IMO).
- dnautics 5y ago
  MacOS filesystem is utf-8 strings with valid Unicode glyphs that are pinned to a certain revision of Unicode.
  - masklinn 5y ago
    > MacOS filesystem is utf-8 strings APFS is utf8, HFS+ is utf16. > with valid Unicode glyphs That doesn’t really mean anything. Apple’s fs do guarantee the paths are correct, as in, valid whatever encoding this has nothing to do with glyphs. APFS also does not perform any normalisation while HFS+ uses a custom variant if NFD. While HFS+’s normalisation has its issues and critics, APFS’s lack of normalisation is probably worse: https://eclecticlight.co/2017/04/06/apfs-is-currently-unusable-with-most-non-english-languages/ https://eclecticlight.co/2017/04/06/apfs-is-currently-unusab...
- roca 5y ago
  The difficulty of printing OsStr is nothing to do with Rust, it's really just the difficulty of printing Linux file names in any context given you don't know what the non-UTF8 bytes mean.
  - int_19h 5y ago
    It's also a good rationale for why filenames should be valid Unicode - because they're data displayed to the end user.