6 ms·

> Was there something in particular you were looking for and didn't find? Unicode handling. Treating a string as a byte array is all fine and dandy if you're o

by shakow 5y ago

> Was there something in particular you were looking for and didn't find? Unicode handling. Treating a string as a byte array is all fine and dandy if you're only processing english latin alphabet data, but it's a PITA as soon as you start using e.g. extended characters (math symbols, fancy quotes, ...) other languages, or emojis.

llimllib 5y ago
There is the `std.unicode` module[1] which provides standard unicode functions (encode, decode, length, iteration over code points), so I don't think it's fair to say that the language's library lacks strings in any real sense. I will re-emphasize that I've not used it, so I cannot speak for its quality. [1]: https://github.com/ziglang/zig/blob/master/lib/std/unicode.zig https://github.com/ziglang/zig/blob/master/lib/std/unicode.z...
chubot 5y ago
What do you need to do with them? All my data is UTF-8, and low level code is generally parsing, which doesn't involve any of those characters. It generally just works with all special characters (e.g. on my blog). I think Unicode on the server or CLI is very different than Unicode on the desktop/GUIs. Since Zig interfaces well with C, it should be set up well for the harder desktop case, because the "real" Unicode libraries handling all the corner cases are written in C (or have C interfaces). I don't think even Python's unicode support is up to the task for desktop apps.
- dmitriid 5y ago
  > What do you need to do with them? All my data is UTF-8, and low level code is generally parsing, which doesn't involve any of those characters. For example, file names on MacOS are Unicode. Depending on how low level your code is and what parsing it does, you will run into issues because UTF-8 will not save you. > I think Unicode on the server or CLI is very different than Unicode on the desktop/GUIs. This is a non-sensical statement. It's the same Unicode. Yes, you won't have all the same use cases as a GUI, but Unicode is the same. The best example I have for when things are not handled is not strictly Unicode-related. WiFi SSIDs are 32 octets. That is any byte value at all. Recent standard modifications allow devices to specify whether the SSID is in UTF-8. And yet. Too many devices assume that, basically, " All my data is UTF-8, and low level code is generally parsing, which doesn't involve any of those characters", and Internet is full of people asking questions like "cannot connect to WiFi with non-English SSID"
  - anonymoushn 5y ago
    Woe be to they who attempt to save Japanese mp3 files on a Linux NAS and read them from a Mac.
    - chubot 5y ago
      Yeah that's true, but this has nothing to do with the programming language, and everything to do with the file system implementation in the kernel, and the OS APIs. The programming language can only do so much about this problem, and doing too much is harmful (e.g. Python 3). The default OS X file system isn't POSIX compliant, which creates a portability issue. In POSIX file systems, filenames are bytes, and file contents are bytes. But that isn't true on OS X.
      - anonymoushn 5y ago
        Agreed all around. "Strings are byte arrays" is imo the right approach to living in a hellworld where programs like filesystems expect you to send them all sorts of different precise arrangements of bytes.
  - chubot 5y ago
    Yeah filenames on OS X are a pain. However, Python 3 has the opposite problem! All the file system APIs were changed to assume that filenames are strings (sequences of code points), when they're actually BYTES in POSIX! (which may or may not be utf-8 encoded, but commonly are) So my point is: when you ask for "unicode" support in a language, you have to be careful what you ask for, or it can create more problems. Having less in the core language is good. For the specific case of filenames, I think you just need basename() and dirname() routines that find '/' (or \) regardless of encoding. You don't need a separate string type. I should have said "localization on the server" vs. "localization on desktop/GUIs". What people really want is localized apps, not just unicode, and having a string type in a language doesn't get you that far. Because localization is inherently tied to the operating system, not the language. The filename example shows that, and there are also several other issues like fonts, left-to-right, etc.
kristoff_it 5y ago
Also here is a blog post that gives more info on how to deal with unicode in Zig. https://zig.news/dude_the_builder/unicode-basics-in-zig-dj3 https://zig.news/dude_the_builder/unicode-basics-in-zig-dj3

by shakow 5y ago

llimllib 5y ago

There is the `std.unicode` module[1] which provides standard unicode functions (encode, decode, length, iteration over code points), so I don't think it's fair to say that the language's library lacks strings in any real sense. I will re-emphasize that I've not used it, so I cannot speak for its quality. [1]: https://github.com/ziglang/zig/blob/master/lib/std/unicode.zig https://github.com/ziglang/zig/blob/master/lib/std/unicode.z...

chubot 5y ago

What do you need to do with them? All my data is UTF-8, and low level code is generally parsing, which doesn't involve any of those characters. It generally just works with all special characters (e.g. on my blog). I think Unicode on the server or CLI is very different than Unicode on the desktop/GUIs. Since Zig interfaces well with C, it should be set up well for the harder desktop case, because the "real" Unicode libraries handling all the corner cases are written in C (or have C interfaces). I don't think even Python's unicode support is up to the task for desktop apps.

dmitriid 5y ago
> What do you need to do with them? All my data is UTF-8, and low level code is generally parsing, which doesn't involve any of those characters. For example, file names on MacOS are Unicode. Depending on how low level your code is and what parsing it does, you will run into issues because UTF-8 will not save you. > I think Unicode on the server or CLI is very different than Unicode on the desktop/GUIs. This is a non-sensical statement. It's the same Unicode. Yes, you won't have all the same use cases as a GUI, but Unicode is the same. The best example I have for when things are not handled is not strictly Unicode-related. WiFi SSIDs are 32 octets. That is any byte value at all. Recent standard modifications allow devices to specify whether the SSID is in UTF-8. And yet. Too many devices assume that, basically, " All my data is UTF-8, and low level code is generally parsing, which doesn't involve any of those characters", and Internet is full of people asking questions like "cannot connect to WiFi with non-English SSID"
- anonymoushn 5y ago
  Woe be to they who attempt to save Japanese mp3 files on a Linux NAS and read them from a Mac.
  - chubot 5y ago
    Yeah that's true, but this has nothing to do with the programming language, and everything to do with the file system implementation in the kernel, and the OS APIs. The programming language can only do so much about this problem, and doing too much is harmful (e.g. Python 3). The default OS X file system isn't POSIX compliant, which creates a portability issue. In POSIX file systems, filenames are bytes, and file contents are bytes. But that isn't true on OS X.
    - anonymoushn 5y ago
      Agreed all around. "Strings are byte arrays" is imo the right approach to living in a hellworld where programs like filesystems expect you to send them all sorts of different precise arrangements of bytes.
- chubot 5y ago
  Yeah filenames on OS X are a pain. However, Python 3 has the opposite problem! All the file system APIs were changed to assume that filenames are strings (sequences of code points), when they're actually BYTES in POSIX! (which may or may not be utf-8 encoded, but commonly are) So my point is: when you ask for "unicode" support in a language, you have to be careful what you ask for, or it can create more problems. Having less in the core language is good. For the specific case of filenames, I think you just need basename() and dirname() routines that find '/' (or \) regardless of encoding. You don't need a separate string type. I should have said "localization on the server" vs. "localization on desktop/GUIs". What people really want is localized apps, not just unicode, and having a string type in a language doesn't get you that far. Because localization is inherently tied to the operating system, not the language. The filename example shows that, and there are also several other issues like fonts, left-to-right, etc.

kristoff_it 5y ago

Also here is a blog post that gives more info on how to deal with unicode in Zig. https://zig.news/dude_the_builder/unicode-basics-in-zig-dj3 https://zig.news/dude_the_builder/unicode-basics-in-zig-dj3