5 ms·

> representing strings as character slices Zig does not do this. It represents strings as byte slices. A UTF-8 character could be multiple codepoints with each

> representing strings as character slices Zig does not do this. It represents strings as byte slices. A UTF-8 character could be multiple codepoints with each codepoint being multiple bytes. A big thing about Zig is not hiding complexity. If UTF-8 were implemented at a language level (whatever that means), then "language level" string operations would be non-linear, which would be very non-Ziggy. I could see value in a standard library UTF-8 implementation, but a LOT of forethought would need to be put into it. I think keeping UTF-8 string manipulation at the third-party library level is a good choice for now. Maybe once the language is finalized, the ecosystem is more developed, and lessons have been learned from the third-party libraries, then the standard library can implement this.

turminal 5y ago
> character slices Sorry, byte slices is what I had in mind. I'm not talking about language level string operations as in concatenation with + or something like that at all, because that certainly wouldn't make sense in language like Zig. I'm not advocating for string functionality in the language, I'm advocating for a way to not allow byte slice functionality on a thing that is clearly not a byte slice.
- von_lohengramm 5y ago
  > a way to not allow byte slice functionality on a thing that is clearly not a byte slice This already exists in the form of structs or opaque types. Both of these approaches would end up being implemented in "userspace" anyways, whether that's standard library or third-party. However, (UTF-8) strings are byte slices. You can do simple manipulation with them as byte slices safely and validly. Split on spaces? Sure. Tokenize? Sure. Find substring? Sure. You can't do things that depend on say UTF-8 graphemes, but you can safely do most things that depend on bytes. For most purposes, treating strings as byte slices is the safest and correct approach.
  - lmm 5y ago
    Doing find substring by find byte subsequence won't behave correctly in many cases, where semantically equivalent strings have multiple different bytesequence representation. Treating strings as byte slices exposes a lot of footguns; it shouldn't be easy just as e.g. treating floating-point numbers as byte sequences shouldn't be easy.
    - von_lohengramm 5y ago
      Technically the shortest UTF-8 representation is _the_ representation and _correctly_normalized_ Unicode is uniquely represented, but fair enough. The unknown input may be slightly malformed. Complexities like this is why one shouldn't underestimate the nuances (and runtime costs!) of implementing proper Unicode. As for representing byte sequences as byte sequences, that is the most basic way to represent strings of text without placing any assumptions on them. It's the assumption of potentially incorrect invariants that's the issue. If you have faculties to handle Unicode correctly (and very few languages do), then using something more opaque may be better fitting than a byte slice.
      - lmm 5y ago
        > Technically the shortest UTF-8 representation is _the_ representation and _correctly_normalized_ Unicode is uniquely represented Not necessarily the shortest (NFC means not using composed characters from later revisions of the standard), and you only get a normalised representation if you've actually normalised it - if you've just accepted and maybe validated some UTF-8 from outside then it probably won't be in normalized form. IMO it's worth having separate types for unicode strings and normalized unicode strings, and maybe the latter should expose more of the codepoint sequence representation, but I don't know if any language implements that.
    - turminal 5y ago
      > it shouldn't be easy just as e.g. treating floating-point numbers as byte sequences shouldn't be easy. That's a nice analogy. > Doing find substring by find byte subsequence won't behave correctly in many cases, where semantically equivalent strings have multiple different bytesequence representation. Unfortunately that's nearly impossible to do sanely in the general case, no matter how the string is represented.
  - turminal 5y ago
    You are talking about what stringy things can be done with byte slices and I'm talking about all the byteslicy things that shouldn't be done with strings. Like subslicing. And accessing individual bytes in it.
coldtea 5y ago
>Zig does not do this. It represents strings as byte slices. That's what the parent meant. Char(aracter) is a byte in C.

turminal 5y ago

> character slices Sorry, byte slices is what I had in mind. I'm not talking about language level string operations as in concatenation with + or something like that at all, because that certainly wouldn't make sense in language like Zig. I'm not advocating for string functionality in the language, I'm advocating for a way to not allow byte slice functionality on a thing that is clearly not a byte slice.

von_lohengramm 5y ago
> a way to not allow byte slice functionality on a thing that is clearly not a byte slice This already exists in the form of structs or opaque types. Both of these approaches would end up being implemented in "userspace" anyways, whether that's standard library or third-party. However, (UTF-8) strings are byte slices. You can do simple manipulation with them as byte slices safely and validly. Split on spaces? Sure. Tokenize? Sure. Find substring? Sure. You can't do things that depend on say UTF-8 graphemes, but you can safely do most things that depend on bytes. For most purposes, treating strings as byte slices is the safest and correct approach.
- lmm 5y ago
  Doing find substring by find byte subsequence won't behave correctly in many cases, where semantically equivalent strings have multiple different bytesequence representation. Treating strings as byte slices exposes a lot of footguns; it shouldn't be easy just as e.g. treating floating-point numbers as byte sequences shouldn't be easy.
  - von_lohengramm 5y ago
    Technically the shortest UTF-8 representation is _the_ representation and _correctly_normalized_ Unicode is uniquely represented, but fair enough. The unknown input may be slightly malformed. Complexities like this is why one shouldn't underestimate the nuances (and runtime costs!) of implementing proper Unicode. As for representing byte sequences as byte sequences, that is the most basic way to represent strings of text without placing any assumptions on them. It's the assumption of potentially incorrect invariants that's the issue. If you have faculties to handle Unicode correctly (and very few languages do), then using something more opaque may be better fitting than a byte slice.
    - lmm 5y ago
      > Technically the shortest UTF-8 representation is _the_ representation and _correctly_normalized_ Unicode is uniquely represented Not necessarily the shortest (NFC means not using composed characters from later revisions of the standard), and you only get a normalised representation if you've actually normalised it - if you've just accepted and maybe validated some UTF-8 from outside then it probably won't be in normalized form. IMO it's worth having separate types for unicode strings and normalized unicode strings, and maybe the latter should expose more of the codepoint sequence representation, but I don't know if any language implements that.
  - turminal 5y ago
    > it shouldn't be easy just as e.g. treating floating-point numbers as byte sequences shouldn't be easy. That's a nice analogy. > Doing find substring by find byte subsequence won't behave correctly in many cases, where semantically equivalent strings have multiple different bytesequence representation. Unfortunately that's nearly impossible to do sanely in the general case, no matter how the string is represented.
- turminal 5y ago
  You are talking about what stringy things can be done with byte slices and I'm talking about all the byteslicy things that shouldn't be done with strings. Like subslicing. And accessing individual bytes in it.

coldtea 5y ago

>Zig does not do this. It represents strings as byte slices. That's what the parent meant. Char(aracter) is a byte in C.