12 ms·

I don't know if string handling has been improved, but it's one of the few things stopping me from using Zig. I look forward to 1.0, which will hopefully have p

• flohofwoe 5y ago
  IMHO Zig's builtin string handling (or rather, lack of) is exactly right for a systems programming language. Zig avoids the biggest problem of C strings and treats strings as ptr/length slices, not as zero-terminated. UTF-8 for string literals is also fine. That's all that's needed (and should be implemented) on the language level, everything else should go into the standard library, and additional specialized string processing libraries (because string handling can never be a "one-size fits all" solution, it's too complex for that).
  • turminal 5y ago
    The issue with treating strings as slices/arrays is that it exposes (writably!) the underlying representation, which is almost never what you want and leads to subtle bugs. The downside is of course the fact that you have to account for encodings in your language now, but picking the one and only sensible encoding really shouldn't be a problem in 2021.
    • von_lohengramm 5y ago
      It's not that simple. Obviously, you'd choose UTF-8, but that exposes another issue: a codepoint is not a character. There's no simple & elegant solution to this at the language level.
      • turminal 5y ago
        > a codepoint is not a character Precisely. That's why I think representing strings as character slices and letting third party libraries handle them is not a good solution. Deciding what features to put in the language vs. the stdlib vs. third party libraries is one of the hardest parts of language design. Personally I believe strings are important and frequent enough they deserve special treatment at the language level. Edit (late addition): I think not treating strings specially is mostly fine in C, but Zig seems to aim at being a little less lowlevel.
        • von_lohengramm 5y ago
          > representing strings as character slices Zig does not do this. It represents strings as byte slices. A UTF-8 character could be multiple codepoints with each codepoint being multiple bytes. A big thing about Zig is not hiding complexity. If UTF-8 were implemented at a language level (whatever that means), then "language level" string operations would be non-linear, which would be very non-Ziggy. I could see value in a standard library UTF-8 implementation, but a LOT of forethought would need to be put into it. I think keeping UTF-8 string manipulation at the third-party library level is a good choice for now. Maybe once the language is finalized, the ecosystem is more developed, and lessons have been learned from the third-party libraries, then the standard library can implement this.
          • turminal 5y ago
            > character slices Sorry, byte slices is what I had in mind. I'm not talking about language level string operations as in concatenation with + or something like that at all, because that certainly wouldn't make sense in language like Zig. I'm not advocating for string functionality in the language, I'm advocating for a way to not allow byte slice functionality on a thing that is clearly not a byte slice.
            • von_lohengramm 5y ago
              > a way to not allow byte slice functionality on a thing that is clearly not a byte slice This already exists in the form of structs or opaque types. Both of these approaches would end up being implemented in "userspace" anyways, whether that's standard library or third-party. However, (UTF-8) strings are byte slices. You can do simple manipulation with them as byte slices safely and validly. Split on spaces? Sure. Tokenize? Sure. Find substring? Sure. You can't do things that depend on say UTF-8 graphemes, but you can safely do most things that depend on bytes. For most purposes, treating strings as byte slices is the safest and correct approach.
              • lmm 5y ago
                Doing find substring by find byte subsequence won't behave correctly in many cases, where semantically equivalent strings have multiple different bytesequence representation. Treating strings as byte slices exposes a lot of footguns; it shouldn't be easy just as e.g. treating floating-point numbers as byte sequences shouldn't be easy.
      • coldtea 5y ago
        So let's have everybody write their own incompatible libs and partial solutions... It worked for C /s
  • diegocg 5y ago
    They can put it on the standard library or the core language or wherever they want, but they absolutely need to provide good string handling. And regardless of where they put the code, this is something that needs to be done by the core team. Otherwise you will end with too many string libraries, all of them trying to solve a particular problem and doing bad at everything else, with bad documentation and different API styles and making difficult to mix code that uses two different libraries.
    • AndyKelley 5y ago
      A lot of people reach for string handling when the actual correct thing to do is intentionally avoid string handling, and only handle strings as opaque encoded UTF-8 bytes, that cannot be reasoned about in terms of human language. I would even argue that having string handling in a standard library (or language) has the potential to cause a net increase in bugs, because of people thinking they are handling strings when actually they are just screwing around with codepoints. Go's string handling is completely broken, for example. As a result of strings in the language, Go programs tend to be more broken than C programs in terms of string handling.
      • gautamcgoel 5y ago
        Can you elaborate on why you think Go's string handling is broken?
      • tdeck 5y ago
        > the actual correct thing to do is intentionally avoid string handling, That sounds nice and all, but wr have 50+ years of protocols and formats and APIs built up around strings. Unless you're just writing code to run on a small microcontroller, you need to be able to parse and generate strings. So its going to be pretty frustrating not to have good support for them, or to have every codebase use its own libraries and idioms for working with strings.
        • AndyKelley 5y ago
          Protocols and formats absolutely should not require decoding strings. I think you are mistaken. Can you name any well-established protocol or format that does not treat strings as opaque encoded bytes? Edit: so far these examples have been given: * HTTP: wrong. the spec does not tell you to decode any strings * CSV: wrong. the spec does not tell you to decode any strings, nor is it necessary to have any unicode awareness in order to properly read and parse the data or deal with the delimiters. Additional request: if you attempt to provide a counter-example, please also point to the place in the spec where it tells you to decode a string.
  • winter_squirrel 5y ago
  • formerly_proven 5y ago
    u8 slices strike me as exactly the correct thing on the language level as well. In particular, it allows very straightforward zero-copy parsing/tokenization of strings. Most parsing of strings doesn't really need to know about UTF-8 and works correctly in the presence of non-ASCII codepoints. String literally are not really UTF-8. Rather, zig source files are UTF-8 (by definition), and string literals are u8 literals; putting some UTF-8 between quotes just puts the literal UTF-8 encoded text into the literal because those are the bytes that are in the source file. In particular, string literals can contain arbitrary binary data and null bytes by way of \x00 escapes. For Unicode handling there's some basic stuff in std.unicode (conversion between different UTF encodings, checking validity, decoding to codepoints etc.). This is used e.g. on Windows for checking filesystem paths. I don't general libraries of encodings is really that important today, iso-8859 and shift JIS might be useful sometimes, but everything else probably doesn't need to bloat up a standard library (iirc Python's codecs package, which contains dozens upon dozens of encodings, is like a third of the standard library by size).
    • anonymoushn 5y ago
      Zig already ships a bunch of libcs so I don't think there's much concern about bloating the standard library.
• ruined 5y ago
  thread tl;dr: it doesn't look like there will be language support for things like codepoints or grapheme indexing or treatment of strings as anything but byte arrays, so ddevault is sad. there is intention from andrewrk and jecolon to provide such features in the standard library before 1.0 release. downside to library vs lang support that is you can expect a good chunk of programmers to ignore the less-ergonomic library features, use the language features at hand, and handle strings incorrectly as byte arrays.
  • dnautics 5y ago
    Does this boil to the semantic question of if the stdlib is part of the language?
    • ruined 5y ago
      i don't think so. the answer is presumed no, because everyone in the relevant thread including language authors and stdlib authors (with plenty of overlap) agree that it is not, and agree there are functional distinctions about what is possible on either side of that dividing line (mostly due to zig not supporting custom operators).
    • anonymoushn 5y ago
      One difference is that there won't be a built-in string type with operator[] defined as something other than returning the byte at the indexed position in the array of bytes. There could be most other things one would expect from language-level string support though.
• coldtea 5y ago
  It seems they don't want unicode strings as part of the language, at best as a library. But unless there's a single sactioned one, this will not end well. And this is met with answers like "just avoid string handling" from the language designers... It's probably because they don't work in any related domain, and even less so have to do with international strings (except as byte buckets they don't care about and don't have to do anything do).
  • anonymoushn 5y ago
    It seems way nicer to have a language that treats strings as byte arrays and use libraries to handle encodings than to have a language that treats strings as UCS-2 and use libraries to handle UTF-8 strings that live inside of UCS-2 strings.
    • coldtea 5y ago
      I don't know, it has been a pain to work with strings in any language that does the above, and has seldom (if ever) been a problem with Java, Go, Swift, or even modern Python 3, and so on...
      • anonymoushn 5y ago
        I'm not sure how to explain the difference in experience here. Python 3's built-in string encoding support has been a source of endless pain to me, and Lua's belief that strings are byte arrays has been much easier to deal with.
      • jstimpfle 5y ago
        I don't get what's so hard about it. Most of my programs deal with UTF-8 simply by doing memcpy(). Parsing code just loops over the bytes and compares to ASCII characters (0-9, A-Z, a-z, \n, \r, \t ...). That's how UTF-8 was designed to be used.
        • SuchAnonMuchWow 5y ago
          I assume you never had to deal with unicode normalization ? When you send your unicode string to an external system (for example a storage server with a database) and latter retrieve the string, only to find out that it has been normalized differently so it no longer match byte-for-byte what is stored in your program, and all of a sudden strcmp no longer works. Or all kind of weirdness like that because every system outside of your program will handle unicode differently, and you will need to adapt to them, and having a string library to do most of the heavy-lifting will avoid the need for every user to rewrite a library from scratch. Not to mention that for every developer which rewrite unicode handling functions, you will probably end up with a function with subtly different behaviors from others, which aggravate the problem for others when they will try to communicate with your system.