4 ms·

There are a couple cases for string indexing, usually involving parsing or regular expressions. You might want to slice the quotes off of a quoted string, or s

There are a couple cases for string indexing, usually involving parsing or regular expressions. You might want to slice the quotes off of a quoted string, or slice from one match of a regular expression to a match of a different regular expression starting at a different index. These come up infrequently enough that it doesn't make sense to make a better API just for these use cases, but frequently enough that it would be a serious impediment if we didn't do some kind of string indexing. I agree, however, that it's completely irrelevant whether the indexes correspond to code units (i.e. byte offsets in UTF-8) or whether they correspond to code points (how it works in Python currently), as long as we have some way to store, compare, and otherwise manipulate locations within a string. Some Rust developers at one point proposed making string indexes their own (opaque) type, as you suggest, so that they couldn't be confused with integers used for other purposes. The extra complexity of such an API meant that this proposal was never really taken seriously, and it only prevents a small category of programming errors. You might be interested in looking at some string APIs which are mostly without string indexing, like Haskell's Data.Text, which is one of the most well-designed string APIs ever made. https://hackage.haskell.org/package/text-1.2.2.1/docs/Data-Text.html https://hackage.haskell.org/package/text-1.2.2.1/docs/Data-T... As for Windows, my Windows apps use UTF-8 everywhere, and then convert to wchar_t at the last possible moment when interacting with the Windows API. I believe this is what UTF-8 Everywhere suggests.

Avernar 10y ago
> You might want to slice the quotes off of a quoted string That's why I really like dealing with UTF-8. As you said you can just index by byte instead of having to worry about code point boundries. This is because a code point will never match inside a larger code point. So if I search for a one byte quote character it will never match the second, third or fourth bytes in a larger code point. Same with other control characters. Works when searching for 2 or 3 byte code points as well.
Animats 10y ago
The advantage of an opaque type is that you should be able to add one to it and advance one rune. Go copies the string functions from C, but doesn't (at least as of last year) offer "advance one rune" and "back up one rune" functions.
- Avernar 10y ago
  If by rune you mean code point that doesn't really gain you anything. If you are searching for something it's faster to just scan a byte at a time. If you want to split the string into "characters" you need to do it a the grapheme level (multiple code points) and for that you need to use a unicode library. But that adds overhead when you're just scanning. A JSON or XML scanner does not need the added overhead of advancing by code point or grapheme.
  - dietrichepp 10y ago
    I don't understand the argument for "characters = grapheme clusters". From my perspective, there are a lot of different ways you'd want to iterate over a string. Grapheme cluster breaks, tailored grapheme cluster breaks, word breaks, line breaks, code points, code units… all of these make sense in some context. However, there are precious few times that I've wanted to iterate over grapheme clusters, so telling people that they should do that instead of something else doesn't make sense to me. (I mean, what problem is so common that we would want to iterate this way by default?) For parsing, it often makes sense to iterate over code points or code units, since many languages are defined in terms of code points (and you can translate that to code units, for performance). XML 1.1, JavaScript, Haskell, etc... many languages are defined in terms of the underlying code points and their character classes in the Unicode standard. JSON and XML 1.0 are not everything.
    - Avernar 10y ago
      We're pretty much on the same page here. When you want to slice a string (because you can only display or store a certain amount) or you want to do text selection and other cursor operations you can't so it by code point. That's where you want to break at character boundries which are graphemes or grapheme clusters. For parsing it's easier to just scan for a byte sequence in UTF-8 because you know what you're looking for ahead of time. If you're looking for a matching quote, brace, etc. you just need to scan for a single byte in your text stream. Adding a smart iterator to the process that moves to the start of each code unit is not necessary and will slow things way down. I just gave JSON and XML as examples and not an exhaustive list. If you know the code points you are scanning for it's way more effecient to scan for their code units. The state machine in a paraer will be operating at the byte level anyways. I have yet to see a good example where processing/iterating by code point is the better choice (other than the grapheme code of the unicode library).
bluejekyll 10y ago
> in Rust... The extra complexity of such an API meant that this proposal was never really taken seriously, In fact it looks like even standard indexes have been deprecated, in favor of Iterators over the string: https://doc.rust-lang.org/stable/std/primitive.str.html#method.char_at https://doc.rust-lang.org/stable/std/primitive.str.html#meth... Iteration is better IMO.
- johncolanduoni 10y ago
  Makes a lot of sense. I get frustrated with how explicit I have to be with Rust sometimes (currently I'm writing something with a lot of reusable, immutable structures and am put off by how much Rc.clone() that requires) but making it known to the programmer that every indexing of a String is effectively an iteration is a good call.
  - bluejekyll 10y ago
    Yeah, this is a tough thing when coming from other languages. I've been using references pretty effectively to get around some issues like this, though in other cases Rc is my only resort. If you have some code up on github, I'd be happy to take a look and see if there's some other options that are less cumbersome.

Avernar 10y ago

> You might want to slice the quotes off of a quoted string That's why I really like dealing with UTF-8. As you said you can just index by byte instead of having to worry about code point boundries. This is because a code point will never match inside a larger code point. So if I search for a one byte quote character it will never match the second, third or fourth bytes in a larger code point. Same with other control characters. Works when searching for 2 or 3 byte code points as well.

Animats 10y ago

The advantage of an opaque type is that you should be able to add one to it and advance one rune. Go copies the string functions from C, but doesn't (at least as of last year) offer "advance one rune" and "back up one rune" functions.

Avernar 10y ago
If by rune you mean code point that doesn't really gain you anything. If you are searching for something it's faster to just scan a byte at a time. If you want to split the string into "characters" you need to do it a the grapheme level (multiple code points) and for that you need to use a unicode library. But that adds overhead when you're just scanning. A JSON or XML scanner does not need the added overhead of advancing by code point or grapheme.
- dietrichepp 10y ago
  I don't understand the argument for "characters = grapheme clusters". From my perspective, there are a lot of different ways you'd want to iterate over a string. Grapheme cluster breaks, tailored grapheme cluster breaks, word breaks, line breaks, code points, code units… all of these make sense in some context. However, there are precious few times that I've wanted to iterate over grapheme clusters, so telling people that they should do that instead of something else doesn't make sense to me. (I mean, what problem is so common that we would want to iterate this way by default?) For parsing, it often makes sense to iterate over code points or code units, since many languages are defined in terms of code points (and you can translate that to code units, for performance). XML 1.1, JavaScript, Haskell, etc... many languages are defined in terms of the underlying code points and their character classes in the Unicode standard. JSON and XML 1.0 are not everything.
  - Avernar 10y ago
    We're pretty much on the same page here. When you want to slice a string (because you can only display or store a certain amount) or you want to do text selection and other cursor operations you can't so it by code point. That's where you want to break at character boundries which are graphemes or grapheme clusters. For parsing it's easier to just scan for a byte sequence in UTF-8 because you know what you're looking for ahead of time. If you're looking for a matching quote, brace, etc. you just need to scan for a single byte in your text stream. Adding a smart iterator to the process that moves to the start of each code unit is not necessary and will slow things way down. I just gave JSON and XML as examples and not an exhaustive list. If you know the code points you are scanning for it's way more effecient to scan for their code units. The state machine in a paraer will be operating at the byte level anyways. I have yet to see a good example where processing/iterating by code point is the better choice (other than the grapheme code of the unicode library).

bluejekyll 10y ago

> in Rust... The extra complexity of such an API meant that this proposal was never really taken seriously, In fact it looks like even standard indexes have been deprecated, in favor of Iterators over the string: https://doc.rust-lang.org/stable/std/primitive.str.html#method.char_at https://doc.rust-lang.org/stable/std/primitive.str.html#meth... Iteration is better IMO.

johncolanduoni 10y ago
Makes a lot of sense. I get frustrated with how explicit I have to be with Rust sometimes (currently I'm writing something with a lot of reusable, immutable structures and am put off by how much Rc.clone() that requires) but making it known to the programmer that every indexing of a String is effectively an iteration is a good call.
- bluejekyll 10y ago
  Yeah, this is a tough thing when coming from other languages. I've been using references pretty effectively to get around some issues like this, though in other cases Rc is my only resort. If you have some code up on github, I'd be happy to take a look and see if there's some other options that are less cumbersome.