3 ms·

I don't get why those are bad? - random-access strings - auto-increment operators

steveklabnik 11y ago
The first is bad because of UTF-8, basically. It's unclear what an index should even return: bytes, codepoints, grapheme clusters? Furthermore, because it's a variable-length encoding, it's not O(1) access, which is what [] implies, and since Rust tries to surface the cost of operations, it would be inappropriate for Rust. (Note that Rust _does_ have a ranged syntax here, which returns bytes, and is O(1)) Pre vs post increment just leads to a lot of confusion, and doesn't give you much over `x += 1`, in my opinion. Not sure what Graydon's reasoning is here.
- MrBuddyCasino 11y ago
  Ok, thanks for the details, I suspected as much. My intuition says string indices should be accessible as grapheme clusters (and, additionally, codepoints), bytes are meaningless anyway. Whenever I want to cut and slice a string, I usually want it to be at the boundary of a letter. Yet most languages don't seem to do it this way, strangely enough.
  - rspeer 11y ago
    Well, that's about efficiency. If you want to be able to slice at grapheme boundary n, you have to iterate through the string counting up to n to find out where it is. Might as well just provide an iterator over graphemes in the first place. You can optimize slicing on codepoint boundaries by storing strings internally in a fixed-width representation, like Python does. But now you have to convert encodings all the time just to use UTF-8 for your I/O. And that still only gets you codepoints. It's not a great tradeoff to do it that way. It wastes memory and CPU time, and randomly accessing characters of strings is just not something you actually need to do often enough to optimize for it.
- comex 11y ago
  Just as additional context, since this seems to come up a lot - that does not mean that UTF-8 is bad. The main alternative is UTF-32, which provides random access to codepoints rather than bytes, but since Unicode graphemes consist of multiple codepoints, UTF-32 is effectively a variable width encoding as well for most purposes, just one that uses much more memory than UTF-8 most of the time and never uses less.
  - steveklabnik 11y ago
    Oh yeah, I don't think any of this makes UTF-8 bad, in fact, I think it's the best choice. It just reveals the inherent complexity in handling string data.
- the_why_of_y 11y ago
  And UTF-16 random-access strings (like in Java and many other things designed in the 90s) make the problem even worse, because with UTF-8 you will notice the bugs as soon as the first non-English-speaker uses your system, but with UTF-16 the affected languages are a lot more obscure.
deleted 11y ago
[deleted]
SamReidHughes 11y ago
Go has increment operators, but they have to be in their own statements, you can't intermingle them in an expression. The reason is, putting them in the same line as other stuff makes for a higher chance of bugs. For example, Tarsnap's CTR bug happened because of this. The removal of a nonce increment would have been very obvious if all increments had to be on a separate line, either by language fiat or properly risk-averse C coding practices.
toolslive 11y ago
i += ++i; what should be the result ? what is the result in C++ ? (undefined)
- ggreer 11y ago
  If you're making a new language, that example could go a couple of ways: 1. It could be a compiler error. The existence of certain operators doesn't mean they can be combined arbitrarily. 2. Sequence points could be defined differently, allowing for such a convoluted line. This would likely hurt performance, as the compiler would have fewer operations to reorder in each sequence point. Typically, increment and decrement are just syntactic sugar. I don't mind if a language lacks them, but I can see why people like them. A convoluted example of UB in C++ is not a good argument against having them. Also: In C++, overloaded operators are function calls, and function calls are sequence points. So if i is an object, the behavior is defined. That said, it would still be bad code. :)
  - TheLoneWolfling 11y ago
    I am not sure about if 2 actually would hurt performance in the real world. Remember the whole "must appear to be in order" thing. If the compiler knows that it doesn't change things if they are out of order, it's free to rearrange them. Note: things like this are part of the potential advantages of static linking.
- eddyb 11y ago
  This could actually work fine in Rust, because it has deterministic evaluation ordering for everything (or almost everything, though I can't think of anything that isn't deterministic). Knowing that `++i` is `{ i += 1; i }`, we have `i += { i += 1; i };` (which should compile already). The nested assignment can be hoisted to obtain `i += 1; i += i;`. That means Rust would compile `i += ++i;` as `i = (i + 1) * 2;`.

The first is bad because of UTF-8, basically. It's unclear what an index should even return: bytes, codepoints, grapheme clusters? Furthermore, because it's a variable-length encoding, it's not O(1) access, which is what [] implies, and since Rust tries to surface the cost of operations, it would be inappropriate for Rust. (Note that Rust _does_ have a ranged syntax here, which returns bytes, and is O(1)) Pre vs post increment just leads to a lot of confusion, and doesn't give you much over `x += 1`, in my opinion. Not sure what Graydon's reasoning is here.

MrBuddyCasino 11y ago
Ok, thanks for the details, I suspected as much. My intuition says string indices should be accessible as grapheme clusters (and, additionally, codepoints), bytes are meaningless anyway. Whenever I want to cut and slice a string, I usually want it to be at the boundary of a letter. Yet most languages don't seem to do it this way, strangely enough.
- rspeer 11y ago
  Well, that's about efficiency. If you want to be able to slice at grapheme boundary n, you have to iterate through the string counting up to n to find out where it is. Might as well just provide an iterator over graphemes in the first place. You can optimize slicing on codepoint boundaries by storing strings internally in a fixed-width representation, like Python does. But now you have to convert encodings all the time just to use UTF-8 for your I/O. And that still only gets you codepoints. It's not a great tradeoff to do it that way. It wastes memory and CPU time, and randomly accessing characters of strings is just not something you actually need to do often enough to optimize for it.
comex 11y ago
Just as additional context, since this seems to come up a lot - that does not mean that UTF-8 is bad. The main alternative is UTF-32, which provides random access to codepoints rather than bytes, but since Unicode graphemes consist of multiple codepoints, UTF-32 is effectively a variable width encoding as well for most purposes, just one that uses much more memory than UTF-8 most of the time and never uses less.
- steveklabnik 11y ago
  Oh yeah, I don't think any of this makes UTF-8 bad, in fact, I think it's the best choice. It just reveals the inherent complexity in handling string data.
the_why_of_y 11y ago
And UTF-16 random-access strings (like in Java and many other things designed in the 90s) make the problem even worse, because with UTF-8 you will notice the bugs as soon as the first non-English-speaker uses your system, but with UTF-16 the affected languages are a lot more obscure.

deleted 11y ago

[deleted]

SamReidHughes 11y ago

Go has increment operators, but they have to be in their own statements, you can't intermingle them in an expression. The reason is, putting them in the same line as other stuff makes for a higher chance of bugs. For example, Tarsnap's CTR bug happened because of this. The removal of a nonce increment would have been very obvious if all increments had to be on a separate line, either by language fiat or properly risk-averse C coding practices.

toolslive 11y ago

i += ++i; what should be the result ? what is the result in C++ ? (undefined)

ggreer 11y ago
If you're making a new language, that example could go a couple of ways: 1. It could be a compiler error. The existence of certain operators doesn't mean they can be combined arbitrarily. 2. Sequence points could be defined differently, allowing for such a convoluted line. This would likely hurt performance, as the compiler would have fewer operations to reorder in each sequence point. Typically, increment and decrement are just syntactic sugar. I don't mind if a language lacks them, but I can see why people like them. A convoluted example of UB in C++ is not a good argument against having them. Also: In C++, overloaded operators are function calls, and function calls are sequence points. So if i is an object, the behavior is defined. That said, it would still be bad code. :)
- TheLoneWolfling 11y ago
  I am not sure about if 2 actually would hurt performance in the real world. Remember the whole "must appear to be in order" thing. If the compiler knows that it doesn't change things if they are out of order, it's free to rearrange them. Note: things like this are part of the potential advantages of static linking.
eddyb 11y ago
This could actually work fine in Rust, because it has deterministic evaluation ordering for everything (or almost everything, though I can't think of anything that isn't deterministic). Knowing that `++i` is `{ i += 1; i }`, we have `i += { i += 1; i };` (which should compile already). The nested assignment can be hoisted to obtain `i += 1; i += i;`. That means Rust would compile `i += ++i;` as `i = (i + 1) * 2;`.