3 ms·

Article claims python 3 uses UTF-8. https://stackoverflow.com/questions/1838170/ https://stackoverflow.com/questions/1838170/ "In Python 3.3 and above, the int

Article claims python 3 uses UTF-8. https://stackoverflow.com/questions/1838170/ https://stackoverflow.com/questions/1838170/ "In Python 3.3 and above, the internal representation of the string will depend on the string, and can be any of latin-1, UCS-2 or UCS-4, as described in PEP 393." Article also says PHP has immutable strings. They are mutable, although often copied. Article also claims majority of popular languages have immutable strings. As well as the ones listed there is also PHP and Rust (and C, but they did say C++ - and obviously Ruby since that's the subject of the article). I'm also a bit surprised by the last sentence. "However, if you do measure a negative performance impact, there is no doubt you are measuring incorrectly." There must surely be programs doing a lot of string building or in-place modification that would benefit from non-frozen.

ameliaquining 11mo ago
In C, C++, and Rust, the question of "are strings in this language mutable or immutable?" isn't applicable, because those languages have transitive mutability qualifiers. So they only need a single string type, and whether you can mutate it or not depends on context. (C++ and Rust have multiple string types, but the differences among them aren't about mutability.) In languages without this feature, a given value is either always mutable or never mutable, and so it's necessary to pick one or the other for string literals.
- capitainenemo 11mo ago
  Sure, that doesn't change the point that mutable strings are a thing in those languages. And I don't think C's const is really a "mutability qualifier" - certainly not a very effective one at any rate.
  - nilslindemann 11mo ago
    For the records, mutable strings, eh, bytearray objects, are also a thing in Python: https://docs.python.org/3/library/stdtypes.html#bytearray-objects https://docs.python.org/3/library/stdtypes.html#bytearray-ob...
chrismorgan 11mo ago
Python strings aren’t even proper Unicode strings. They’re sequences of code points rather than scalar values, meaning they can contain surrogates. This is incompatible with basically everything: UTF-* as used by sensible things, and unvalidated UTF-16 as used in the likes of JavaScript, Windows wide strings and Qt.
- nilslindemann 11mo ago
  But isn't 'surrogateescape' supposed to address this? (no expert) https://vstinner.github.io/pep-383.html https://vstinner.github.io/pep-383.html
  - chrismorgan 11mo ago
    surrogateescape is something else altogether. It’s a hack to allow non-Unicode file names/environment variables/command line arguments in an otherwise-Unicode environment, by smuggling them through a part of the surrogate range (0x80 to 0xFF → U+DC80 to U+DCFF) which otherwise can’t occur (since it’s invalid Unicode). It’s a cunning hack that makes a lot of sense: they used a design error in one place (Python string representation) to cancel out a design error in another place (POSIX being late to the game on Unicode)!
    - Dylan16807 11mo ago
      It's not taking advantage of the weird way python strings work. You can put that hack on top of any string format that converts back and forth with unicode.
      - chrismorgan 11mo ago
        No you can’t: it only works if your string type is something other than a sequence of Unicode scalar values. In Rust, for example, strings must be valid UTF-8, so this hack is not possible.
        11mo ago
byroot 11mo ago
> There must surely be programs doing a lot of string building or in-place modification that would benefit from non-frozen. The point is that the magic comment (or the --enable-frozen-string-literal) only applies to literals. If you have some code using mutable strings to iteratively append to it, flipping that switch doesn't change that. It just means you'll have to explicitly create a mutable string. So it doesn't change the performance profile.
byroot 11mo ago
> can be any of latin-1, UCS-2 or UCS-4, as described in PEP 393 My bad, I haven't seriously used Python for over 15 years now, so I stand corrected (and will clarify the post). My main point stands though, Python strings have an internal representation, but it's not exposed to the user like Ruby strings. > Article also says PHP has immutable strings. They are mutable, although often copied. Same. Thank you for the correction, I'll update the post.
- capitainenemo 11mo ago
  Cool, although I feel if on one side you have Java, JavaScript, Python, Go and on the other Perl, PHP, C/C++, Ruby, Rust it's hard to say overwhelming majority in either direction. Also someone below claims python byte arrays can be considered mutable strings, although I have no idea of the stringy ergonomics of that and whether it would be convenient to do - I try to avoid python too.
  - capitainenemo 11mo ago
    ... and honestly, since java has both stringbuffer and string I feel it's really in the "has mutable" camp too

ameliaquining 11mo ago

In C, C++, and Rust, the question of "are strings in this language mutable or immutable?" isn't applicable, because those languages have transitive mutability qualifiers. So they only need a single string type, and whether you can mutate it or not depends on context. (C++ and Rust have multiple string types, but the differences among them aren't about mutability.) In languages without this feature, a given value is either always mutable or never mutable, and so it's necessary to pick one or the other for string literals.

capitainenemo 11mo ago
Sure, that doesn't change the point that mutable strings are a thing in those languages. And I don't think C's const is really a "mutability qualifier" - certainly not a very effective one at any rate.
- nilslindemann 11mo ago
  For the records, mutable strings, eh, bytearray objects, are also a thing in Python: https://docs.python.org/3/library/stdtypes.html#bytearray-objects https://docs.python.org/3/library/stdtypes.html#bytearray-ob...

chrismorgan 11mo ago

Python strings aren’t even proper Unicode strings. They’re sequences of code points rather than scalar values, meaning they can contain surrogates. This is incompatible with basically everything: UTF-* as used by sensible things, and unvalidated UTF-16 as used in the likes of JavaScript, Windows wide strings and Qt.

nilslindemann 11mo ago
But isn't 'surrogateescape' supposed to address this? (no expert) https://vstinner.github.io/pep-383.html https://vstinner.github.io/pep-383.html
- chrismorgan 11mo ago
  surrogateescape is something else altogether. It’s a hack to allow non-Unicode file names/environment variables/command line arguments in an otherwise-Unicode environment, by smuggling them through a part of the surrogate range (0x80 to 0xFF → U+DC80 to U+DCFF) which otherwise can’t occur (since it’s invalid Unicode). It’s a cunning hack that makes a lot of sense: they used a design error in one place (Python string representation) to cancel out a design error in another place (POSIX being late to the game on Unicode)!
  - Dylan16807 11mo ago
    It's not taking advantage of the weird way python strings work. You can put that hack on top of any string format that converts back and forth with unicode.
    - chrismorgan 11mo ago
      No you can’t: it only works if your string type is something other than a sequence of Unicode scalar values. In Rust, for example, strings must be valid UTF-8, so this hack is not possible.
      - 11mo ago

byroot 11mo ago

> There must surely be programs doing a lot of string building or in-place modification that would benefit from non-frozen. The point is that the magic comment (or the --enable-frozen-string-literal) only applies to literals. If you have some code using mutable strings to iteratively append to it, flipping that switch doesn't change that. It just means you'll have to explicitly create a mutable string. So it doesn't change the performance profile.

byroot 11mo ago

> can be any of latin-1, UCS-2 or UCS-4, as described in PEP 393 My bad, I haven't seriously used Python for over 15 years now, so I stand corrected (and will clarify the post). My main point stands though, Python strings have an internal representation, but it's not exposed to the user like Ruby strings. > Article also says PHP has immutable strings. They are mutable, although often copied. Same. Thank you for the correction, I'll update the post.

capitainenemo 11mo ago
Cool, although I feel if on one side you have Java, JavaScript, Python, Go and on the other Perl, PHP, C/C++, Ruby, Rust it's hard to say overwhelming majority in either direction. Also someone below claims python byte arrays can be considered mutable strings, although I have no idea of the stringy ergonomics of that and whether it would be convenient to do - I try to avoid python too.
- capitainenemo 11mo ago
  ... and honestly, since java has both stringbuffer and string I feel it's really in the "has mutable" camp too