3 ms·

1) Hopefully memory-safety without GC will be mainstream soon. Rust is showing the way. 2) I agree that time and space complexity should be part of the contrac

1) Hopefully memory-safety without GC will be mainstream soon. Rust is showing the way. 2) I agree that time and space complexity should be part of the contract. 3) I agree that escape hatches are needed, but they shouldn't be the default. Most code isn't performance sensitive, but all code is bug sensitive. 4) The best internal representation is probably UTF-8, but I don't feel strongly about that. What matters is that the available string operations are specified in terms of Unicode, not bytes. I grew up in Russia and I remember very well the "fun" of programming with Cyrillic strings in a mixture of Win-1251, KOI8-R, and Unicode. Working with strings as untagged bytes is a disaster everywhere except English speaking countries.

geocar 11y ago
I once observed a lot of bugs showing up in a Python project because of transparent conversion between byte-strings and unicode-strings. Programmers would test with byte-strings, everything is fine, customer is in east Europe, and you know what happens next. I think I decided that UTF-8 was actually worse than UTF-8-xor-69 so that the innermost code should be able to tell whether or note this was a bytestring or not. Explicit tagging found loads of bugs that day. An array of bytes is a storage medium. If you don't need the number of utf-8 characters (and given things like combining characters and invisible spaces, it's rare that's what I actually need) and just need to print and compare, byte-strings are efficient and acceptable. The text remains utf-8 encoded, but `length` is `O(1)` or a simple scan, and `substr` (or similar) are `O(1)`+the cost of allocating. However if I need to recode things, an array of 32-bit (or 64-bit hah) ints is sometimes better. If the language has a real iterator, UTF-8 might still be okay, but then you are still using a bytestring anyway. Lastly, if the "string" actually knows its character set, then using things besides UTF-8 might be good because it saves a lot of space. Maybe. I'm not really convinced because the tag for the character set (as you rightly think necessary) takes up space too. Dealing with these kinds of "strings" is really complicated, so I'd agree that anyone who would jump into complexity without some real justification is, as you say, a bozo.
- cousin_it 11y ago
  Most importantly to me, there must be no way to convert from string to exposed bytes (or vice versa) without explicitly specifying an encoding. I guess we agree on that!
  - geocar 11y ago
    I think we agree as well. I think a lot of trouble showed up when people thought "string" meant "text", and new programmers learned (in a 7-bit world) how to make programs this way. And then unicode. I try to be specific and say "array of bytes" (or sometimes "bytestring") if I mean something that's accessed a byte-at-a-time, but since most text is just copied; sent verbatim to the screen, to the HTTP response, etc, I worry about the `O(N)` access and counting algorithms showing up in languages.
- ucho 11y ago
  > the tag for the character set (as you rightly think necessary) takes up space too Java recently (JEP 254) moved to internal flag for Strings (ISO-8859-1 or UTF16) as most memory was used by strings without any special characters. It was decided that additional complexity was worth it. I guess that getting rid of UTF-16 would be even better as it is the worst Unicode encoding.

geocar 11y ago

I once observed a lot of bugs showing up in a Python project because of transparent conversion between byte-strings and unicode-strings. Programmers would test with byte-strings, everything is fine, customer is in east Europe, and you know what happens next. I think I decided that UTF-8 was actually worse than UTF-8-xor-69 so that the innermost code should be able to tell whether or note this was a bytestring or not. Explicit tagging found loads of bugs that day. An array of bytes is a storage medium. If you don't need the number of utf-8 characters (and given things like combining characters and invisible spaces, it's rare that's what I actually need) and just need to print and compare, byte-strings are efficient and acceptable. The text remains utf-8 encoded, but `length` is `O(1)` or a simple scan, and `substr` (or similar) are `O(1)`+the cost of allocating. However if I need to recode things, an array of 32-bit (or 64-bit hah) ints is sometimes better. If the language has a real iterator, UTF-8 might still be okay, but then you are still using a bytestring anyway. Lastly, if the "string" actually knows its character set, then using things besides UTF-8 might be good because it saves a lot of space. Maybe. I'm not really convinced because the tag for the character set (as you rightly think necessary) takes up space too. Dealing with these kinds of "strings" is really complicated, so I'd agree that anyone who would jump into complexity without some real justification is, as you say, a bozo.

cousin_it 11y ago
Most importantly to me, there must be no way to convert from string to exposed bytes (or vice versa) without explicitly specifying an encoding. I guess we agree on that!
- geocar 11y ago
  I think we agree as well. I think a lot of trouble showed up when people thought "string" meant "text", and new programmers learned (in a 7-bit world) how to make programs this way. And then unicode. I try to be specific and say "array of bytes" (or sometimes "bytestring") if I mean something that's accessed a byte-at-a-time, but since most text is just copied; sent verbatim to the screen, to the HTTP response, etc, I worry about the `O(N)` access and counting algorithms showing up in languages.
ucho 11y ago
> the tag for the character set (as you rightly think necessary) takes up space too Java recently (JEP 254) moved to internal flag for Strings (ISO-8859-1 or UTF16) as most memory was used by strings without any special characters. It was decided that additional complexity was worth it. I guess that getting rid of UTF-16 would be even better as it is the worst Unicode encoding.