4 ms·

If I ever wrote a language, handling strings wrt unicode would be the first thing I would fix so so that it's clear, concise and especially consistent. It both

If I ever wrote a language, handling strings wrt unicode would be the first thing I would fix so so that it's clear, concise and especially consistent. It bothers me that in 2019 it can still be difficult to manage even basic things with strings, and yet, whenever we talk/argue/bike-shed about 'languages' we get caught up in intellectual meandering. Please, give us strings that make sense. Then argue about monads.

saagarjha 8y ago
> give us strings that make sense This is really, really hard.
asveikau 8y ago
I don't think a lot of people realize that Unicode itself, in any representation, is complex, and there is no one abstraction that will make it always easy in all cases. So I think your goal is unobtainable. Some of that complexity is because world languages themselves are not simple. (eg. If you were to ditch Unicode and start from scratch, it would remain hard to do bi-di text, to pick a random example.)
- sonnyblarney 8y ago
  Totally see what you are saying (the underlying issue is language itself), but it's pragmatically obtainable.
  - nicky0 8y ago
    The think is, strings that make sense has been a major goal of Swift, that's why they have been iterating on it so heavily. FWIW I think Swift's is probably one of the cleanest String abstractions out there with respect to unicode correctness balanced against efficiency and usability.
    - asveikau 8y ago
      To be clear I applaud successful attempts to make it better, what I doubt is a possibility to make the problem completely go away without users of the api ever having to think.
  - masklinn 8y ago
    Ah, so your solution is the same old canard: just ignore the complexity and call it pragmatic.
    - sonnyblarney 8y ago
      No. How about a string that returns a Unicode character length? That's not 'ignoring complexity' and it's pragmatic. That would help in English as it's really hard today just to deal emojis.
      - asveikau 8y ago
        The point I tried to make: I suspect if you dug really deep into what is a character or glyph you might find "character length" is an imprecise and messy notion, with differing performance costs associated with differing definitions which may or may not apply to you. So even if you pick one definition and decide to live with it always... You'll still have to know what tradeoffs you (or an API designer or implementer) made.
        sonnyblarney 8y ago
        First - a combined unicode length is definitely universally useful. Second - a grapheme length is universally useful. Third - I didn't say 'pick one definition' - you did. A simple function which took as a parameter some kind of linguistic construct would be immensely useful in cases of choice/ambiguity. 'It's complicated' is the utterly the wrong answer as to why we 'can't have better strings'. Strings in almost every programming language were designed before Emjojis and true internationalization. If we designed them today, they would have support for dealing with most of the problems we face. And FYI - there are not infinite corner cases. They are limited, and we can develop constructs for dealing with all of them.
        asveikau 8y ago
        There are myriad problems today that 99% of developers couldn't be bothered to think about, but can be a source of bugs. eg. I'm going to guess every time you call tolower() have you haven't thought "I wonder what this does to Turkish I", but it's still there. An API won't fix us from having to think, and having ambiguities occasionally become bugs. (The fact that you've acknowledged the need for multiple string length functions -- I agree with it, but it also means you'll sometimes use the wrong one.) As I said in another comment, I still applaud efforts to improve the status quo.
Twisell 8y ago
Tell that to the Python community where switch to UTF8 was/is a huge battle! Swift is meant to be retro compatible with Obj-C and all. As someone pointed out windows in the meantime have basically no announced battle plan to transition out of UTF16. I this game latecomer were rewarded because they started with UTF8 or switched from ASCI. But innovators actually tried UTF16 in the mean time an now have to do the job twice in a retro-compatible manner.
BigWhale 8y ago
In what language do you want them to make sense? A Hindi string would be very different to an Arabic strig, or a japanese string, or an emoji string.

saagarjha 8y ago

> give us strings that make sense This is really, really hard.

asveikau 8y ago

I don't think a lot of people realize that Unicode itself, in any representation, is complex, and there is no one abstraction that will make it always easy in all cases. So I think your goal is unobtainable. Some of that complexity is because world languages themselves are not simple. (eg. If you were to ditch Unicode and start from scratch, it would remain hard to do bi-di text, to pick a random example.)

sonnyblarney 8y ago
Totally see what you are saying (the underlying issue is language itself), but it's pragmatically obtainable.
- nicky0 8y ago
  The think is, strings that make sense has been a major goal of Swift, that's why they have been iterating on it so heavily. FWIW I think Swift's is probably one of the cleanest String abstractions out there with respect to unicode correctness balanced against efficiency and usability.
  - asveikau 8y ago
    To be clear I applaud successful attempts to make it better, what I doubt is a possibility to make the problem completely go away without users of the api ever having to think.
- masklinn 8y ago
  Ah, so your solution is the same old canard: just ignore the complexity and call it pragmatic.
  - sonnyblarney 8y ago
    No. How about a string that returns a Unicode character length? That's not 'ignoring complexity' and it's pragmatic. That would help in English as it's really hard today just to deal emojis.
    - asveikau 8y ago
      The point I tried to make: I suspect if you dug really deep into what is a character or glyph you might find "character length" is an imprecise and messy notion, with differing performance costs associated with differing definitions which may or may not apply to you. So even if you pick one definition and decide to live with it always... You'll still have to know what tradeoffs you (or an API designer or implementer) made.
      - sonnyblarney 8y ago
        First - a combined unicode length is definitely universally useful. Second - a grapheme length is universally useful. Third - I didn't say 'pick one definition' - you did. A simple function which took as a parameter some kind of linguistic construct would be immensely useful in cases of choice/ambiguity. 'It's complicated' is the utterly the wrong answer as to why we 'can't have better strings'. Strings in almost every programming language were designed before Emjojis and true internationalization. If we designed them today, they would have support for dealing with most of the problems we face. And FYI - there are not infinite corner cases. They are limited, and we can develop constructs for dealing with all of them.
        asveikau 8y ago
        There are myriad problems today that 99% of developers couldn't be bothered to think about, but can be a source of bugs. eg. I'm going to guess every time you call tolower() have you haven't thought "I wonder what this does to Turkish I", but it's still there. An API won't fix us from having to think, and having ambiguities occasionally become bugs. (The fact that you've acknowledged the need for multiple string length functions -- I agree with it, but it also means you'll sometimes use the wrong one.) As I said in another comment, I still applaud efforts to improve the status quo.

Twisell 8y ago

Tell that to the Python community where switch to UTF8 was/is a huge battle! Swift is meant to be retro compatible with Obj-C and all. As someone pointed out windows in the meantime have basically no announced battle plan to transition out of UTF16. I this game latecomer were rewarded because they started with UTF8 or switched from ASCI. But innovators actually tried UTF16 in the mean time an now have to do the job twice in a retro-compatible manner.

BigWhale 8y ago

In what language do you want them to make sense? A Hindi string would be very different to an Arabic strig, or a japanese string, or an emoji string.