7 ms·

Language design is a curious mixture of grand ideas and fiddly details I hadn't heard this last one before, but it's SO right ... I always wondered why JS and

by chubot 3y ago

Language design is a curious mixture of grand ideas and fiddly details I hadn't heard this last one before, but it's SO right ... I always wondered why JS and PHP and Perl got so many details "wrong" (e.g. with Perl, one definition of "wrong" is that Perl 6 / Raku didn't make the same design choice) Turns out there's an avalanche of details, and they interact in many ways! Python did better, but I strongly argue both Python 3 and Python 2 got strings wrong. (Array of code points isn't generally useful, and it's hard to implement efficiently. See fish shell discussion about wchar_t on the front page now; also see Guile Scheme) OCaml seems to have gotten mutable strings wrong (for some time), and also I think the split between regular sum types and GADTs is awkward. And also most people argue that objects vs. records vs. modules is suboptimal. And a bunch of mistakes with syntactic consistency, apparently. Looks like almost every language had problems with for loops and closures, including C# and Go - https://news.ycombinator.com/item?id=37575204 https://news.ycombinator.com/item?id=37575204 So basically I agree that plowing through all the details -- and really observing their consequences in real programs -- is more important and time-consuming than grand ideas. But if you lack any grand ideas, then the language will probably turn out poorly too. And you probably won't have any reason to finish it.

gautamcgoel 3y ago
Can you explain how you think strings should work?
- deleted 3y ago
  [deleted]
- klodolph 3y ago
  I think the main alternative design is to treat strings like in Rust or Go. The problem with the “array of code points” idea is that you end up with the most general implementation, which is a UTF-32 string, and then you end up with the fastest implementation, which is a UTF-8 string, and maybe throw in UCS-2 for good measure. These all have the same asymptotic performance characteristics, but allow ASCII strings (which are extremely common) to be stored with less memory. The cost is that now you have two or three different string representations floating around. This approach is used by Python and Java, for example. The Rust / Go approach is to assume that you don’t need O(1) access to the Nth code point in a string, which is probably reasonable, since that’s rarely necessary or even useful. You get a lot of complexity savings from only using one encoding, and the main tradeoff is that certain languages take 50% more space in memory. Python and Java both date back to an era where fixed-width string encodings were the norm.
  - gautamcgoel 3y ago
    Thanks for your response. Personally I fall into the "strings are arrays of bytes" camp (which is also shared by Go). A difference between my view and that of the Go designers is that I don't feel that it is important to support Unicode by default and am perfectly happy to assume that every character corresponds to a single byte. Obviously that makes internationalization harder, but the advantage is that strings are much simpler to reason about. For example, the number of characters in the string is simply the length of the string. I would be fine having a separate Unicode string type in the standard library for those instances when you really need Unicode; this design makes the common case much simpler at the expense of making the rare case harder. I also don't see that mutability is such a huge deal unless you absolutely insist that your language support string interning.
    - saghm 3y ago
      "strings are arrays of bytes" combined with the assumption that "characters are a single byte" sounds basically the same as the "array of code points" that the parent comment is disagreeing with
      - foolswisdom 3y ago
        Code points are not bytes.
        saghm 3y ago
        Sure, but if you're insisting that the string be represented as one byte per character, you end up with the exact same properties with "array of code points" and "array of bytes"
        bruce511 3y ago
        Sort-of, but no, because code points are not characters. There's a big difference between "get the 5th code point" and "get the 5th character". Because multiple code points can be used in a single character, it not possible to do random-character-access in a unicode-encoded string.
- o11c 3y ago
  Unfortunately, strings cross at least 3 different problems: * charset encoding. Cases worth supporting include Ascii, Latin1, Xascii, WeirdLegacyStatefulEncoding, WhateverMyLocaleSaysExceptNotReally, UTF8, UTF16, UCS2, UTF32, and sloppy variants thereof. Note that not supporting sloppiness means it is impossible to access a lot of old data (for example, `git` committers and commit messages). Note that it is impossible to make a 1-to-1 mapping between sloppy UTF-8 and sloppy UTF-16, so if all strings have a single representation (unless it is some weird representation not yet mentioned), it is either impossible to support all strings encountered on non-Windows platforms, or impossible to support all strings encountered on non-Windows platforms. I am a strong proponent of APIs supporting multiple compile-time-known string representations, with transparent conversion where safe. * ownership. Cases worth supporting: Value (but finite size), Refcounted (yes, this is important, the problem with std::string is that it was mutable), Tail or full Slice thereof, borrowed Zero-terminated or X(not) terminated, Literal (known statically allocated), and Alternating (the one that does SSO, switching between Value and Refcounted; IME it is important for Refcounted to efficiently support Literal). Note that there is no need for an immutable string to support Unique ownership. That's 8 different ownership policies, and if your Z/X implementation doesn't support recovering an optional owner, you also need to support at least one "maybe owned, maybe borrowed" (needed for things like efficient map insertion if the key might already exist; making the insert function a template does not suffice to handle all cases). It is important that, to the extent possible, all these (immutable) strings offer the same API, and can be converted implicitly where safe (exception: legacy code might make implicit conversion to V useful, despite being technically wrong). (there should be some Mutable string-like thing but it need not provide the API, only push/pop off the end followed by conversion to R; consider particularly the implementation of comma-separated list stringification) * domain meaning. The language itself should support at least Format strings for printf/scanf/strftime etc. (these "should" be separate types, but if relying on the C compiler to check them for you, don't actually have to be). Common library-supported additions include XML, JSON, and SQL strings, to make injection attacks impossible at the type level. Thinking of compilers (but not limited to them), there also needs to be dedicated types for "string representing a filepath" vs "string representing a file's contents" (the web's concept of "blob URL" is informative, but suffers from overloading the string type in the first place). Importantly, it must be easy to write literals of the appropriate type and convert explicitly as needed, so there should not be any file-related APIs that take strings. (related, it's often useful to have the concept of "single-line string" and "word" (which, among other cases, makes it possible to ); the exact definition thereof depending on context. So it may be useful to be able to tag strings as "all characters (or, separately, the first character or the last character (though "last character" is far less useful)) are one of [abc...]"; reasonably granularity being: NUL, whitespace characters individually, other C0 controls, ASCII symbols individually, digit 0, digit 1, digits 2-7, digits 8-9, letters a-f, letters A-F, letters g-z, letters G-Z, DEL, C1 controls, other latin1, U+0100 through U+07FF, U+0800 through U+FFFF excluding surrogates, low surrogates, high surrogates, and U+10000 through U+10FFFF, and illegal values U+110000 and higher (maybe splitting 31-bit from 32-bit?) (several of these are impossible under certain encodings and strictness levels). Actually supporting all of this in the compiler proper is complicated and likely to be deferred, but thinking about it informs both language and library design. Particularly, consider "partial template casting")
- chubot 3y ago
  The short answer is somewhere between Go and Rust strings, which are newer languages that use UTF-8 for interior representation, and also favor it for exterior encoding. Roughly speaking, Java and JavaScript are in the UTF-16 camp, and Python 2 and 3 are in the code points camp. C and C++ have unique problems, but you could also put them in the code points camp. So there are at least 3 different camps, and a whole bunch of weird variations, like string width being compile-time selectable in interpreters. More details here: https://www.oilshell.org/blog/2023/06/ysh-design.html#text https://www.oilshell.org/blog/2023/06/ysh-design.html#text and here: https://www.oilshell.org/blog/2023/06/surrogate-pair.html#history-windows-infected-javascript-json-and-python https://www.oilshell.org/blog/2023/06/surrogate-pair.html#hi... A main design issue is that string APIs shouldn't depend on a mutable global variable -- the default encoding, or default file system encoding. That's an idea that's a disaster in C, and also a disaster in Python. It leads to buggy programs. Go and Rust differ in their philosophies, but neither of them has that design problem.
- librasteve 3y ago
  Raku introduced the concept of NFG - Normal Form Grapheme - as a way to represent any Unicode string in its logical ‘visual character’ grapheme form. Sequences of combining characters that don’t have a canonical single codepoint form are given a synthetic codepoint so that string methods including regexes can operate on grapheme characters without ever causing splitting side effects. Of course there are methods for manipulating at the codepoint level as well.
k__ 3y ago
Languages have trends too. Weak typing/Implicit conversion was cool in the 90s. JS and PHP would have magnitudes better if they were typed like Python, but that's history.
- deleted 3y ago
  [deleted]
- IggleSniggle 3y ago
  I can appreciate why it was cool. "What if everything could automatically do the right thing when it interacts with other things, and you wouldn't need any of this ritual boilerplate stuff? Why do we keep needing to convert from string to number to string over and over and over it's crazy. The language should just do the right thing!" I think it's very easy to be sympathetic to the design trend.
  - k__ 3y ago
    Yes. Then there is the fact that PHP and JavaScript were basically domain specific languages at their inception. PHP was a hypertext pre-processor. Pearl, but stripped down for templating. JavaScript was an alternative to Java applets. When you don't need the full power of Java, you could throw in a few small scripts. To make things a bit less intimidating, the weak typing seemed like a good idea, and for these use-cases it might very well have been. If you don't know how to program and only want to display some fancy styled text, why would you need to understand that a number isn't a string? But there was the pattern, that "real" programming languages like Java and C/C++ were hard to learn for the average "creator", so they took JS and PHP as entry and built up their skills from there, building bigger and bigger solutions with those tools.
- chubot 3y ago
  There are definitely trends, but the pattern I see with early JS and PHP is they simply didn't anticipate the consequences of certain decisions. What happens is you make a decision "locally", for some use case or program. And then you don't realize how it affects other programs. The well known "Wat" talk about JS by Gary B. is basically a bunch of these unintended consequences. So the problem is really to see the whole picture, and to get some experience using the language. But there's that catch 22 -- by the time you release it, it's almost set in stone. I remember in the early 2000's Guido and the Python team would always refer to "Python 3000" -- the one chance they had to break things!
matheusmoreira 3y ago
Yeah, it's extremely difficult. Everything interacts with everything else. A recent snag I've run into recently with my language: I want to have mutable strings and lists but that complicates their use as hash table keys. I'm so used to the C mindset of just modifying everything in place. It wasn't until I actually tried this that I understood why so many languages just copy everything. I researched how other languages dealt with this problem and many only allow immutable objects as hash keys. Most interesting was Ruby which allows mutable objects as keys, warns programmers that modifying keys can invalidate the hash table and provides a rehashing method to fix it. https://ruby-doc.org/core/Hash.html https://ruby-doc.org/core/Hash.html > Modifying a Hash key while it is in use damages the hash’s index. > You can repair the hash index using method rehash > A String key is always safe > That’s because an unfrozen String passed as a key will be replaced by a duplicated and frozen String I like Ruby a lot.
hardware2win 3y ago
If lang design is so insanely hard, then how C# designers get so many things right?
- bmoxb 3y ago
  I haven't used C# enough to comment on the language itself, but I do know it was designed (at least initially) by Anders Hejlsberg who is very experienced in language design.
- chubot 3y ago
  If basketball is so hard, then how did Michael Jordan score so many points?
  - chubot 3y ago
    Also I pointed out an issue in my comment that C# got wrong, then fixed -- closures created in loops
    - hardware2win 3y ago
      I didnt say that C# is perfect But it feels like there is at least order or two of magnitude of difference. Especially if you start comparing std libs.

by chubot 3y ago

gautamcgoel 3y ago

Can you explain how you think strings should work?

deleted 3y ago
[deleted]
klodolph 3y ago
I think the main alternative design is to treat strings like in Rust or Go. The problem with the “array of code points” idea is that you end up with the most general implementation, which is a UTF-32 string, and then you end up with the fastest implementation, which is a UTF-8 string, and maybe throw in UCS-2 for good measure. These all have the same asymptotic performance characteristics, but allow ASCII strings (which are extremely common) to be stored with less memory. The cost is that now you have two or three different string representations floating around. This approach is used by Python and Java, for example. The Rust / Go approach is to assume that you don’t need O(1) access to the Nth code point in a string, which is probably reasonable, since that’s rarely necessary or even useful. You get a lot of complexity savings from only using one encoding, and the main tradeoff is that certain languages take 50% more space in memory. Python and Java both date back to an era where fixed-width string encodings were the norm.
- gautamcgoel 3y ago
  Thanks for your response. Personally I fall into the "strings are arrays of bytes" camp (which is also shared by Go). A difference between my view and that of the Go designers is that I don't feel that it is important to support Unicode by default and am perfectly happy to assume that every character corresponds to a single byte. Obviously that makes internationalization harder, but the advantage is that strings are much simpler to reason about. For example, the number of characters in the string is simply the length of the string. I would be fine having a separate Unicode string type in the standard library for those instances when you really need Unicode; this design makes the common case much simpler at the expense of making the rare case harder. I also don't see that mutability is such a huge deal unless you absolutely insist that your language support string interning.
  - saghm 3y ago
    "strings are arrays of bytes" combined with the assumption that "characters are a single byte" sounds basically the same as the "array of code points" that the parent comment is disagreeing with
    - foolswisdom 3y ago
      Code points are not bytes.
      - saghm 3y ago
        Sure, but if you're insisting that the string be represented as one byte per character, you end up with the exact same properties with "array of code points" and "array of bytes"
        bruce511 3y ago
        Sort-of, but no, because code points are not characters. There's a big difference between "get the 5th code point" and "get the 5th character". Because multiple code points can be used in a single character, it not possible to do random-character-access in a unicode-encoded string.
o11c 3y ago
Unfortunately, strings cross at least 3 different problems: * charset encoding. Cases worth supporting include Ascii, Latin1, Xascii, WeirdLegacyStatefulEncoding, WhateverMyLocaleSaysExceptNotReally, UTF8, UTF16, UCS2, UTF32, and sloppy variants thereof. Note that not supporting sloppiness means it is impossible to access a lot of old data (for example, `git` committers and commit messages). Note that it is impossible to make a 1-to-1 mapping between sloppy UTF-8 and sloppy UTF-16, so if all strings have a single representation (unless it is some weird representation not yet mentioned), it is either impossible to support all strings encountered on non-Windows platforms, or impossible to support all strings encountered on non-Windows platforms. I am a strong proponent of APIs supporting multiple compile-time-known string representations, with transparent conversion where safe. * ownership. Cases worth supporting: Value (but finite size), Refcounted (yes, this is important, the problem with std::string is that it was mutable), Tail or full Slice thereof, borrowed Zero-terminated or X(not) terminated, Literal (known statically allocated), and Alternating (the one that does SSO, switching between Value and Refcounted; IME it is important for Refcounted to efficiently support Literal). Note that there is no need for an immutable string to support Unique ownership. That's 8 different ownership policies, and if your Z/X implementation doesn't support recovering an optional owner, you also need to support at least one "maybe owned, maybe borrowed" (needed for things like efficient map insertion if the key might already exist; making the insert function a template does not suffice to handle all cases). It is important that, to the extent possible, all these (immutable) strings offer the same API, and can be converted implicitly where safe (exception: legacy code might make implicit conversion to V useful, despite being technically wrong). (there should be some Mutable string-like thing but it need not provide the API, only push/pop off the end followed by conversion to R; consider particularly the implementation of comma-separated list stringification) * domain meaning. The language itself should support at least Format strings for printf/scanf/strftime etc. (these "should" be separate types, but if relying on the C compiler to check them for you, don't actually have to be). Common library-supported additions include XML, JSON, and SQL strings, to make injection attacks impossible at the type level. Thinking of compilers (but not limited to them), there also needs to be dedicated types for "string representing a filepath" vs "string representing a file's contents" (the web's concept of "blob URL" is informative, but suffers from overloading the string type in the first place). Importantly, it must be easy to write literals of the appropriate type and convert explicitly as needed, so there should not be any file-related APIs that take strings. (related, it's often useful to have the concept of "single-line string" and "word" (which, among other cases, makes it possible to ); the exact definition thereof depending on context. So it may be useful to be able to tag strings as "all characters (or, separately, the first character or the last character (though "last character" is far less useful)) are one of [abc...]"; reasonably granularity being: NUL, whitespace characters individually, other C0 controls, ASCII symbols individually, digit 0, digit 1, digits 2-7, digits 8-9, letters a-f, letters A-F, letters g-z, letters G-Z, DEL, C1 controls, other latin1, U+0100 through U+07FF, U+0800 through U+FFFF excluding surrogates, low surrogates, high surrogates, and U+10000 through U+10FFFF, and illegal values U+110000 and higher (maybe splitting 31-bit from 32-bit?) (several of these are impossible under certain encodings and strictness levels). Actually supporting all of this in the compiler proper is complicated and likely to be deferred, but thinking about it informs both language and library design. Particularly, consider "partial template casting")
chubot 3y ago
The short answer is somewhere between Go and Rust strings, which are newer languages that use UTF-8 for interior representation, and also favor it for exterior encoding. Roughly speaking, Java and JavaScript are in the UTF-16 camp, and Python 2 and 3 are in the code points camp. C and C++ have unique problems, but you could also put them in the code points camp. So there are at least 3 different camps, and a whole bunch of weird variations, like string width being compile-time selectable in interpreters. More details here: https://www.oilshell.org/blog/2023/06/ysh-design.html#text https://www.oilshell.org/blog/2023/06/ysh-design.html#text and here: https://www.oilshell.org/blog/2023/06/surrogate-pair.html#history-windows-infected-javascript-json-and-python https://www.oilshell.org/blog/2023/06/surrogate-pair.html#hi... A main design issue is that string APIs shouldn't depend on a mutable global variable -- the default encoding, or default file system encoding. That's an idea that's a disaster in C, and also a disaster in Python. It leads to buggy programs. Go and Rust differ in their philosophies, but neither of them has that design problem.
librasteve 3y ago
Raku introduced the concept of NFG - Normal Form Grapheme - as a way to represent any Unicode string in its logical ‘visual character’ grapheme form. Sequences of combining characters that don’t have a canonical single codepoint form are given a synthetic codepoint so that string methods including regexes can operate on grapheme characters without ever causing splitting side effects. Of course there are methods for manipulating at the codepoint level as well.

k__ 3y ago

Languages have trends too. Weak typing/Implicit conversion was cool in the 90s. JS and PHP would have magnitudes better if they were typed like Python, but that's history.

deleted 3y ago
[deleted]
IggleSniggle 3y ago
I can appreciate why it was cool. "What if everything could automatically do the right thing when it interacts with other things, and you wouldn't need any of this ritual boilerplate stuff? Why do we keep needing to convert from string to number to string over and over and over it's crazy. The language should just do the right thing!" I think it's very easy to be sympathetic to the design trend.
- k__ 3y ago
  Yes. Then there is the fact that PHP and JavaScript were basically domain specific languages at their inception. PHP was a hypertext pre-processor. Pearl, but stripped down for templating. JavaScript was an alternative to Java applets. When you don't need the full power of Java, you could throw in a few small scripts. To make things a bit less intimidating, the weak typing seemed like a good idea, and for these use-cases it might very well have been. If you don't know how to program and only want to display some fancy styled text, why would you need to understand that a number isn't a string? But there was the pattern, that "real" programming languages like Java and C/C++ were hard to learn for the average "creator", so they took JS and PHP as entry and built up their skills from there, building bigger and bigger solutions with those tools.
chubot 3y ago
There are definitely trends, but the pattern I see with early JS and PHP is they simply didn't anticipate the consequences of certain decisions. What happens is you make a decision "locally", for some use case or program. And then you don't realize how it affects other programs. The well known "Wat" talk about JS by Gary B. is basically a bunch of these unintended consequences. So the problem is really to see the whole picture, and to get some experience using the language. But there's that catch 22 -- by the time you release it, it's almost set in stone. I remember in the early 2000's Guido and the Python team would always refer to "Python 3000" -- the one chance they had to break things!

matheusmoreira 3y ago

Yeah, it's extremely difficult. Everything interacts with everything else. A recent snag I've run into recently with my language: I want to have mutable strings and lists but that complicates their use as hash table keys. I'm so used to the C mindset of just modifying everything in place. It wasn't until I actually tried this that I understood why so many languages just copy everything. I researched how other languages dealt with this problem and many only allow immutable objects as hash keys. Most interesting was Ruby which allows mutable objects as keys, warns programmers that modifying keys can invalidate the hash table and provides a rehashing method to fix it. https://ruby-doc.org/core/Hash.html https://ruby-doc.org/core/Hash.html > Modifying a Hash key while it is in use damages the hash’s index. > You can repair the hash index using method rehash > A String key is always safe > That’s because an unfrozen String passed as a key will be replaced by a duplicated and frozen String I like Ruby a lot.

hardware2win 3y ago

If lang design is so insanely hard, then how C# designers get so many things right?

bmoxb 3y ago
I haven't used C# enough to comment on the language itself, but I do know it was designed (at least initially) by Anders Hejlsberg who is very experienced in language design.
chubot 3y ago
If basketball is so hard, then how did Michael Jordan score so many points?
- chubot 3y ago
  Also I pointed out an issue in my comment that C# got wrong, then fixed -- closures created in loops
  - hardware2win 3y ago
    I didnt say that C# is perfect But it feels like there is at least order or two of magnitude of difference. Especially if you start comparing std libs.