10 ms·

The old MacOS (pre-X) did just that. Strings were all "Pascal strings", ie. with the first byte containing the length of the actual string. Building blocks for

The old MacOS (pre-X) did just that. Strings were all "Pascal strings", ie. with the first byte containing the length of the actual string. Building blocks for memory were also very different from stdlib, notably the use of Handles, which were pointers of pointers, so that the OS could move a block of data around to defragment the heap behind your back without breaking the memory addressing.

tialaramex 4y ago
Pascal strings are also kind of bad though. All sub-string operations need allocation, or have to be defined with intermediate results which aren't "really" strings, so in that sense it's not an improvement on Zero-terminated strings. Equality tests are cheaper which is nice, since strings of different lengths compare unequal immediately, but most things aren't really improved. C++ string_view is closer to the Right Thing™ - a slice, but C++ doesn't (yet) define anywhere what the encoding is, so... that's not what it could be. Rust's str is a slice and it's defined as UTF-8 encoded.
- WalterBright 4y ago
  D's strings were defined to be UTF-8 back in 2000. wstring is UTF-16, and dstring is UTF-32. Back then it wasn't clear which encoding method would turn out to be dominant, so we did all three. (Java was built on UTF-16.) As it eventually became clear, UTF-8 is da winnah, and the other formats are sideshows. Windows, which uses UTF-16, is handled by converting UTF-8 to -16 just before calling a Windows function, and converting anything coming back to UTF-8. D doesn't distinguish between a string and a string view.
  - saagarjha 4y ago
    What’s the ownership story for string views?
    - eco 4y ago
      They don't own anything. It's just a pointer and length. They don't allocate/deallocate.
      - saagarjha 4y ago
        I mean clearly something needs to own the buffer for a new string.
        tialaramex 3y ago
        Sure, but that's not the string_view's problem, you can't just make string_views, the string you want to borrow a view into needs to exist first. Imagine you go to a library and insist on borrowing "My Cousin Rachel", but they don't have it. "Oh I don't care whether you have the book, I just want to borrow it" is clearly nonsense. If they don't have it, you can't borrow it.
        saagarjha 3y ago
        Walter is talking about D, and he said this: > D doesn't distinguish between a string and a string view. In C++ std::string owns the buffer and std::string_view borrows it. If there is no difference between the two in D, then how is this difference bridged?
    - WalterBright 4y ago
      You can use automatic memory management and not worry about it. Or you can use D's prototype ownership/borrowing system. Or you can encapsulate them in something that manages the memory. Or you can do ownership/borrowing by convention (it's not hard to do).
      - saagarjha 4y ago
        Automatic memory management makes copies?
  - eco 4y ago
    A lot of people don't know about this but Microsoft is taking steps to move everything over to utf-8. They added a setting in Windows 10 to switch the code page over to utf-8 and then in Windows 11 they made it on by default. Individual applications can turn it on for themselves so they don't need to rely on the system setting being checked. With that you can, in theory, just use the -A variants of the winapi with utf-8 strings. I haven't tried it out yet as we still support prior Windows releases but it's nice that Microsoft has found a way out from the utf-16 mess.
    - WalterBright 4y ago
      The A-variants had problems years ago, which is why D abandoned them in favor of the W versions. I don't mind seeing UTF-16 fade away. We've been considering scaling back the D support for UTF-16/32 in the runtime library, in favor of just using converters as necessary. We recommend using UTF-8 as much as practical.
WalterBright 4y ago
> with the first byte containing the length of the actual string And the wheels fall off with the first string longer than 255 characters.
- initplus 4y ago
  You can fix this issue by using a variable width integer encoding for the size.
  - burntsushi 4y ago
    It might fix that particular issue, but you still have the same problem that NUL terminated strings have: it's not possible to cheaply create views/slices of a string using the same type.
- mikewarot 4y ago
  Which is why Free Pascal strings are so awesome. I've personally stuffed a billion bytes on one, without issues. They are automatic reference counted, and as close to magic as you can get. You can return one from a function without issue. However, Free Pascal has the worst documentation of any major project I've ever encountered (The exact opposite of Turbo Pascal), so I can't link to a good reference. Their Wiki is a black hole of nuance and sucks all useful stuff off the internet.
marssaxman 4y ago
I remember that era well! During the first few years I used C, I never touched its standard library at all, using the Mac Toolbox instead. This was a common practice, which later carried over into C++.

tialaramex 4y ago

Pascal strings are also kind of bad though. All sub-string operations need allocation, or have to be defined with intermediate results which aren't "really" strings, so in that sense it's not an improvement on Zero-terminated strings. Equality tests are cheaper which is nice, since strings of different lengths compare unequal immediately, but most things aren't really improved. C++ string_view is closer to the Right Thing™ - a slice, but C++ doesn't (yet) define anywhere what the encoding is, so... that's not what it could be. Rust's str is a slice and it's defined as UTF-8 encoded.

WalterBright 4y ago
D's strings were defined to be UTF-8 back in 2000. wstring is UTF-16, and dstring is UTF-32. Back then it wasn't clear which encoding method would turn out to be dominant, so we did all three. (Java was built on UTF-16.) As it eventually became clear, UTF-8 is da winnah, and the other formats are sideshows. Windows, which uses UTF-16, is handled by converting UTF-8 to -16 just before calling a Windows function, and converting anything coming back to UTF-8. D doesn't distinguish between a string and a string view.
- saagarjha 4y ago
  What’s the ownership story for string views?
  - eco 4y ago
    They don't own anything. It's just a pointer and length. They don't allocate/deallocate.
    - saagarjha 4y ago
      I mean clearly something needs to own the buffer for a new string.
      - tialaramex 3y ago
        Sure, but that's not the string_view's problem, you can't just make string_views, the string you want to borrow a view into needs to exist first. Imagine you go to a library and insist on borrowing "My Cousin Rachel", but they don't have it. "Oh I don't care whether you have the book, I just want to borrow it" is clearly nonsense. If they don't have it, you can't borrow it.
        saagarjha 3y ago
        Walter is talking about D, and he said this: > D doesn't distinguish between a string and a string view. In C++ std::string owns the buffer and std::string_view borrows it. If there is no difference between the two in D, then how is this difference bridged?
  - WalterBright 4y ago
    You can use automatic memory management and not worry about it. Or you can use D's prototype ownership/borrowing system. Or you can encapsulate them in something that manages the memory. Or you can do ownership/borrowing by convention (it's not hard to do).
    - saagarjha 4y ago
      Automatic memory management makes copies?
- eco 4y ago
  A lot of people don't know about this but Microsoft is taking steps to move everything over to utf-8. They added a setting in Windows 10 to switch the code page over to utf-8 and then in Windows 11 they made it on by default. Individual applications can turn it on for themselves so they don't need to rely on the system setting being checked. With that you can, in theory, just use the -A variants of the winapi with utf-8 strings. I haven't tried it out yet as we still support prior Windows releases but it's nice that Microsoft has found a way out from the utf-16 mess.
  - WalterBright 4y ago
    The A-variants had problems years ago, which is why D abandoned them in favor of the W versions. I don't mind seeing UTF-16 fade away. We've been considering scaling back the D support for UTF-16/32 in the runtime library, in favor of just using converters as necessary. We recommend using UTF-8 as much as practical.

WalterBright 4y ago

> with the first byte containing the length of the actual string And the wheels fall off with the first string longer than 255 characters.

initplus 4y ago
You can fix this issue by using a variable width integer encoding for the size.
- burntsushi 4y ago
  It might fix that particular issue, but you still have the same problem that NUL terminated strings have: it's not possible to cheaply create views/slices of a string using the same type.
mikewarot 4y ago
Which is why Free Pascal strings are so awesome. I've personally stuffed a billion bytes on one, without issues. They are automatic reference counted, and as close to magic as you can get. You can return one from a function without issue. However, Free Pascal has the worst documentation of any major project I've ever encountered (The exact opposite of Turbo Pascal), so I can't link to a good reference. Their Wiki is a black hole of nuance and sucks all useful stuff off the internet.

marssaxman 4y ago

I remember that era well! During the first few years I used C, I never touched its standard library at all, using the Mac Toolbox instead. This was a common practice, which later carried over into C++.