4 ms·

Delphi/Pascal had the best strings Single byte storage, so it works well with utf-8. Byte length field, so it knows the length of every string and it can stor

Delphi/Pascal had the best strings Single byte storage, so it works well with utf-8. Byte length field, so it knows the length of every string and it can store null bytes. An additional null byte past the end and pointer shifting, so every Pascal string is also a C pchar string. Reference-counted copy-on-write, so you can copy it in constant time for reading and treat it as if the string was fully copied for writing. Low-level memory management, so it can store 2 GB long strings. And in most situations you do not need a string builder, because strings are freed immediately and very fast when the refcounts zeros without staying around as garbage. Behaves as a value type with the null pointer being the empty string, so you can never get a null pointer exception. Any string operation returns always a valid string. (Optionally) index checked with the length, so you can never get a buffer overflow. Unfortunately the newer FreePascal/Delphi versions made it all very confusing by adding an encoding field. In the past you could just assume all strings are UTF-8 as code style rule. Now each string has an individual encoding. It is converted automatically from one encoding, but not always, so in any function that needs to access the characters, you kind of need to check if it is called with an utf-8 string, or latin1 string, or some other encoding.

vintagedave 8y ago
> Unfortunately the newer FreePascal/Delphi versions made it all very confusing by adding an encoding field I can clear this up. This change is really useful and important if you are using strings. A string of bytes has attached metadata saying what it is. Is it ANSI of some sort, or UTF8, or...? Is it a specific encoding, such as Windows-1252? Without that data, all you have are bytes, and you don't know how to interpret them. Thus, RawByteString (bytes); UTF8String (UTF8); and ANSI strings with the encoding, plus UnicodeString which is native Unicode on whichever platform (eg, on Windows it matches Windows UTF16.) This data is essential to convert to and from different string types. I don't know where conversion does "not always" happen - can't think of anywhere offhand. But if you ever run into issues, there are RTL functions for conversion. Check out the TEncoding class: http://docwiki.embarcadero.com/Libraries/Tokyo/en/System.SysUtils.TEncoding http://docwiki.embarcadero.com/Libraries/Tokyo/en/System.Sys... > In the past you could just assume all strings are UTF-8 as code style rule. This was an incorrect assumption, because before the encoding metadata was added, you would have been using an AnsiString there, and that, by definition ('Ansi'), is not UTF8. These days, if you have a UTF8 string, you can place it in a UTF8String type. Correctness enforced by libraries is much better than a coding convention that a certain type contains a subtly different payload. That way lies horror. Metadata and strong typing is much safer. -- I do agree with you that Delphi has the best strings :) Copy on write and embedded length both seem a real win, after twenty years of use, not to mention great string twiddling methods. I'm looking at adding string_view support to the strings currently (for C++17 support); one thing it highlights is how much more powerful the inbuilt String types are, and how much string_view is a workaround for a problem in C++'s string design which other string libraries - not just ours, but ours is IMO very good - do not suffer from.
jchb 8y ago
So now Swift has the best strings :) Or at least will have, when these changes are released. > Single byte storage, so it works well with utf-8. Check, underlying storage is a byte array with the utf-8 encoding. > Byte length field, so it knows the length of every string and it can store null bytes. Check. > An additional null byte past the end and pointer shifting, so every Pascal string is also a C pchar string. Check, getting a C string from a Swift string is O(1). > Reference-counted copy-on-write, so you can copy it in constant time for reading and treat it as if the string was fully copied for writing. Check. > Low-level memory management, so it can store 2 GB long strings. Check, just tested with a 4 GB string on macOS. > And in most situations you do not need a string builder, because strings are freed immediately and very fast when the refcounts zeros without staying around as garbage. Check, Swift objects are reference counted, storage of intermediary strings will be freed as soon as they go out of scope. > Behaves as a value type with the null pointer being the empty string, so you can never get a null pointer exception. Check, String is a value type. > (Optionally) index checked with the length, so you can never get a buffer overflow. Check, this is the default - trap on out-of-bounds access rather than an illegal memory access. If you really want can disable by compiling with -Ounchecked. > Unfortunately the newer FreePascal/Delphi versions made it all very confusing by adding an encoding field. In the past you could just assume all strings are UTF-8 as code style rule Check. String operations are performed on grapheme clusters (rather than for example UTF-8 code points), which is generally the right level of abstraction to work with. There are "views" for accessing specific encodings.
masklinn 8y ago
> Delphi/Pascal had the best strings Delphi/Pascal had fucking terrible string. > Single byte storage, so it works well with utf-8. No awareness of encoding, so easy to break your text if you try to do anything other than pipe it through. > Byte length field, so it knows the length of every string and it can store null bytes. Single-byte length field, so your strings can't exceed 255 bytes, and you need a pointer indirection just to get the length of your string. > An additional null byte past the end and pointer shifting, so every Pascal string is also a C pchar string. Scratch that, 254. > Reference-counted copy-on-write, so you can copy it in constant time for reading and treat it as if the string was fully copied for writing. Rc'd cow, so you never know whether your write is going to be free or incur a complete copy. > Behaves as a value type with the null pointer being the empty string, so you can never get a null pointer exception. Any string operation returns always a valid string. The null pointer is the empty string, you don't get to know whether you got no input or got an empty input. > (Optionally) index checked with the length, so you can never get a buffer overflow. (Optionally) you're completely unable to implement generic string operations because the string type carries a length and the language gives no way to be generic over that.
- vintagedave 8y ago
  With all due respect, > No awareness of encoding > Single-byte length field > Scratch that, 254. is completely wrong. Or rather, hasn't been right for several decades. A modern Delphi String has unlimited length (well, 32-bit value, a multi-gigabyte string is okay for 1996, I think), carries encoding (see my comment in this same thread), etc. COW is another argument, but one that seems to have won out over time. Many string implementations use many tricks to achieve something similar, or go with COW directly.
- benibela 8y ago
  Now you are confusing everything. Delphi has three single byte string types since 1996: (ansi)string, shortstring and string[length], and the last two are mostly there for backward compatibility. >No awareness of encoding, so easy to break your text if you try to do anything other than pipe it through. Which is good, when you have data with an unknown encoding. Unix file names for example, you can create a file that has an invalid utf-8 file name, and many tools that want to store the file name in a string with an encoding, just cannot access that file >Single-byte length field, so your strings can't exceed 255 bytes, and you need a pointer indirection just to get the length of your string. Ansistrings have a 32-bit length. I have not used Delphi on a 64 bit system, but in FreePascal they have a 64-bit length there. shortstrings have a 255 max length, but since they are short they are mostly stored on the stack and you do not need any pointer indirection. >Rc'd cow, so you never know whether your write is going to be free or incur a complete copy. Shortstrings are not rc'd cow and are always copied. Since they are short and on the stack that copying is fast. No write on long strings is free, since you can get a cache miss. And you only make a copy if some other function still has a reference string. How could that happen? The function should only keep the string around if it still needs the old value, and then it would need to make a copy anyways. No cow always leads to more copying. >The null pointer is the empty string, you don't get to know whether you got no input or got an empty input. If you want no input, you can use a pointer to a string. >(Optionally) you're completely unable to implement generic string operations because the string type carries a length and the language gives no way to be generic over that. Ansistrings do not have a length in the type, nor the basic shortstring. Nowadays you would define all string operations for ansistrings (=utf8strings).

vintagedave 8y ago

> Unfortunately the newer FreePascal/Delphi versions made it all very confusing by adding an encoding field I can clear this up. This change is really useful and important if you are using strings. A string of bytes has attached metadata saying what it is. Is it ANSI of some sort, or UTF8, or...? Is it a specific encoding, such as Windows-1252? Without that data, all you have are bytes, and you don't know how to interpret them. Thus, RawByteString (bytes); UTF8String (UTF8); and ANSI strings with the encoding, plus UnicodeString which is native Unicode on whichever platform (eg, on Windows it matches Windows UTF16.) This data is essential to convert to and from different string types. I don't know where conversion does "not always" happen - can't think of anywhere offhand. But if you ever run into issues, there are RTL functions for conversion. Check out the TEncoding class: http://docwiki.embarcadero.com/Libraries/Tokyo/en/System.SysUtils.TEncoding http://docwiki.embarcadero.com/Libraries/Tokyo/en/System.Sys... > In the past you could just assume all strings are UTF-8 as code style rule. This was an incorrect assumption, because before the encoding metadata was added, you would have been using an AnsiString there, and that, by definition ('Ansi'), is not UTF8. These days, if you have a UTF8 string, you can place it in a UTF8String type. Correctness enforced by libraries is much better than a coding convention that a certain type contains a subtly different payload. That way lies horror. Metadata and strong typing is much safer. -- I do agree with you that Delphi has the best strings :) Copy on write and embedded length both seem a real win, after twenty years of use, not to mention great string twiddling methods. I'm looking at adding string_view support to the strings currently (for C++17 support); one thing it highlights is how much more powerful the inbuilt String types are, and how much string_view is a workaround for a problem in C++'s string design which other string libraries - not just ours, but ours is IMO very good - do not suffer from.

jchb 8y ago

So now Swift has the best strings :) Or at least will have, when these changes are released. > Single byte storage, so it works well with utf-8. Check, underlying storage is a byte array with the utf-8 encoding. > Byte length field, so it knows the length of every string and it can store null bytes. Check. > An additional null byte past the end and pointer shifting, so every Pascal string is also a C pchar string. Check, getting a C string from a Swift string is O(1). > Reference-counted copy-on-write, so you can copy it in constant time for reading and treat it as if the string was fully copied for writing. Check. > Low-level memory management, so it can store 2 GB long strings. Check, just tested with a 4 GB string on macOS. > And in most situations you do not need a string builder, because strings are freed immediately and very fast when the refcounts zeros without staying around as garbage. Check, Swift objects are reference counted, storage of intermediary strings will be freed as soon as they go out of scope. > Behaves as a value type with the null pointer being the empty string, so you can never get a null pointer exception. Check, String is a value type. > (Optionally) index checked with the length, so you can never get a buffer overflow. Check, this is the default - trap on out-of-bounds access rather than an illegal memory access. If you really want can disable by compiling with -Ounchecked. > Unfortunately the newer FreePascal/Delphi versions made it all very confusing by adding an encoding field. In the past you could just assume all strings are UTF-8 as code style rule Check. String operations are performed on grapheme clusters (rather than for example UTF-8 code points), which is generally the right level of abstraction to work with. There are "views" for accessing specific encodings.

masklinn 8y ago

> Delphi/Pascal had the best strings Delphi/Pascal had fucking terrible string. > Single byte storage, so it works well with utf-8. No awareness of encoding, so easy to break your text if you try to do anything other than pipe it through. > Byte length field, so it knows the length of every string and it can store null bytes. Single-byte length field, so your strings can't exceed 255 bytes, and you need a pointer indirection just to get the length of your string. > An additional null byte past the end and pointer shifting, so every Pascal string is also a C pchar string. Scratch that, 254. > Reference-counted copy-on-write, so you can copy it in constant time for reading and treat it as if the string was fully copied for writing. Rc'd cow, so you never know whether your write is going to be free or incur a complete copy. > Behaves as a value type with the null pointer being the empty string, so you can never get a null pointer exception. Any string operation returns always a valid string. The null pointer is the empty string, you don't get to know whether you got no input or got an empty input. > (Optionally) index checked with the length, so you can never get a buffer overflow. (Optionally) you're completely unable to implement generic string operations because the string type carries a length and the language gives no way to be generic over that.

vintagedave 8y ago
With all due respect, > No awareness of encoding > Single-byte length field > Scratch that, 254. is completely wrong. Or rather, hasn't been right for several decades. A modern Delphi String has unlimited length (well, 32-bit value, a multi-gigabyte string is okay for 1996, I think), carries encoding (see my comment in this same thread), etc. COW is another argument, but one that seems to have won out over time. Many string implementations use many tricks to achieve something similar, or go with COW directly.
benibela 8y ago
Now you are confusing everything. Delphi has three single byte string types since 1996: (ansi)string, shortstring and string[length], and the last two are mostly there for backward compatibility. >No awareness of encoding, so easy to break your text if you try to do anything other than pipe it through. Which is good, when you have data with an unknown encoding. Unix file names for example, you can create a file that has an invalid utf-8 file name, and many tools that want to store the file name in a string with an encoding, just cannot access that file >Single-byte length field, so your strings can't exceed 255 bytes, and you need a pointer indirection just to get the length of your string. Ansistrings have a 32-bit length. I have not used Delphi on a 64 bit system, but in FreePascal they have a 64-bit length there. shortstrings have a 255 max length, but since they are short they are mostly stored on the stack and you do not need any pointer indirection. >Rc'd cow, so you never know whether your write is going to be free or incur a complete copy. Shortstrings are not rc'd cow and are always copied. Since they are short and on the stack that copying is fast. No write on long strings is free, since you can get a cache miss. And you only make a copy if some other function still has a reference string. How could that happen? The function should only keep the string around if it still needs the old value, and then it would need to make a copy anyways. No cow always leads to more copying. >The null pointer is the empty string, you don't get to know whether you got no input or got an empty input. If you want no input, you can use a pointer to a string. >(Optionally) you're completely unable to implement generic string operations because the string type carries a length and the language gives no way to be generic over that. Ansistrings do not have a length in the type, nor the basic shortstring. Nowadays you would define all string operations for ansistrings (=utf8strings).