4 ms·

String handling in Rust is definitely not as accessible as some other languages, but none of the complexity is incidental. The String/str split is actually fund

String handling in Rust is definitely not as accessible as some other languages, but none of the complexity is incidental. The String/str split is actually fundamental and pervasive, but internally all string operations are performed on these two types. Path represents a file system path, so it has a totally different API, and CString and OsString are for translating to other representations in external systems (libc and the OS, of course), not for use outside of that context. The String/str divide is the unavoidable consequence of not having garbage collection. A String is owned, a str is borrowed. This blog post goes into more information. This is nowhere near the chaos that C++ has.

eridius 11y ago
You can avoid the split without garbage collection, e.g. Swift has a single String type which is a retain-counted copy-on-write type. But this does mean that working with strings in Swift can have hidden costs (e.g. copies where you might not expect them), plus the cost of atomic reference counting. So this solution isn't appropriate for a language like Rust.
- tatterdemalion 11y ago
  You're right, I meant "with Rust's memory management story," which excludes universal reference counting. People differ on whether or not universal reference counting is a form of garbage collection with weak guarantees or a form of manual memory manage with flexible semantics.
  - pcwalton 11y ago
    > People differ on whether or not universal reference counting is a form of garbage collection with weak guarantees or a form of manual memory manage with flexible semantics. Which is pretty unfortunate, because the memory management field has always treated reference counting as a form of garbage collection—because it is [1] [2]. The idea that reference counting is somehow not GC is as far as I can tell a recent one (popularized, I think, by Cocoa and iOS?) This is not just a semantic quibble, because there are successful garbage collection techniques that combine tracing with reference counting (e.g. ulterior reference counting [3]) to achieve GC, so treating tracing as the only way to do GC makes no sense. In fact, David Bacon observed in 2004 that reference counting and tracing garbage collection are best viewed as just two extremes of a continuum [4], and many "optimizations" that you apply to one scheme or the other are really just moving the scheme you choose toward the opposite pole. [1]: http://www.memorymanagement.org/glossary/g.html#term-garbage-collection http://www.memorymanagement.org/glossary/g.html#term-garbage... [2]: http://researcher.watson.ibm.com/researcher/files/us-bacon/Bacon03Pure.pdf http://researcher.watson.ibm.com/researcher/files/us-bacon/B... [3]: http://www.cs.utexas.edu/users/mckinley/papers/urc-oopsla-2003.pdf http://www.cs.utexas.edu/users/mckinley/papers/urc-oopsla-20... [4]: https://www.cs.virginia.edu/~cs415/reading/bacon-garbage.pdf https://www.cs.virginia.edu/~cs415/reading/bacon-garbage.pdf
    - jurip 11y ago
      > The idea that reference counting is somehow not GC is as far as I can tell a recent one (popularized, I think, by Cocoa and iOS?) I remember seeing lots of "but Python doesn't have REAL GC" arguments years ago. But Cocoa's foray into GC, subsequent retreat back to manual ref counting and then ARC probably have affected this too, yeah.
      - pjmlp 11y ago
        On Apple's case it is even worse in terms of urban legends, because the anti-GC crowd thinks it is some kind of victory. What they miss is that RC is a form of GC algorithms and that Apple only gave up on Objective-C's GC, because they never managed to make it stable. The way one could link GC enabled with non GC enabled frameworks, plus the fragility of underlying C code for memory management meant Apple developer forums were full of crash reports. So Apple did a "Worse is Better" pivot and basically made the compiler automate the retain/release calls from Cocoa that were being written by hand. Similarly Swift adopts ARC, because it needs to interoperate with Objective-C's runtime and having a GC would introduce extra performance issues, or force them to use a RC cache mechanism like .NET has for COM. But of course, the majority without the background knowledge of how the decision came to be or interest in compiler design, just uses Apple's decision as some kind of proof against GC.
        pcwalton 11y ago
        I'm not really sure Apple gave up on tracing GC just because they couldn't make it stable. It also happened because they were concerned about pauses (especially when compared to Android at the time, which had a poor GC). That may or may not have been a valid concern; certainly the success of hybrid RC systems in the literature has shown that there is some merit to the idea that RC is useful to reduce pauses, though the situation is complex. The thing is, when viewed in the proper light Apple never introduced and abandoned GC. Rather, Objective-C always had GC, and the aborted "GC" attempt was actually an attempt to switch the GC algorithm.
        11y ago
  - pjmlp 11y ago
    According to the majority of CS books and papers about garbage collection, RC is yet another set of garbage collection algorithms.
- pcwalton 11y ago
  Reference counting is a form of garbage collection. Since the problem is readily solvable with any garbage collector (just make substrings point to the parent string), it's readily solvable with reference counting.
- deathanatos 11y ago
  > Swift has a single String type Which as someone who greatly appreciates the separation Rust offers, this makes me curious: does the string type either not enforce its contents being Unicode, or is it not capable of storing arbitrary OS paths? Because AFAICT, the two are mutually exclusive. The rub here is that OS paths — on POSIX operating systems¹ — are byte strings that we like to pretend are text; but since no particular encoding is enforced, you end up with effectively just byte strings. A String type that validates itself as valid Unicode is not capable of representing these (as arbitrary byte strings are not necessarily valid Unicode). But if we let the String type not validate, then I can pass non-Unicode to functions that take a "String" — something I'd rather not do. There's also Python's byte smuggling, but I find that a bit weird. ¹While I call out POSIX, I'm told Window's pathnames are sequences of unsigned 16-bit values that most people expect to be UTF-16 — but that fact isn't enforced by the OS.
  - millstone 11y ago
    The filesystem used on OS X and iOS is HFS+, which stores filenames as a normalized UTF-16. The POSIX C interface (open(2), etc) takes UTF-8. If you try to make a file with invalid UTF-8, the filesystem will silently munge it. For example, if your path has a trailing 0xc3 byte, that byte gets rewritten into the string "%c3". So you don't actually get arbitrary byte strings. I don't know what happens if you do something like mount an NFS share with filenames that aren't valid Unicode. I would guess a fair amount of stuff will break. Swift strings are Unicode, and Unicode is certainly capable of representing arbitrary byte streams, by just converting to and from any 8 bit fixed-width encoding (not UTF-8!). There are explicit conversion functions like -fileSystemRepresentation which handle the conversion between filesystem paths and Strings.
nly 11y ago
> This is nowhere near the chaos that C++ has. Standard C++ only has value-semantic C strings, and only supports fixed-width character encodings. It's not very rich, but I'd hardly call it 'chaos'. I'm happy to let my GUI library or DBMS handle collation and normalisation.

eridius 11y ago

You can avoid the split without garbage collection, e.g. Swift has a single String type which is a retain-counted copy-on-write type. But this does mean that working with strings in Swift can have hidden costs (e.g. copies where you might not expect them), plus the cost of atomic reference counting. So this solution isn't appropriate for a language like Rust.

tatterdemalion 11y ago
You're right, I meant "with Rust's memory management story," which excludes universal reference counting. People differ on whether or not universal reference counting is a form of garbage collection with weak guarantees or a form of manual memory manage with flexible semantics.
- pcwalton 11y ago
  > People differ on whether or not universal reference counting is a form of garbage collection with weak guarantees or a form of manual memory manage with flexible semantics. Which is pretty unfortunate, because the memory management field has always treated reference counting as a form of garbage collection—because it is [1] [2]. The idea that reference counting is somehow not GC is as far as I can tell a recent one (popularized, I think, by Cocoa and iOS?) This is not just a semantic quibble, because there are successful garbage collection techniques that combine tracing with reference counting (e.g. ulterior reference counting [3]) to achieve GC, so treating tracing as the only way to do GC makes no sense. In fact, David Bacon observed in 2004 that reference counting and tracing garbage collection are best viewed as just two extremes of a continuum [4], and many "optimizations" that you apply to one scheme or the other are really just moving the scheme you choose toward the opposite pole. [1]: http://www.memorymanagement.org/glossary/g.html#term-garbage-collection http://www.memorymanagement.org/glossary/g.html#term-garbage... [2]: http://researcher.watson.ibm.com/researcher/files/us-bacon/Bacon03Pure.pdf http://researcher.watson.ibm.com/researcher/files/us-bacon/B... [3]: http://www.cs.utexas.edu/users/mckinley/papers/urc-oopsla-2003.pdf http://www.cs.utexas.edu/users/mckinley/papers/urc-oopsla-20... [4]: https://www.cs.virginia.edu/~cs415/reading/bacon-garbage.pdf https://www.cs.virginia.edu/~cs415/reading/bacon-garbage.pdf
  - jurip 11y ago
    > The idea that reference counting is somehow not GC is as far as I can tell a recent one (popularized, I think, by Cocoa and iOS?) I remember seeing lots of "but Python doesn't have REAL GC" arguments years ago. But Cocoa's foray into GC, subsequent retreat back to manual ref counting and then ARC probably have affected this too, yeah.
    - pjmlp 11y ago
      On Apple's case it is even worse in terms of urban legends, because the anti-GC crowd thinks it is some kind of victory. What they miss is that RC is a form of GC algorithms and that Apple only gave up on Objective-C's GC, because they never managed to make it stable. The way one could link GC enabled with non GC enabled frameworks, plus the fragility of underlying C code for memory management meant Apple developer forums were full of crash reports. So Apple did a "Worse is Better" pivot and basically made the compiler automate the retain/release calls from Cocoa that were being written by hand. Similarly Swift adopts ARC, because it needs to interoperate with Objective-C's runtime and having a GC would introduce extra performance issues, or force them to use a RC cache mechanism like .NET has for COM. But of course, the majority without the background knowledge of how the decision came to be or interest in compiler design, just uses Apple's decision as some kind of proof against GC.
      - pcwalton 11y ago
        I'm not really sure Apple gave up on tracing GC just because they couldn't make it stable. It also happened because they were concerned about pauses (especially when compared to Android at the time, which had a poor GC). That may or may not have been a valid concern; certainly the success of hybrid RC systems in the literature has shown that there is some merit to the idea that RC is useful to reduce pauses, though the situation is complex. The thing is, when viewed in the proper light Apple never introduced and abandoned GC. Rather, Objective-C always had GC, and the aborted "GC" attempt was actually an attempt to switch the GC algorithm.
        11y ago
- pjmlp 11y ago
  According to the majority of CS books and papers about garbage collection, RC is yet another set of garbage collection algorithms.
pcwalton 11y ago
Reference counting is a form of garbage collection. Since the problem is readily solvable with any garbage collector (just make substrings point to the parent string), it's readily solvable with reference counting.
deathanatos 11y ago
> Swift has a single String type Which as someone who greatly appreciates the separation Rust offers, this makes me curious: does the string type either not enforce its contents being Unicode, or is it not capable of storing arbitrary OS paths? Because AFAICT, the two are mutually exclusive. The rub here is that OS paths — on POSIX operating systems¹ — are byte strings that we like to pretend are text; but since no particular encoding is enforced, you end up with effectively just byte strings. A String type that validates itself as valid Unicode is not capable of representing these (as arbitrary byte strings are not necessarily valid Unicode). But if we let the String type not validate, then I can pass non-Unicode to functions that take a "String" — something I'd rather not do. There's also Python's byte smuggling, but I find that a bit weird. ¹While I call out POSIX, I'm told Window's pathnames are sequences of unsigned 16-bit values that most people expect to be UTF-16 — but that fact isn't enforced by the OS.
- millstone 11y ago
  The filesystem used on OS X and iOS is HFS+, which stores filenames as a normalized UTF-16. The POSIX C interface (open(2), etc) takes UTF-8. If you try to make a file with invalid UTF-8, the filesystem will silently munge it. For example, if your path has a trailing 0xc3 byte, that byte gets rewritten into the string "%c3". So you don't actually get arbitrary byte strings. I don't know what happens if you do something like mount an NFS share with filenames that aren't valid Unicode. I would guess a fair amount of stuff will break. Swift strings are Unicode, and Unicode is certainly capable of representing arbitrary byte streams, by just converting to and from any 8 bit fixed-width encoding (not UTF-8!). There are explicit conversion functions like -fileSystemRepresentation which handle the conversion between filesystem paths and Strings.

nly 11y ago

> This is nowhere near the chaos that C++ has. Standard C++ only has value-semantic C strings, and only supports fixed-width character encodings. It's not very rich, but I'd hardly call it 'chaos'. I'm happy to let my GUI library or DBMS handle collation and normalisation.