9 ms·
Swift String’s ABI and UTF-8
- deleted 8y ago[deleted]
- ComputerGuru 8y agoWait. A programming language written this side of 2010 stored strings in a bastardized sometimes-ansii sometimes-utf16? blink Why on earth not just use utf8 from the start? Surely no micro-optimization made possible by their choice could be worth such a convoluted design?
- dep_b 8y agoProbably backwards compatibility with NSString
- Aqua_Geek 8y agoIt might be due to interop with Objective-C — Foundation's NSString uses UTF-16: https://developer.apple.com/documentation/foundation/nsstring https://developer.apple.com/documentation/foundation/nsstrin...
- ubernostrum 8y agoPython (as of version 3.3) does something similar. Source code is assumed UTF-8 by default, and all other string creation requires explicit conversion from bytes. So when creating and storing a string object, Python looks at the widest code point that will be in the string, and chooses the narrowest encoding -- latin-1, UCS-2, or UCS-4 -- that can store the string as fixed-width units. This gets all the benefits people usually want from "just use UTF-8", and then some -- strings containing only code points in the latin-1 range (not just the ASCII range) take one byte per code point -- and also keeps the property of code units being fixed width no matter what's in the string. Which means programmers don't have to deal with leaky abstractions that are all too common in languages that expose "Unicode" strings which are really byte sequences in some particular encoding of Unicode.
- rectang 8y agoThe tradeoff is that at unpredictable moments, memory requirements for string content can quadruple. Python is inexorably committed to the idioms which depend assume fixed width characters -- there's no persuading the community to use e.g. functions to obtain substrings rather than array indexes. So this is an understandable design decision.
- ubernostrum 8y agoassume fixed width characters Python strings are not iterables of characters. They're iterables of Unicode code points. This is why leaking the internal storage up to the programmer is problematic; prior to 3.3, you'd routinely see artifacts of the internal storage (like surrogate pairs) which broke the "strings are iterables of code points" abstraction. e.g. functions to obtain substrings rather than array indexes Strings are iterables of code points. Indexing into a string yields the code point at the requested index. While I'd like to have an abstraction for sequences of graphemes, strings-as-code-points is not the worst thing that a language could do. And all the "just use this thing that does exactly the same thing with a different name because I want indexing/length but also want to insist people don't call them that" is frankly pointless.
- rectang 8y ago> And all the "just use this thing that does exactly the same thing with a different name because I want indexing/length but also want to insist people don't call them that" is frankly pointless. Array index syntax over variable width data is problematic: either deceptively expensive -- O(n) for what looks like an O(1) operation -- or wrong. I suspect in that we agree. As for the alternative, I'm talking about Tom Christiansen's argument here: https://bugs.python.org/msg142041 https://bugs.python.org/msg142041 To paraphrase Tom's examples, this usage of array indexes is more idiomatic... s = "for finding the biggest of all the strings" x_at = s.index("big") y_at = s.index("the", x_at) some = s[x_at:y_at] print("GOT", some) ... than this, which the Python community would never adopt: import re s = "for finding the biggest of all the strings" some = re.search("(big.*?)the", s).group(1) print("GOT", some) The first involves more logical actions indexing into the string. If those operations are both O(1) and correct, then that's not a serious problem, which is what justifies the Python 3.3+ design. However, in terms of language design for handling Unicode strings, I prefer the tradeoffs of the second idiom: a single O(n) operation which is relatively easy to anticipate and plan for, rather than unpredictable memory blowups.
- josephg 8y agoThere's a huge performance benefit to using a complex string type like this in user-facing applications. The reason is that most strings are really small - like, a few bytes small. When strings are smaller than pointers, allocating them on the heap is silly and inefficient. So NSString uses a bunch of wacky encodings internally to pack very small strings into the 8 byte pointer that would otherwise be used to point to an object on the heap. I suspect that microoptimizations like this have done a lot of the work in making iOS outperform Android clock-for-clock. Although I suppose right now thats less of a big deal given how fast their A-series chips are as well.
- kevin_thibedeau 8y agoMost of iOS is native code. That's why it beats Android.
- saagarjha 8y ago> So NSString uses a bunch of wacky encodings internally to pack very small strings into the 8 byte pointer that would otherwise be used to point to an object on the heap. Nitpick: these pointers are odd, so they can't point to a valid object on the heap (malloc is 16-byte aligned on macOS). So it makes sense to reuse them by tagging them and packing a string (or number, or date) into the remaining bits to save on an allocation.
- ken 8y agoThe article says: Chinese text is "over 3x faster than before", and "ASCII benefits even more, despite the old model having a dedicated storage representation and fast paths for ASCII-only strings". That sounds like a pretty good reason to me.
- akvadrako 8y agoUTF-8 is very inefficient for some character sets - of course it shouldn't be the only option.
- masklinn 8y agoBecause Swift was built on the foundations (heh) of and to cheaply interact with Objective-C, and Obj-C is of that misguided generation of languages which believed UCS2 was a good idea, and later had to back-define it into UTF-16.
- iknowstuff 8y agoSwift's String implementation is just porn to me. They had a few misguided attempts in Swift 1 through 3, but their final design is truly marvelous and points programmers who have no idea about encodings towards the right solutions, like by simply counting grapheme clusters correctly, avoiding cloning thanks to views and not allowing for direct string[subscript] access without deliberately stepping down into utf8 or utf16 codepoint representations. The fact that Swift is aiming for ABI stability, something neither C++ nor Rust have because we all rely on C for FFI, is very interesting.
- eridius 8y agoUnfortunately, since Swift's String implementation operates on grapheme clusters by default, a lot of parsing code people write is actually subtly broken in the presence of combining characters. As a trivial example, let's say the input is a comma-delimited string (say, a line of CSV without quotes). The obvious way to split this is let fields = line.split(separator: ",") But given the input "foo,\u{301}bar" (which looks like foo,́bar), this won't split correctly and you'll end up with a single field that contains a comma. The correct way to split this is let fields = line.unicodeScalars.split(separator: ",").map(Substring.init) This will get you the correct 2 fields (at the cost of an intermediate array, as there is no lazy split).
- Twisell 8y agoHave you opened a ticket about that? This is more due to the split implementation than to the format itself. Room for improvement.
- deleted 8y ago[deleted]
- eridius 8y agoThis is not an issue with the split implementation. Split is behaving correctly (according to the defined semantics of String). You'd get the exact same issue if you said str.index(of: ",") instead; in both cases, the "," is a Character, not a UnicodeScalar, and "," != ",\u{301}".
- josephg 8y agoThis is really great to hear. The different internal string representations really age a language: - C sort of assumes all strings are ASCII - Java, C#, Obj-C, Javascript and Python2 were all written when it was assumed Unicode would have no more than 65536 characters. They all use 2 byte encodings, which has become the worst of all worlds. ASCII text is twice the size it needs to be, you still need to handle the mismatch between array length and codepoint length, and errors are harder to find because there are very few common characters that actually cross the 2 byte boundary - although the popularity of emoji has changed this. - Go, Dart, Rust and Zig all return to single byte encodings. They all use UTF-8 internally, and have different functions to interact with the string as a list of bytes, and the string as a list of unicode codepoints. (In Go's case its because Rob Pike was behind both Go and UTF8.) Its delightful to see Swift stepping away from Obj-C's mistakes and improving things internally. Although in Obj-C's defence, NSString has always hidden its internal representation, which did a lot of work to protect developers from the footguns. One of the big ironies of all this is how well C has stood the test of time. C strings interoperate beautifully with UTF8 - all thats missing is some libc helper methods to count & iterate through UTF8 codepoints. strlen / strncpy / strcmp / etc all work perfectly when dealing with UTF8. The only change is that you need to supply lengths in bytes not characters.
- paulddraper 8y ago> C strings interoperate beautifully with UTF8 C strings interoperate terribly with almost any encoding, including ASCII and UTF8. C strings cannot contain the NUL character.