5 ms·

I have never written a single line of iOS/macos code, but I'm interested in Swift because it's a cool language. Can anyone explain why this is done now and not

I have never written a single line of iOS/macos code, but I'm interested in Swift because it's a cool language. Can anyone explain why this is done now and not when Swift was first released? I mean, UTF-8 was already the clear winner when Swift started. Is it some obj-c compat story?

saagarjha 8y ago
Objective-C strings are UTF-16, so it’s likely this was the case. The String ABI has diverged since then, but there is a need to finalize those changes now due to ABI stability.
- danpalmer 8y ago
  And worth noting, Objective-C’s strings (actually Foundation’s NSString) is UTF-16 because at the time it was implemented UTF-8 had become as universal as it is now. UTF-16 still seemed like a possible option. And changing the existing strings from 16-8 would have been a huge undertaking - Swift adds an interoperability layer and more flexible compiler/language that makes the transition much easier, not to mention a whole new standard library that has been designed to make this transition work well.
gok 8y ago
The commonly-accepted wisdom among Unicode people was that UTF-8 is good for transmission and storage, but bad for in memory representation. It now appears pretty conclusive that this was wrong.
- acdha 8y ago
  It's not just now — the memory-access argument has been known to be invalid since 1996 when Unicode 2.0 added surrogate pairs and broke the ability to use C-style indexing without having to decode the entire sequence. UTF-16 added overhead versus UTF-8 but it doesn't make the implementation any more efficient unless you don't care about correctness. Swift was created two decades later so the only justification which seems to make sense would be compatibility with Objective C and UTF-16 APIs.
- rstuart4133 8y ago
  They are still partially right, and I'd go so far as to say were totally right if efficiency wasn't a concern. From the programmers point of view utf-32 is still by far the easiest memory representation to use. Back in the day, when the Unicode people were pushing it as an alternative to whatever the ISO standard is that solved the same problem, they argued it was better than the ISO standard because the was only one encoding, UCS2 whereas ISO defined 8, 16 and 32 bit encodings. They said they could get away with that because 16 bits was enough for all modern languages (read: languages in use). Windows, Java and Javascript among others drank the kool-aid. But they were wrong, 16 bits was not enough. USC2 morphed into UTF-16, which is actually worse than UTF-8 because if has the byte ordering problem. To make matters worse, Unicode decided to abandon grapheme == code point. (This means there is ä and ä. They look identical, but one is '\xe4' and the other is a combining diaeresis, '\61\u0308', and the latter is now preferred.) I don't know why that did that as it makes string handling fragile, but I notice the encoding chosen for UTF-16 drastically reduced the number of code points available, so I wonder if they were worried about running out. The end result means that makes utf-32 actually harder to parse than utf-8 (which is quite an achievement), so you may as well use utf-8 for everything.

saagarjha 8y ago

Objective-C strings are UTF-16, so it’s likely this was the case. The String ABI has diverged since then, but there is a need to finalize those changes now due to ABI stability.

danpalmer 8y ago
And worth noting, Objective-C’s strings (actually Foundation’s NSString) is UTF-16 because at the time it was implemented UTF-8 had become as universal as it is now. UTF-16 still seemed like a possible option. And changing the existing strings from 16-8 would have been a huge undertaking - Swift adds an interoperability layer and more flexible compiler/language that makes the transition much easier, not to mention a whole new standard library that has been designed to make this transition work well.

gok 8y ago

The commonly-accepted wisdom among Unicode people was that UTF-8 is good for transmission and storage, but bad for in memory representation. It now appears pretty conclusive that this was wrong.

acdha 8y ago
It's not just now — the memory-access argument has been known to be invalid since 1996 when Unicode 2.0 added surrogate pairs and broke the ability to use C-style indexing without having to decode the entire sequence. UTF-16 added overhead versus UTF-8 but it doesn't make the implementation any more efficient unless you don't care about correctness. Swift was created two decades later so the only justification which seems to make sense would be compatibility with Objective C and UTF-16 APIs.
rstuart4133 8y ago
They are still partially right, and I'd go so far as to say were totally right if efficiency wasn't a concern. From the programmers point of view utf-32 is still by far the easiest memory representation to use. Back in the day, when the Unicode people were pushing it as an alternative to whatever the ISO standard is that solved the same problem, they argued it was better than the ISO standard because the was only one encoding, UCS2 whereas ISO defined 8, 16 and 32 bit encodings. They said they could get away with that because 16 bits was enough for all modern languages (read: languages in use). Windows, Java and Javascript among others drank the kool-aid. But they were wrong, 16 bits was not enough. USC2 morphed into UTF-16, which is actually worse than UTF-8 because if has the byte ordering problem. To make matters worse, Unicode decided to abandon grapheme == code point. (This means there is ä and ä. They look identical, but one is '\xe4' and the other is a combining diaeresis, '\61\u0308', and the latter is now preferred.) I don't know why that did that as it makes string handling fragile, but I notice the encoding chosen for UTF-16 drastically reduced the number of code points available, so I wonder if they were worried about running out. The end result means that makes utf-32 actually harder to parse than utf-8 (which is quite an achievement), so you may as well use utf-8 for everything.