10 ms·

Swift UTF-8 String

123 points by gok 8y ago

StefanKarpinski 8y ago
Seems like the right move. Julia made a similar transition in 1.0 for similar reasons and it has turned out very well.
saagarjha 8y ago
Great work by the Swift team on stabilizing this (especially around UTF-8) for Swift 5. One question: how do Objective-C's tagged NSString pointers fit into this? Are these imported as "opaque" strings? I see that _SmallString seems to store its characters inline; are there any plans to do tagged pointers in Swift as well?
- rkunde 8y ago
  It would have to be opaque as there is no backing allocation of any kind so the string can’t provide a pointer to the contiguous memory. (Plus tagged strings use an encoding scheme that makes the incompatible with UTF-8 anyway) It’s possible that a smallstring version of the tagged pointer is created in some situations. SmallString actually was just a tagged cocoa string at one point but that was removed in favor of more powerful implementations. The latest implementation allocates no memory on the heap either but can store more characters, so using a tagged pointer would be a step back.
- Someone 8y ago
  ”An opaque string is capable of handling all string operations through resilient function calls. […] Currently, these are used for lazily-bridged NSStrings that do not provide access to contiguous UTF-8 (i.e. ASCII) bytes in memory.” ⇒ it seems tagged NSString pointers indeed map to opaque strings. Because strings are structs in Swift, I don’t expect them to implement tagged pointers in Swift. Alternatively, you can see the SmallString struct as a tagged pointer that’s twice as large as a regular pointer.
ilovecaching 8y ago
Is anyone using Swift outside of Apple specific stuff professionally? When they first announced swift it felt like it might catch on as general purpose gced language. The language itself is quite beautiful in a lot of ways (although being OO was obviously an awful step backwards). But it feels like it's still just the crazy language Apple uses, just like Objective-C was.
- sooheon 8y ago
  Jeremy Howard and Chris Lattner are starting to push Swift for Tensorflow: - https://www.github.com/tensorflow/swift https://www.github.com/tensorflow/swift - https://www.fast.ai/2019/03/06/fastai-swift/ https://www.fast.ai/2019/03/06/fastai-swift/
  - pjmlp 8y ago
    They must push it really well for it to ever be considered an alternative to Julia, ML.NET, Python/C++, Deeplearning4j on Windows.
- bbrian 8y ago
  Google are beginning to use it for Tensorflow: https://www.youtube.com/watch?v=s65BigoMV_I https://www.youtube.com/watch?v=s65BigoMV_I There's a course currently running in University of San Francisco run by Fast.ai which is using Swift-Tensorflow taught by Chris Lattner for two lessons: https://www.usfca.edu/data-institute/certificates/deep-learning-part-two https://www.usfca.edu/data-institute/certificates/deep-learn... Not exactly what you asked, but pertinent.
  - ilovecaching 8y ago
    I don't expect that will do very well. ML is already crowded, and Python is already solidly at the top. Having worked in ML myself, no one would switch if it meant dealing with a real statically typed language they could hack away at easily in a repl or jupyter notebooks, or access mypy, numpy, scikit, etc. ML is an ecosystem of tools that are all in Python currently.
sonnyblarney 8y ago
If I ever wrote a language, handling strings wrt unicode would be the first thing I would fix so so that it's clear, concise and especially consistent. It bothers me that in 2019 it can still be difficult to manage even basic things with strings, and yet, whenever we talk/argue/bike-shed about 'languages' we get caught up in intellectual meandering. Please, give us strings that make sense. Then argue about monads.
- saagarjha 8y ago
  > give us strings that make sense This is really, really hard.
- asveikau 8y ago
  I don't think a lot of people realize that Unicode itself, in any representation, is complex, and there is no one abstraction that will make it always easy in all cases. So I think your goal is unobtainable. Some of that complexity is because world languages themselves are not simple. (eg. If you were to ditch Unicode and start from scratch, it would remain hard to do bi-di text, to pick a random example.)
  - sonnyblarney 8y ago
    Totally see what you are saying (the underlying issue is language itself), but it's pragmatically obtainable.
    - nicky0 8y ago
      The think is, strings that make sense has been a major goal of Swift, that's why they have been iterating on it so heavily. FWIW I think Swift's is probably one of the cleanest String abstractions out there with respect to unicode correctness balanced against efficiency and usability.
      - asveikau 8y ago
        To be clear I applaud successful attempts to make it better, what I doubt is a possibility to make the problem completely go away without users of the api ever having to think.
faragon 8y ago
Wise move, in my opinion. I hope Python will do that too, eventually.
- kevin_thibedeau 8y ago
  Python implemented multi-representation strings in 3.3. It picks the best encoding from ASCII, UTF8, UCS2, or UCS4. UTF8 is the preferred encoding for strings exposed to the C API. https://www.python.org/dev/peps/pep-0393/ https://www.python.org/dev/peps/pep-0393/
  - masklinn 8y ago
    CPython provides but does not use UTF8 internally, it's a cache for PyUnicode_AsUTF8 (formerly _PyUnicode_AsString), which exists to convert Python strings to char* in order to interoperate with C libraries. It is not the canonical representation of the string. pypy, on the other hand, made this change in the latest release (7.1.0): https://twitter.com/pypyproject/status/1095971192513708032 https://twitter.com/pypyproject/status/1095971192513708032
  - ken 8y ago
    The "kind" field is one of [uninitialized, Latin-1, UCS-2, UCS-4]. I'm not sure how it would pick UTF-8. (In memory usage, the Python way is strictly worse: adding one 4-byte character to an ASCII string forces the entire string to be upgraded to UCS-4. The advantage of the Python way is O(1) access to codepoints, but that's an operation that rarely comes up in practice, when dealing with Unicode strings.) I'm not sure how that would be compatible with Python, either. Either you're giving up string[index], or you're causing this one subscript operation to take O(n) time, or you need a new index type just for strings (which is what Swift does). None of these seem very 'Pythonic'. EDIT: Apparently recent versions of pypy solve this by computing their own index into your string, which seems like it would be terrible for memory usage, but I look forward to their blog post about it.
- sambe 8y ago
  What do you hope Python will do?
ezoe 8y ago
I wonder how long Windows can hold UTF-16 as the default native encodings. They recently added UTF-8 locale.
- ygra 8y ago
  CP65001 has existed for quite a while. There just have been plenty of places where it never really worked (I think the console was one of them). Nonetheless, UTF-16 will most likely stay. They might add wrappers around all API functions that convert arguments first, so we'd get CreateWindow8 alongside CreateWindowW and CreateWindowA, but I wouldn't hold my breath. The duality of API functions was to enable programs to be built for NT and 9x with the same codebase. It was never really intended for people to call them directly to enable non-Unicode-aware applications on NT. The file system cannot migrate from UTF-16 anyway without breaking things, so that will stay as is. IMHO there's pretty much nothing to gain, except a lot of risk in trying to change Windows to UTF-8 internally and the current interface might be an inconvenience to some developers, but not as much as to justify changing the whole OS.
  - repolfx 8y ago
    It's even harder than that - APIs have expected performance characteristics, and APIs that involve string manipulation can change their performance when needing to work directly with UTF-8, e.g. working with file paths might become slower.
  - vbezhenar 8y ago
    When the world uses UTF-8 and you have to constantly recode from/to with every WinAPI call, that's definitely an unnecessary overhead.
    - badestrand 8y ago
      Obviously it is unnecessary but I doubt it affects the perceived speed of the average desktop application in any way.
  - ChrisSD 8y ago
skrebbel 8y ago
I have never written a single line of iOS/macos code, but I'm interested in Swift because it's a cool language. Can anyone explain why this is done now and not when Swift was first released? I mean, UTF-8 was already the clear winner when Swift started. Is it some obj-c compat story?
- saagarjha 8y ago
  Objective-C strings are UTF-16, so it’s likely this was the case. The String ABI has diverged since then, but there is a need to finalize those changes now due to ABI stability.
  - danpalmer 8y ago
    And worth noting, Objective-C’s strings (actually Foundation’s NSString) is UTF-16 because at the time it was implemented UTF-8 had become as universal as it is now. UTF-16 still seemed like a possible option. And changing the existing strings from 16-8 would have been a huge undertaking - Swift adds an interoperability layer and more flexible compiler/language that makes the transition much easier, not to mention a whole new standard library that has been designed to make this transition work well.
- gok 8y ago
  The commonly-accepted wisdom among Unicode people was that UTF-8 is good for transmission and storage, but bad for in memory representation. It now appears pretty conclusive that this was wrong.
  - acdha 8y ago
    It's not just now — the memory-access argument has been known to be invalid since 1996 when Unicode 2.0 added surrogate pairs and broke the ability to use C-style indexing without having to decode the entire sequence. UTF-16 added overhead versus UTF-8 but it doesn't make the implementation any more efficient unless you don't care about correctness. Swift was created two decades later so the only justification which seems to make sense would be compatibility with Objective C and UTF-16 APIs.
garganzol 8y ago
UTF-16 allows to quickly index a char in a string covering the most languages that way. UTF-8 is a different story, as nearly every byte of UTF-8 string is a surrogate for most languages except English/ASCII. Now, how can a char be indexed then? Only by slow iteration from the beginning of a string? UTF-8 is a good choice for storage, true. However, it seems to be a not so good choice for string processing from the algorithmic point of view. Or maybe there is a catch that makes UTF-8 strings well-suited for string processing as well?
- WorldMaker 8y ago
  > UTF-16 allows to quickly index a char in a string covering the most languages that way. Nope. UTF-16 is not UCS-2 anymore for multiple reasons. You cannot assume you can index a UTF-16 string as if it was UCS-2 anymore. UCS-4 (32-bit) is a possible option, though not generally recommended either. There are multiple tricks for getting great performance in UTF-8 string processing such as codepoint maps of various sorts. It's quite well suited for string processing, and it's better to use UTF-8 at this point simply because it also helps from falling into the UTF-16 is not (and has not been for some time) UCS-2 trap, and UCS-2 cannot represent a lot of Unicode today (including many emoji).
  - garganzol 8y ago
    Never made a claim that UTF-16 is UCS-2. "Most languages" does not mean "all languages". Still, many popular platforms treat it like that in terms of char indexing. They are able to get away with it since a surrogate char is a rare guest in UTF-16. P.S. It's surprising how easily HN crowd downvotes something when it falls out of a whimsical "popular contemporary view on things". Having a lot of years of text processing behind my shoulders, I raised an important question and got an immediate down-vote attack. But never mind, I'll survive.
    - ygra 8y ago
      All those languages make it quite clear that you only get access to a code unit that way and other APIs usually exist to actually get access to code points, which then usually take a string and an index. That the most visible mechanism of accessing parts of a string is based on code units along with UTF-16 masking bad code unless Emoji are involved (for most developers at least), is a problem, though. UTF-8 makes the failure case pop up much faster as pretty much every language doesn't only use ASCII.

123 points by gok 8y ago

Seems like the right move. Julia made a similar transition in 1.0 for similar reasons and it has turned out very well.

Great work by the Swift team on stabilizing this (especially around UTF-8) for Swift 5. One question: how do Objective-C's tagged NSString pointers fit into this? Are these imported as "opaque" strings? I see that _SmallString seems to store its characters inline; are there any plans to do tagged pointers in Swift as well?

rkunde 8y ago
It would have to be opaque as there is no backing allocation of any kind so the string can’t provide a pointer to the contiguous memory. (Plus tagged strings use an encoding scheme that makes the incompatible with UTF-8 anyway) It’s possible that a smallstring version of the tagged pointer is created in some situations. SmallString actually was just a tagged cocoa string at one point but that was removed in favor of more powerful implementations. The latest implementation allocates no memory on the heap either but can store more characters, so using a tagged pointer would be a step back.
Someone 8y ago
”An opaque string is capable of handling all string operations through resilient function calls. […] Currently, these are used for lazily-bridged NSStrings that do not provide access to contiguous UTF-8 (i.e. ASCII) bytes in memory.” ⇒ it seems tagged NSString pointers indeed map to opaque strings. Because strings are structs in Swift, I don’t expect them to implement tagged pointers in Swift. Alternatively, you can see the SmallString struct as a tagged pointer that’s twice as large as a regular pointer.

ilovecaching 8y ago

Is anyone using Swift outside of Apple specific stuff professionally? When they first announced swift it felt like it might catch on as general purpose gced language. The language itself is quite beautiful in a lot of ways (although being OO was obviously an awful step backwards). But it feels like it's still just the crazy language Apple uses, just like Objective-C was.

sooheon 8y ago
Jeremy Howard and Chris Lattner are starting to push Swift for Tensorflow: - https://www.github.com/tensorflow/swift https://www.github.com/tensorflow/swift - https://www.fast.ai/2019/03/06/fastai-swift/ https://www.fast.ai/2019/03/06/fastai-swift/
- pjmlp 8y ago
  They must push it really well for it to ever be considered an alternative to Julia, ML.NET, Python/C++, Deeplearning4j on Windows.
bbrian 8y ago
Google are beginning to use it for Tensorflow: https://www.youtube.com/watch?v=s65BigoMV_I https://www.youtube.com/watch?v=s65BigoMV_I There's a course currently running in University of San Francisco run by Fast.ai which is using Swift-Tensorflow taught by Chris Lattner for two lessons: https://www.usfca.edu/data-institute/certificates/deep-learning-part-two https://www.usfca.edu/data-institute/certificates/deep-learn... Not exactly what you asked, but pertinent.
- ilovecaching 8y ago
  I don't expect that will do very well. ML is already crowded, and Python is already solidly at the top. Having worked in ML myself, no one would switch if it meant dealing with a real statically typed language they could hack away at easily in a repl or jupyter notebooks, or access mypy, numpy, scikit, etc. ML is an ecosystem of tools that are all in Python currently.

sonnyblarney 8y ago

If I ever wrote a language, handling strings wrt unicode would be the first thing I would fix so so that it's clear, concise and especially consistent. It bothers me that in 2019 it can still be difficult to manage even basic things with strings, and yet, whenever we talk/argue/bike-shed about 'languages' we get caught up in intellectual meandering. Please, give us strings that make sense. Then argue about monads.

saagarjha 8y ago
> give us strings that make sense This is really, really hard.
asveikau 8y ago
I don't think a lot of people realize that Unicode itself, in any representation, is complex, and there is no one abstraction that will make it always easy in all cases. So I think your goal is unobtainable. Some of that complexity is because world languages themselves are not simple. (eg. If you were to ditch Unicode and start from scratch, it would remain hard to do bi-di text, to pick a random example.)
- sonnyblarney 8y ago
  Totally see what you are saying (the underlying issue is language itself), but it's pragmatically obtainable.
  - nicky0 8y ago
    The think is, strings that make sense has been a major goal of Swift, that's why they have been iterating on it so heavily. FWIW I think Swift's is probably one of the cleanest String abstractions out there with respect to unicode correctness balanced against efficiency and usability.
    - asveikau 8y ago
      To be clear I applaud successful attempts to make it better, what I doubt is a possibility to make the problem completely go away without users of the api ever having to think.

faragon 8y ago

Wise move, in my opinion. I hope Python will do that too, eventually.

kevin_thibedeau 8y ago
Python implemented multi-representation strings in 3.3. It picks the best encoding from ASCII, UTF8, UCS2, or UCS4. UTF8 is the preferred encoding for strings exposed to the C API. https://www.python.org/dev/peps/pep-0393/ https://www.python.org/dev/peps/pep-0393/
- masklinn 8y ago
  CPython provides but does not use UTF8 internally, it's a cache for PyUnicode_AsUTF8 (formerly _PyUnicode_AsString), which exists to convert Python strings to char* in order to interoperate with C libraries. It is not the canonical representation of the string. pypy, on the other hand, made this change in the latest release (7.1.0): https://twitter.com/pypyproject/status/1095971192513708032 https://twitter.com/pypyproject/status/1095971192513708032
- ken 8y ago
  The "kind" field is one of [uninitialized, Latin-1, UCS-2, UCS-4]. I'm not sure how it would pick UTF-8. (In memory usage, the Python way is strictly worse: adding one 4-byte character to an ASCII string forces the entire string to be upgraded to UCS-4. The advantage of the Python way is O(1) access to codepoints, but that's an operation that rarely comes up in practice, when dealing with Unicode strings.) I'm not sure how that would be compatible with Python, either. Either you're giving up string[index], or you're causing this one subscript operation to take O(n) time, or you need a new index type just for strings (which is what Swift does). None of these seem very 'Pythonic'. EDIT: Apparently recent versions of pypy solve this by computing their own index into your string, which seems like it would be terrible for memory usage, but I look forward to their blog post about it.
sambe 8y ago
What do you hope Python will do?

ezoe 8y ago

I wonder how long Windows can hold UTF-16 as the default native encodings. They recently added UTF-8 locale.

ygra 8y ago
CP65001 has existed for quite a while. There just have been plenty of places where it never really worked (I think the console was one of them). Nonetheless, UTF-16 will most likely stay. They might add wrappers around all API functions that convert arguments first, so we'd get CreateWindow8 alongside CreateWindowW and CreateWindowA, but I wouldn't hold my breath. The duality of API functions was to enable programs to be built for NT and 9x with the same codebase. It was never really intended for people to call them directly to enable non-Unicode-aware applications on NT. The file system cannot migrate from UTF-16 anyway without breaking things, so that will stay as is. IMHO there's pretty much nothing to gain, except a lot of risk in trying to change Windows to UTF-8 internally and the current interface might be an inconvenience to some developers, but not as much as to justify changing the whole OS.
- repolfx 8y ago
  It's even harder than that - APIs have expected performance characteristics, and APIs that involve string manipulation can change their performance when needing to work directly with UTF-8, e.g. working with file paths might become slower.
- vbezhenar 8y ago
  When the world uses UTF-8 and you have to constantly recode from/to with every WinAPI call, that's definitely an unnecessary overhead.
  - badestrand 8y ago
    Obviously it is unnecessary but I doubt it affects the perceived speed of the average desktop application in any way.
- ChrisSD 8y ago

skrebbel 8y ago

I have never written a single line of iOS/macos code, but I'm interested in Swift because it's a cool language. Can anyone explain why this is done now and not when Swift was first released? I mean, UTF-8 was already the clear winner when Swift started. Is it some obj-c compat story?

saagarjha 8y ago
Objective-C strings are UTF-16, so it’s likely this was the case. The String ABI has diverged since then, but there is a need to finalize those changes now due to ABI stability.
- danpalmer 8y ago
  And worth noting, Objective-C’s strings (actually Foundation’s NSString) is UTF-16 because at the time it was implemented UTF-8 had become as universal as it is now. UTF-16 still seemed like a possible option. And changing the existing strings from 16-8 would have been a huge undertaking - Swift adds an interoperability layer and more flexible compiler/language that makes the transition much easier, not to mention a whole new standard library that has been designed to make this transition work well.
gok 8y ago
The commonly-accepted wisdom among Unicode people was that UTF-8 is good for transmission and storage, but bad for in memory representation. It now appears pretty conclusive that this was wrong.
- acdha 8y ago
  It's not just now — the memory-access argument has been known to be invalid since 1996 when Unicode 2.0 added surrogate pairs and broke the ability to use C-style indexing without having to decode the entire sequence. UTF-16 added overhead versus UTF-8 but it doesn't make the implementation any more efficient unless you don't care about correctness. Swift was created two decades later so the only justification which seems to make sense would be compatibility with Objective C and UTF-16 APIs.

garganzol 8y ago

UTF-16 allows to quickly index a char in a string covering the most languages that way. UTF-8 is a different story, as nearly every byte of UTF-8 string is a surrogate for most languages except English/ASCII. Now, how can a char be indexed then? Only by slow iteration from the beginning of a string? UTF-8 is a good choice for storage, true. However, it seems to be a not so good choice for string processing from the algorithmic point of view. Or maybe there is a catch that makes UTF-8 strings well-suited for string processing as well?

WorldMaker 8y ago
> UTF-16 allows to quickly index a char in a string covering the most languages that way. Nope. UTF-16 is not UCS-2 anymore for multiple reasons. You cannot assume you can index a UTF-16 string as if it was UCS-2 anymore. UCS-4 (32-bit) is a possible option, though not generally recommended either. There are multiple tricks for getting great performance in UTF-8 string processing such as codepoint maps of various sorts. It's quite well suited for string processing, and it's better to use UTF-8 at this point simply because it also helps from falling into the UTF-16 is not (and has not been for some time) UCS-2 trap, and UCS-2 cannot represent a lot of Unicode today (including many emoji).
- garganzol 8y ago
  Never made a claim that UTF-16 is UCS-2. "Most languages" does not mean "all languages". Still, many popular platforms treat it like that in terms of char indexing. They are able to get away with it since a surrogate char is a rare guest in UTF-16. P.S. It's surprising how easily HN crowd downvotes something when it falls out of a whimsical "popular contemporary view on things". Having a lot of years of text processing behind my shoulders, I raised an important question and got an immediate down-vote attack. But never mind, I'll survive.
  - ygra 8y ago
    All those languages make it quite clear that you only get access to a code unit that way and other APIs usually exist to actually get access to code points, which then usually take a string and an index. That the most visible mechanism of accessing parts of a string is based on code units along with UTF-16 masking bad code unless Emoji are involved (for most developers at least), is a problem, though. UTF-8 makes the failure case pop up much faster as pretty much every language doesn't only use ASCII.