22 ms·

Why Is Swift's String API So Hard?

97 points by chmaynard 11y ago

cballard 11y ago
> I think that having different conceptual kinds of strings be distinct types would eliminate a lot of bugs. I'm not aware of any language or standard library that does this, though. Foundation seems like the obvious one? NSURL vs. NSString?
- steveklabnik 11y ago
  In the Rust standard library, we have String, str, OsString, Path, and PathBuf, off the top of my head, with external crates implementing things like URLs, ropes, and others. That said, I completely feel what this blog post is saying: strings are _hard_, especially if you're not just doing ASCII.
  - x5n1 11y ago
    why are they so hard?
    - ben0x539 11y ago
      Because we spent a lot of time optimizing every assumption for the ascii-only case, and letting go of that simplification is distressing, I guess.
      - steveklabnik 11y ago
        Yes. And many languages had ASCII-only stuff for a long time, so "real" APIs seem much, much more complex. Of course, that complexity was always there, but us English speakers could mostly just ignore it...
    - Gankro 11y ago
      There's two aspects of complexity with strings: 1) 40 years of crazy encodings and languages. 2) human languages are wildly diverse and basically any assumption you wish to apply is broken. For 1, any system that wants to deal with the outside world needs to deal with: operating system encodings (arbitrary bytes on unix, malformed UCS2 on windows), C representation (null-terminated strings), systems that only work with ASCII, systems that only work with utf8, systems that work with arbitrary encodings/languages (HTML). This is arguably unnecessary complexity that exists because of short-sighted decisions in the past. 2 is the necessary complexity; the fact that languages are really complicated. There are thousands of symbols in writing. Do you try to encode these symbols in a monolithic manner, or in a compositional way? For historical reasons, you can often do both! ë can be a single character, or e with an accent modifier. How do you handle string searching in such a model? Do you match `noel` with `noël`? What's the length of noël? 4 characters? 5 characters? bytes? graphemes? codepoints? Can you correctly reverse noël (do it wrong and you can get leön)? Different letters which have similar/identical representations but different semantics/origins! Is Ε "capital e" or "capital ε"? How do you upper-case or lower-case these letters? Do you expect to_upper(to_lower(char)) to roundtrip (it won't)? Do you expect capitalization to be doable in-place (it's not)? Do you expect capitalization to be region-specific (it is)? Are any of these operations even coherent in a language like Japanese? Why are you trying to do them? God help you if you want to display this text. Are you ready to handle right-to-left text? Are you assuming that your font is monospace (hey there terminal and text editors)? C̢̫a̘̺̯n ̘̜̦̹y̷̫̼̘̩o̶͉u̗̩̻̞ ̻ẹ͡v̴̤͎̹e̶̫̠̤̭̺̤̞n̛̞̹̣̩̲͉̮ ̜͖̪͔̖d̤e̘̯ͅa̺l̟̀ ͚̗̣w̭i̸͇̠̥̣̜̥t̸h̸̻̮̼̙̹ ̗̺̱̣̰̱̙z̟a̺͜l̠̦̖̟̰͍g҉̜͖͓̫ơ̩̹̰͕?̹̳̼̯̘̺̟
      - mikeash 11y ago
        I think my favorite part of this article is how my Zalgo example doesn't render even remotely correctly in any browser I've tried it in.
  - Gankro 11y ago
    std provides String (utf8), OsString (bytes/wtf8), PathBuf (bytes/wtf8), CString (null-terminated). Each has their own unsized view: str, OsStr, Path, and CStr. We also provide AsciiExt for those times where you really truly believe you want to be working with a String as Ascii. That said, we generally try to make it as ergonomic as possible to pass a plain str where a Path/OsStr is expected. This is because utf8 is a subset of wtf8, so it's always fine to convert in that direction blindly (and it's really nice to just be like `File::open("foo.txt")` when hacking something together). This is why so many interfaces are riddled with something like `P: As<Path>`. The differentiation largely exists for the other direction, IMO. Paths and OsStrs aren't guaranteed to be valid UTF8, and shouldn't be provided where a proper utf8 string is expected. Path is just a convenience wrapper over OsStr that understands the platform's seperator conventions and provides convenient utilities.
  - erickt 11y ago
    Don't forget we kinda sorta have byte characters and strings a la `b'x'` and `b"foo"`, which are really just simpler ways of expressing byte slices. Unfortunately they lack string specific methods until we get specialization (fingers crossed).
- kazinator 11y ago
  Typing strings to solve bugs is the epitome of an anti-pattern. If you want type, yank the data out of the string representation and into an actual data type! "The string is a stark data structure and everywhere it is passed there is much duplication of process. It is a perfect vehicle for hiding information." -- Alan Perlis, Epigram #34.
  - jasode 11y ago
    >, yank the data out of the string representation Well, the very act of "yanking" can be helped by more well-defined behavior policies of "typed strings". It is unavoidable that the programmer will have to confront a contiguous piece of memory (string) with no boundaries between characters whether it's a block of bytes on disk, network stream, or the clipboard buffer. It can be helpful to differentiate the various types of strings with a formal type system before the yanking takes place. The Alan Perlis epigram is wise but I don't think it applies here. I think's he's talking about abusing strings as an adhoc data format by stuffing them with multi-value data and then turning right around and parsing them. For example, concatenating name+":"+id+":"+amount into a string with colons like "John:389432:$5.99", passing the "complex" string around, and then later parsing out the colons with string.split(). Basically, the programmer is making an in-memory csv (comma separated values) data format instead of using "C struct {}" or passing 3 separate formal parameters.
    - mannykannot 11y ago
      > ...before the yanking takes place. I take it that you are advocating for typed strings in addition to fully-formed types for things like URLs and SQL, but I think all that will lead to is a lot of gratuitous casting of strings just so they can be passed to the fully-formed types. Furthermore, few of these casts could be statically checked for validity.
      - jasode 11y ago
        >, but I think all that will lead to is a lot of gratuitous casting of strings But if programmers are not directly doing memcpy() of bytes directly to the stack frame, they are already doing a roundabout version of "casting". If a programmer has to "yank" from a string to put it into a more organized data type, the "yank" has to be done within the framework of semantics. If one "yanks" or "slices", or "substr", etc characters 3 to 15, the exact behavior depends on whether contiguous bytes is ASCII, UTF8, UTF16, or binary blob (which means NUL(0) is valid byte in the middle of the block of bytes). That "meaning overlayed onto those bytes to manipulate it correctly" has to decided somewhere. It is either implicit or explicit. Likewise, if one pulls 4 bytes from disk or network, the programmer has to assign meaning to it and say it is the 4-bytes of an IEEE floating point (float) or 4-bytes of a integer (int). The programmer may later want to do a bitshift (<<) which is not applicable to a float.
- fizx 11y ago
  /Regexp?/
- mikeash 11y ago
  Sort of. One spot of trouble is that NSString paths still exist and show up all over the place, even though NSURL is slowly replacing them. Another spot of trouble is that NSURL ends up in a similar situation, because sometimes NSURL means "an arbitrary URL" but often it means "a local file URL and nothing else." NSURL can hold arbitrary URLs with arbitrary schemas, but there are a ton of APIs in the frameworks which take NSURLs but only if they're file: URLs. It would be really nice to have a NSFileRef class or something like that which means, really, this is a reference to a local file, promise.
  - cballard 11y ago
    Yeah, I feel like all of the NSString-based APIs should have been deprecated by now?
dilap 11y ago
While you can technically store whatever you want in there, the assumption and convention the bytes in a string in Go will be utf8. E.g.: for i, x := range someString // iterates unicode code points Works well in practice. (There are the unicode and unicode/norm packages to do more complex unicode operations.)
- ridiculous_fish 11y ago
  I don't think it works well for text manipulation. For example, in Go, how would you truncate a string and add ellipsis, without dropping an accent or otherwise splitting a grapheme cluster?
tigeba 11y ago
When I was working on one of our SDKs (pre Swift 2.0), I found it rather maddening that the Swift string class had native support for emoji, but developers were left to create their own implementation of very basic features like indexOf, length, subString, etc.
- pilif 11y ago
  ... until you realise these "very basic" features are not very basic at all and pretty much depend on your current use-case. What length do you mean? Byte length? Character length? Code Point length? In case of indexOf, how do you handle surrogate pairs? does indexOf('ä') only find 'ä' (LATIN SMALL LETTER A WITH DIAERESIS) or also 'ä' (LATIN SMALL LETTER A followed by COMBINING DIAERESIS)? The good thing about the Swift API is that it gives you all the building blocks needed to actually having a chance at getting this right. Many other languages sweep those things under a rug and you're screwed or you'll have a much, much harder job to get it right if you need to.
  - cookiecaper 11y ago
    I think there's a pretty strong generalized use case for things like length (count the number of characters in a string) and indexOf (return the position of a character in a string). No reason to punish the 99% and force them to write their own implementations of these very conventional string operations to accommodate the 1% that means something not normally meant.
    - mikeash 11y ago
      Once you decide what a "character" means, these operations are easy. You want the count of grapheme clusters? string.characters.count. You want the index of a particular UTF-16 code unit? string.utf16.indexOf(codeUnit). If you need that index in the form of the number of code units from the start of the string, string.utf16.startIndex.distanceTo(index).
    - anon1385 11y ago
      Based on Github searches I've done in the past a lot of code calling length on NSString was broken. Those people probably thought they were in the 99% who just needed the convenient 'normal' length. People not realising that length isn't giving them what they want is one of the things the Swift API aims to solve. Giving the wrong choice for most situations (and number of UTF-16 code units isn't what people want most of the time) the easy convenient name is just a recipe for broken code. It's bad API design, or at least unfortunate historical accident, and it's good to improve things when there is an opportunity to do so.
chii 11y ago
i think this is apropo http://www.joelonsoftware.com/articles/Unicode.html http://www.joelonsoftware.com/articles/Unicode.html
zeveb 11y ago
> Human-readable text, file paths, SQL statements, and others are all conceptually different, and this should be represented as different types at the language level. I think that having different conceptual kinds of strings be distinct types would eliminate a lot of bugs. I'm not aware of any language or standard library that does this, though. The Common Lisp standard includes pathname objects, with an implementation-defined (since different platforms have different conventions) mapping from strings to path objects. It doesn't specifically address the type-safety issue, though, since just about every standard function which accepts a pathname will also accept a namestring. There exist libraries such as CLSQL which enable representation of SQL queries as objects. Common Lisp: doing it right for so long that most people haven't even heard of it! Also: > When measuring a string to ensure that it fits in a 140-character tweet, you want to go by unicode code points. Really? Doesn't one actually want 140 graphemes? Regardless of what one wants, does Twitter enforce 140 code points, or 140 bytes? It seems to me that graphemes = characters, and that the fact that some characters are made up of multiple codepoints is as important as the fact that some code points are made up of multiple bytes.
- zzalpha 11y ago
  I wonder if part of the reason for this is that Common Lisp dates back to a time when filesystem path structures were a lot more variable. These days, you either have Unix-style or Windows-style paths and that's basically it But go back 30 years and there was quite a bit more variety that would make an abstraction layer all but a requirement if you wanted to define a portable language spec.
  - rprospero 11y ago
    I remember reading a LISP book from the 80's that gently bragged about how all the legacy C code would have to be rewritten after upcoming operating systems moved beyond identifying files by a tree of folders, but that your LISP code would be future proof.
    - jdmichal 11y ago
      This strikes me as one of those instances were the cost of the abstraction exceeds the savings. There are some things which are simply core to your code. Attempting to abstract those things is going to create a lot of pain when, realistically, those things changing would require component rewrites anyway. If you're working with files, you've probably made many implicit assumptions about how the file system works and how it is represented. So you are either going to have a leaky abstraction that works, or a non-leaky abstraction that has those assumptions built in and will break when porting to a system without those assumptions anyway.
deleted 11y ago
[deleted]
consto 11y ago
>Human-readable text, file paths, SQL statements, and others are all conceptually different, and this should be represented as different types at the language level. String, Path/URL Objects, Prepared Statements.
ex3ndr 11y ago
the main problem with swift strings is actually naming. You don't have simple method replace(src:String, dst:String) but you have weird stringByReplacingOccurrencesOfString(src, withString: dest, options: NSStringCompareOptions(), range: nil) Why not to have simple convenience method?
- cballard 11y ago
  That is not a Swift method, it's an Objective-C method on NSString, which Swift String can be silently converted to.
  - zw 11y ago
    Not anymore; those methods are implemented in an extension on `String` that don't always involve bridging back.
    - mikeash 11y ago
      They're still only available if you import Foundation. They're just NSString methods with wrappers. The real problem is just that Swift itself doesn't have methods for locating a string in another string, or replacing strings in another string. This is why I was careful to say that Swift's API is the best in terms of its fundamental design. It still has a lot of missing functionality compared to other languages right now.
- alexbock 11y ago
  Aside from the fact that they carried that style of naming over from Objective-C, the one benefit I see is that it makes it obvious that it returns the result in a new string rather than modifying the object itself. In C++ you would need to look at the signature to tell which way a method named replace worked, e.g. void A::replace(const A& source, const A& dest); vs A A::replace(const A& source, const A& dest) const; That said, I think you can get this same benefit without such a verbose name. Perhaps something like "withReplacement"?
  - 11y ago
deleted 11y ago
[deleted]
CydeWeys 11y ago
Maybe it's just my first-world-privilege at work, but, honestly, why would I want to put up with a language that makes all of my day-to-day coding tasks so much more difficult than it is in other programming languages? It seems like I'm losing more than I'm gaining. 99% of what I do is in English. Everything that's not in English is going through an i18n library to convert labels and such. It doesn't seem like I'm gaining much of anything, but I sure am losing a lot. No default indexOf, length, and substring methods? Ouuuch.
- Analemma_ 11y ago
  Exactly. I understand where pilif and Lx1oG-AWb6h_ZG0 are coming from when they say that indexOf, length, and substring actually get a little ill-defined when Unicode is involved, but I can't see the logical leap from that to it being a good idea to exclude them entirely. The motto of a good API/SDK is "Make the easy case easy, and the hard case possible"; Swift seems to fail the first condition. Surely there are default assumptions for these methods that work most of the time, with the option for more complicated functionality if you need it. Maybe all this complexity is necessary when you're writing huge i18n'd apps, but the point is that if it's too much bother to even do Hello World, I'm never going to get to the huge i18n'ed apps.
  - anon1385 11y ago
    The problem with strings is that there is very little that is easy. The lesson is to avoid using strings as much as possible, although sadly fashions in programming seem to have been heading in the other direction for a long time. I don't think hiding difficult operations behind an easy facade that sometimes breaks is a good idea. It makes people think that working with strings is easy. It results in lots of people writing broken or even insecure code[1]. The idea that you only have to deal with unicode strings if you are writing an i18n'd app is wrong. File names are unicode. The content of the pasteboard is unicode. The results of calling a web API are unicode. Text the user types into a text field is unicode. Unless your app doesn't handle outside text data at all then it's going to encounter non-ascii text. And if you aren't handling text from outside the app then you probably won't have much need for string operations anyway. [1] do a github search for 'UTF8String' and 'length' to see how much code using NSString is passing the wrong length value into C apis.
chvid 11y ago
This article reads like the Stockholm syndrome for programmers - come on - any modern language needs to handle this stuff in a sane manner. Recently I wanted to skip the first character of a string and then convert it to a float. Here is the beauty: (text.substringFromIndex(text.startIndex.advancedBy(1)) as NSString).floatValue (Notice the start index "concept" and then the "cast" into the other string implementation (NSString) to get the to float conversion method.) In any normal language it would be something like: Float.parse(text.substring(1)) String handling is in Swift is half-finished at best.
- agumonkey 11y ago
  I get the idea behind very explicit method names, but seriously, python-like (or whoever invented it) slices are the most straightforward and succint I've seen so far, even without the [concrete:syntax].
- Gankro 11y ago
  Do you want to proceed forward one byte, "character", grapheme, or codepoint?
- mikeash 11y ago
  Let's at least write your example in a sane manner: Float(String(s.characters.dropFirst(1))) Is that so bad? The need to convert back to String is slightly annoying, but overall this is not bad. And of course there is the inevitable question: when you say "skip the first character," do you mean the first grapheme cluster, the first code point, the first UTF-16 code unit, or the first UTF-8 code unit? You say that "any modern language" needs to handle this stuff in a sane language? Which modern languages handle all of those possibilities sanely? I agree that the String API has some holes in it. Which, you know, is why I say things like "The String API does have some holes in it...." But in terms of how it's designed, it's the only one I've seen that gets it right.
  - fauigerzigerk 11y ago
    That's pretty clean. It is also rather inefficient. I think dropFirst copies the sequence and then the result is copied again to create a String and then finally it gets parsed into a Float. No problem for one Float, but parsing a largish CSV file like that may be a different matter. At least I think that's what's happening. I'm not 100% sure.
patsplat 11y ago
From the OP: > (Incidentally, I think that representing all these different concepts as a single string type is a mistake. Human-readable text, file paths, SQL statements, and others are all conceptually different, and this should be represented as different types at the language level. I think that having different conceptual kinds of strings be distinct types would eliminate a lot of bugs. I'm not aware of any language or standard library that does this, though.) There are many sub types which can be constructed from strings. Most languages treat at least following as special types that can be constructed from/to strings: * numeric types (integer, float, etc) * regular expressions * file paths * dates * XML * JSON The real absence is SQL. It is shocking how such an apparently old standard has no stdlib parsing support. Until one realizes there is no such thing as "Standard SQL". Any sufficiently visible "SQL Parser" project eventually will be buried in a sea of vendor-specific edge cases.
fauigerzigerk 11y ago
I think the Swift String API has a lot going for it conceptually. But I have a big problem with knowing so little about the performance and memory usage characteristics of the functions I'm calling and the strings I'm keeping in data structures. The problem is exacerbated by the fact that the API is extremely incomplete. A ton of things can only be done by resorting to NSString functions and that, I think, requires the String being copied into a UTF-16 representation underneath. Or does it? I don't know. That's exactly the problem. The idea of putting grapheme clusters at the center of the string universe is great when it comes to text that users see and manipulate. It's excellent for writing word processors. But it is not convenient for analysing large amounts of text data or for parsing semi structured text formats where we have a lot of fixed length stuff that could be conveniently accessed using numeric slice indexes. I think Swift's String class would work very well as a view on top of a UTF-8 buffer.
- mikeash 11y ago
  Is a String class the right type to use for parsing text formats with lots of fixed length stuff? Maybe Array<UInt8> would be more suitable.
  - fauigerzigerk 11y ago
    There's a lot of middle ground between parsing a purely structured format and UI oriented string manipulation. Most of what I have done in my life involved semi-structured data with both natural language text and structured parts with fixed lengths. It's extremely inconvenient to have no string functions available when working with byte arrays. I actually like the C approach a lot in principle, because it doesn't make you choose one or the other.
    - mikeash 11y ago
      Maybe it would be best to have a lot of these things we think of as "string" functions available on arrays, like sub-sequence matching, replacement, trimming, even something like regex. Then strings can be "all that, plus unicode."
coldtea 11y ago
>I think that having different conceptual kinds of strings be distinct types would eliminate a lot of bugs. I'm not aware of any language or standard library that does this, though.) Rebol does that AFAIK.
mpweiher 11y ago
> Human-readable text, file paths, SQL statements, and others are all conceptually different, and this should be represented as different types at the language level. Objective-Smalltalk[1] has Polymorphic Identifiers[2][3], which are URIs used as identifiers in the language. So that handles file paths and web addresses. SQL is not solved directly, but XPath can be encoded in the URI and there are mappings from relational DB APIs to URIs[4][5][6]. I hope we can get rid of identifiers encoded as strings once and for all. [1] http://objective.st/URIs http://objective.st/URIs [2] http://dl.acm.org/citation.cfm?id=2508169 http://dl.acm.org/citation.cfm?id=2508169 [3] https://www.hpi.uni-potsdam.de/hirschfeld/publications/media/WeiherHirschfeld_2013_PolymorphicIdentifiersUniformResourceAccessInObjectiveSmalltalk_AcmDL.pdf https://www.hpi.uni-potsdam.de/hirschfeld/publications/media... [4] http://blog.dreamfactory.com/add-a-rest-api-to-any-sql-db-in-minutes http://blog.dreamfactory.com/add-a-rest-api-to-any-sql-db-in... [5] http://restsql.org/doc/Overview.html http://restsql.org/doc/Overview.html [6] http://www.slashdb.com http://www.slashdb.com

97 points by chmaynard 11y ago

> I think that having different conceptual kinds of strings be distinct types would eliminate a lot of bugs. I'm not aware of any language or standard library that does this, though. Foundation seems like the obvious one? NSURL vs. NSString?

steveklabnik 11y ago
In the Rust standard library, we have String, str, OsString, Path, and PathBuf, off the top of my head, with external crates implementing things like URLs, ropes, and others. That said, I completely feel what this blog post is saying: strings are _hard_, especially if you're not just doing ASCII.
- x5n1 11y ago
  why are they so hard?
  - ben0x539 11y ago
    Because we spent a lot of time optimizing every assumption for the ascii-only case, and letting go of that simplification is distressing, I guess.
    - steveklabnik 11y ago
      Yes. And many languages had ASCII-only stuff for a long time, so "real" APIs seem much, much more complex. Of course, that complexity was always there, but us English speakers could mostly just ignore it...
  - Gankro 11y ago
    There's two aspects of complexity with strings: 1) 40 years of crazy encodings and languages. 2) human languages are wildly diverse and basically any assumption you wish to apply is broken. For 1, any system that wants to deal with the outside world needs to deal with: operating system encodings (arbitrary bytes on unix, malformed UCS2 on windows), C representation (null-terminated strings), systems that only work with ASCII, systems that only work with utf8, systems that work with arbitrary encodings/languages (HTML). This is arguably unnecessary complexity that exists because of short-sighted decisions in the past. 2 is the necessary complexity; the fact that languages are really complicated. There are thousands of symbols in writing. Do you try to encode these symbols in a monolithic manner, or in a compositional way? For historical reasons, you can often do both! ë can be a single character, or e with an accent modifier. How do you handle string searching in such a model? Do you match `noel` with `noël`? What's the length of noël? 4 characters? 5 characters? bytes? graphemes? codepoints? Can you correctly reverse noël (do it wrong and you can get leön)? Different letters which have similar/identical representations but different semantics/origins! Is Ε "capital e" or "capital ε"? How do you upper-case or lower-case these letters? Do you expect to_upper(to_lower(char)) to roundtrip (it won't)? Do you expect capitalization to be doable in-place (it's not)? Do you expect capitalization to be region-specific (it is)? Are any of these operations even coherent in a language like Japanese? Why are you trying to do them? God help you if you want to display this text. Are you ready to handle right-to-left text? Are you assuming that your font is monospace (hey there terminal and text editors)? C̢̫a̘̺̯n ̘̜̦̹y̷̫̼̘̩o̶͉u̗̩̻̞ ̻ẹ͡v̴̤͎̹e̶̫̠̤̭̺̤̞n̛̞̹̣̩̲͉̮ ̜͖̪͔̖d̤e̘̯ͅa̺l̟̀ ͚̗̣w̭i̸͇̠̥̣̜̥t̸h̸̻̮̼̙̹ ̗̺̱̣̰̱̙z̟a̺͜l̠̦̖̟̰͍g҉̜͖͓̫ơ̩̹̰͕?̹̳̼̯̘̺̟
    - mikeash 11y ago
      I think my favorite part of this article is how my Zalgo example doesn't render even remotely correctly in any browser I've tried it in.
- Gankro 11y ago
  std provides String (utf8), OsString (bytes/wtf8), PathBuf (bytes/wtf8), CString (null-terminated). Each has their own unsized view: str, OsStr, Path, and CStr. We also provide AsciiExt for those times where you really truly believe you want to be working with a String as Ascii. That said, we generally try to make it as ergonomic as possible to pass a plain str where a Path/OsStr is expected. This is because utf8 is a subset of wtf8, so it's always fine to convert in that direction blindly (and it's really nice to just be like `File::open("foo.txt")` when hacking something together). This is why so many interfaces are riddled with something like `P: As<Path>`. The differentiation largely exists for the other direction, IMO. Paths and OsStrs aren't guaranteed to be valid UTF8, and shouldn't be provided where a proper utf8 string is expected. Path is just a convenience wrapper over OsStr that understands the platform's seperator conventions and provides convenient utilities.
- erickt 11y ago
  Don't forget we kinda sorta have byte characters and strings a la `b'x'` and `b"foo"`, which are really just simpler ways of expressing byte slices. Unfortunately they lack string specific methods until we get specialization (fingers crossed).
kazinator 11y ago
Typing strings to solve bugs is the epitome of an anti-pattern. If you want type, yank the data out of the string representation and into an actual data type! "The string is a stark data structure and everywhere it is passed there is much duplication of process. It is a perfect vehicle for hiding information." -- Alan Perlis, Epigram #34.
- jasode 11y ago
  >, yank the data out of the string representation Well, the very act of "yanking" can be helped by more well-defined behavior policies of "typed strings". It is unavoidable that the programmer will have to confront a contiguous piece of memory (string) with no boundaries between characters whether it's a block of bytes on disk, network stream, or the clipboard buffer. It can be helpful to differentiate the various types of strings with a formal type system before the yanking takes place. The Alan Perlis epigram is wise but I don't think it applies here. I think's he's talking about abusing strings as an adhoc data format by stuffing them with multi-value data and then turning right around and parsing them. For example, concatenating name+":"+id+":"+amount into a string with colons like "John:389432:$5.99", passing the "complex" string around, and then later parsing out the colons with string.split(). Basically, the programmer is making an in-memory csv (comma separated values) data format instead of using "C struct {}" or passing 3 separate formal parameters.
  - mannykannot 11y ago
    > ...before the yanking takes place. I take it that you are advocating for typed strings in addition to fully-formed types for things like URLs and SQL, but I think all that will lead to is a lot of gratuitous casting of strings just so they can be passed to the fully-formed types. Furthermore, few of these casts could be statically checked for validity.
    - jasode 11y ago
      >, but I think all that will lead to is a lot of gratuitous casting of strings But if programmers are not directly doing memcpy() of bytes directly to the stack frame, they are already doing a roundabout version of "casting". If a programmer has to "yank" from a string to put it into a more organized data type, the "yank" has to be done within the framework of semantics. If one "yanks" or "slices", or "substr", etc characters 3 to 15, the exact behavior depends on whether contiguous bytes is ASCII, UTF8, UTF16, or binary blob (which means NUL(0) is valid byte in the middle of the block of bytes). That "meaning overlayed onto those bytes to manipulate it correctly" has to decided somewhere. It is either implicit or explicit. Likewise, if one pulls 4 bytes from disk or network, the programmer has to assign meaning to it and say it is the 4-bytes of an IEEE floating point (float) or 4-bytes of a integer (int). The programmer may later want to do a bitshift (<<) which is not applicable to a float.
fizx 11y ago
/Regexp?/
mikeash 11y ago
Sort of. One spot of trouble is that NSString paths still exist and show up all over the place, even though NSURL is slowly replacing them. Another spot of trouble is that NSURL ends up in a similar situation, because sometimes NSURL means "an arbitrary URL" but often it means "a local file URL and nothing else." NSURL can hold arbitrary URLs with arbitrary schemas, but there are a ton of APIs in the frameworks which take NSURLs but only if they're file: URLs. It would be really nice to have a NSFileRef class or something like that which means, really, this is a reference to a local file, promise.
- cballard 11y ago
  Yeah, I feel like all of the NSString-based APIs should have been deprecated by now?

dilap 11y ago

While you can technically store whatever you want in there, the assumption and convention the bytes in a string in Go will be utf8. E.g.: for i, x := range someString // iterates unicode code points Works well in practice. (There are the unicode and unicode/norm packages to do more complex unicode operations.)

ridiculous_fish 11y ago
I don't think it works well for text manipulation. For example, in Go, how would you truncate a string and add ellipsis, without dropping an accent or otherwise splitting a grapheme cluster?

tigeba 11y ago

When I was working on one of our SDKs (pre Swift 2.0), I found it rather maddening that the Swift string class had native support for emoji, but developers were left to create their own implementation of very basic features like indexOf, length, subString, etc.

pilif 11y ago
... until you realise these "very basic" features are not very basic at all and pretty much depend on your current use-case. What length do you mean? Byte length? Character length? Code Point length? In case of indexOf, how do you handle surrogate pairs? does indexOf('ä') only find 'ä' (LATIN SMALL LETTER A WITH DIAERESIS) or also 'ä' (LATIN SMALL LETTER A followed by COMBINING DIAERESIS)? The good thing about the Swift API is that it gives you all the building blocks needed to actually having a chance at getting this right. Many other languages sweep those things under a rug and you're screwed or you'll have a much, much harder job to get it right if you need to.
- cookiecaper 11y ago
  I think there's a pretty strong generalized use case for things like length (count the number of characters in a string) and indexOf (return the position of a character in a string). No reason to punish the 99% and force them to write their own implementations of these very conventional string operations to accommodate the 1% that means something not normally meant.
  - mikeash 11y ago
    Once you decide what a "character" means, these operations are easy. You want the count of grapheme clusters? string.characters.count. You want the index of a particular UTF-16 code unit? string.utf16.indexOf(codeUnit). If you need that index in the form of the number of code units from the start of the string, string.utf16.startIndex.distanceTo(index).
  - anon1385 11y ago
    Based on Github searches I've done in the past a lot of code calling length on NSString was broken. Those people probably thought they were in the 99% who just needed the convenient 'normal' length. People not realising that length isn't giving them what they want is one of the things the Swift API aims to solve. Giving the wrong choice for most situations (and number of UTF-16 code units isn't what people want most of the time) the easy convenient name is just a recipe for broken code. It's bad API design, or at least unfortunate historical accident, and it's good to improve things when there is an opportunity to do so.

chii 11y ago

i think this is apropo http://www.joelonsoftware.com/articles/Unicode.html http://www.joelonsoftware.com/articles/Unicode.html

zeveb 11y ago

> Human-readable text, file paths, SQL statements, and others are all conceptually different, and this should be represented as different types at the language level. I think that having different conceptual kinds of strings be distinct types would eliminate a lot of bugs. I'm not aware of any language or standard library that does this, though. The Common Lisp standard includes pathname objects, with an implementation-defined (since different platforms have different conventions) mapping from strings to path objects. It doesn't specifically address the type-safety issue, though, since just about every standard function which accepts a pathname will also accept a namestring. There exist libraries such as CLSQL which enable representation of SQL queries as objects. Common Lisp: doing it right for so long that most people haven't even heard of it! Also: > When measuring a string to ensure that it fits in a 140-character tweet, you want to go by unicode code points. Really? Doesn't one actually want 140 graphemes? Regardless of what one wants, does Twitter enforce 140 code points, or 140 bytes? It seems to me that graphemes = characters, and that the fact that some characters are made up of multiple codepoints is as important as the fact that some code points are made up of multiple bytes.

zzalpha 11y ago
I wonder if part of the reason for this is that Common Lisp dates back to a time when filesystem path structures were a lot more variable. These days, you either have Unix-style or Windows-style paths and that's basically it But go back 30 years and there was quite a bit more variety that would make an abstraction layer all but a requirement if you wanted to define a portable language spec.
- rprospero 11y ago
  I remember reading a LISP book from the 80's that gently bragged about how all the legacy C code would have to be rewritten after upcoming operating systems moved beyond identifying files by a tree of folders, but that your LISP code would be future proof.
  - jdmichal 11y ago
    This strikes me as one of those instances were the cost of the abstraction exceeds the savings. There are some things which are simply core to your code. Attempting to abstract those things is going to create a lot of pain when, realistically, those things changing would require component rewrites anyway. If you're working with files, you've probably made many implicit assumptions about how the file system works and how it is represented. So you are either going to have a leaky abstraction that works, or a non-leaky abstraction that has those assumptions built in and will break when porting to a system without those assumptions anyway.

deleted 11y ago

[deleted]

consto 11y ago

>Human-readable text, file paths, SQL statements, and others are all conceptually different, and this should be represented as different types at the language level. String, Path/URL Objects, Prepared Statements.

ex3ndr 11y ago

the main problem with swift strings is actually naming. You don't have simple method replace(src:String, dst:String) but you have weird stringByReplacingOccurrencesOfString(src, withString: dest, options: NSStringCompareOptions(), range: nil) Why not to have simple convenience method?

cballard 11y ago
That is not a Swift method, it's an Objective-C method on NSString, which Swift String can be silently converted to.
- zw 11y ago
  Not anymore; those methods are implemented in an extension on `String` that don't always involve bridging back.
  - mikeash 11y ago
    They're still only available if you import Foundation. They're just NSString methods with wrappers. The real problem is just that Swift itself doesn't have methods for locating a string in another string, or replacing strings in another string. This is why I was careful to say that Swift's API is the best in terms of its fundamental design. It still has a lot of missing functionality compared to other languages right now.
alexbock 11y ago
Aside from the fact that they carried that style of naming over from Objective-C, the one benefit I see is that it makes it obvious that it returns the result in a new string rather than modifying the object itself. In C++ you would need to look at the signature to tell which way a method named replace worked, e.g. void A::replace(const A& source, const A& dest); vs A A::replace(const A& source, const A& dest) const; That said, I think you can get this same benefit without such a verbose name. Perhaps something like "withReplacement"?
- 11y ago

deleted 11y ago

[deleted]

CydeWeys 11y ago

Maybe it's just my first-world-privilege at work, but, honestly, why would I want to put up with a language that makes all of my day-to-day coding tasks so much more difficult than it is in other programming languages? It seems like I'm losing more than I'm gaining. 99% of what I do is in English. Everything that's not in English is going through an i18n library to convert labels and such. It doesn't seem like I'm gaining much of anything, but I sure am losing a lot. No default indexOf, length, and substring methods? Ouuuch.

Analemma_ 11y ago
Exactly. I understand where pilif and Lx1oG-AWb6h_ZG0 are coming from when they say that indexOf, length, and substring actually get a little ill-defined when Unicode is involved, but I can't see the logical leap from that to it being a good idea to exclude them entirely. The motto of a good API/SDK is "Make the easy case easy, and the hard case possible"; Swift seems to fail the first condition. Surely there are default assumptions for these methods that work most of the time, with the option for more complicated functionality if you need it. Maybe all this complexity is necessary when you're writing huge i18n'd apps, but the point is that if it's too much bother to even do Hello World, I'm never going to get to the huge i18n'ed apps.
- anon1385 11y ago
  The problem with strings is that there is very little that is easy. The lesson is to avoid using strings as much as possible, although sadly fashions in programming seem to have been heading in the other direction for a long time. I don't think hiding difficult operations behind an easy facade that sometimes breaks is a good idea. It makes people think that working with strings is easy. It results in lots of people writing broken or even insecure code[1]. The idea that you only have to deal with unicode strings if you are writing an i18n'd app is wrong. File names are unicode. The content of the pasteboard is unicode. The results of calling a web API are unicode. Text the user types into a text field is unicode. Unless your app doesn't handle outside text data at all then it's going to encounter non-ascii text. And if you aren't handling text from outside the app then you probably won't have much need for string operations anyway. [1] do a github search for 'UTF8String' and 'length' to see how much code using NSString is passing the wrong length value into C apis.

chvid 11y ago

This article reads like the Stockholm syndrome for programmers - come on - any modern language needs to handle this stuff in a sane manner. Recently I wanted to skip the first character of a string and then convert it to a float. Here is the beauty: (text.substringFromIndex(text.startIndex.advancedBy(1)) as NSString).floatValue (Notice the start index "concept" and then the "cast" into the other string implementation (NSString) to get the to float conversion method.) In any normal language it would be something like: Float.parse(text.substring(1)) String handling is in Swift is half-finished at best.

agumonkey 11y ago
I get the idea behind very explicit method names, but seriously, python-like (or whoever invented it) slices are the most straightforward and succint I've seen so far, even without the [concrete:syntax].
Gankro 11y ago
Do you want to proceed forward one byte, "character", grapheme, or codepoint?
mikeash 11y ago
Let's at least write your example in a sane manner: Float(String(s.characters.dropFirst(1))) Is that so bad? The need to convert back to String is slightly annoying, but overall this is not bad. And of course there is the inevitable question: when you say "skip the first character," do you mean the first grapheme cluster, the first code point, the first UTF-16 code unit, or the first UTF-8 code unit? You say that "any modern language" needs to handle this stuff in a sane language? Which modern languages handle all of those possibilities sanely? I agree that the String API has some holes in it. Which, you know, is why I say things like "The String API does have some holes in it...." But in terms of how it's designed, it's the only one I've seen that gets it right.
- fauigerzigerk 11y ago
  That's pretty clean. It is also rather inefficient. I think dropFirst copies the sequence and then the result is copied again to create a String and then finally it gets parsed into a Float. No problem for one Float, but parsing a largish CSV file like that may be a different matter. At least I think that's what's happening. I'm not 100% sure.

patsplat 11y ago

From the OP: > (Incidentally, I think that representing all these different concepts as a single string type is a mistake. Human-readable text, file paths, SQL statements, and others are all conceptually different, and this should be represented as different types at the language level. I think that having different conceptual kinds of strings be distinct types would eliminate a lot of bugs. I'm not aware of any language or standard library that does this, though.) There are many sub types which can be constructed from strings. Most languages treat at least following as special types that can be constructed from/to strings: * numeric types (integer, float, etc) * regular expressions * file paths * dates * XML * JSON The real absence is SQL. It is shocking how such an apparently old standard has no stdlib parsing support. Until one realizes there is no such thing as "Standard SQL". Any sufficiently visible "SQL Parser" project eventually will be buried in a sea of vendor-specific edge cases.

fauigerzigerk 11y ago

I think the Swift String API has a lot going for it conceptually. But I have a big problem with knowing so little about the performance and memory usage characteristics of the functions I'm calling and the strings I'm keeping in data structures. The problem is exacerbated by the fact that the API is extremely incomplete. A ton of things can only be done by resorting to NSString functions and that, I think, requires the String being copied into a UTF-16 representation underneath. Or does it? I don't know. That's exactly the problem. The idea of putting grapheme clusters at the center of the string universe is great when it comes to text that users see and manipulate. It's excellent for writing word processors. But it is not convenient for analysing large amounts of text data or for parsing semi structured text formats where we have a lot of fixed length stuff that could be conveniently accessed using numeric slice indexes. I think Swift's String class would work very well as a view on top of a UTF-8 buffer.

mikeash 11y ago
Is a String class the right type to use for parsing text formats with lots of fixed length stuff? Maybe Array<UInt8> would be more suitable.
- fauigerzigerk 11y ago
  There's a lot of middle ground between parsing a purely structured format and UI oriented string manipulation. Most of what I have done in my life involved semi-structured data with both natural language text and structured parts with fixed lengths. It's extremely inconvenient to have no string functions available when working with byte arrays. I actually like the C approach a lot in principle, because it doesn't make you choose one or the other.
  - mikeash 11y ago
    Maybe it would be best to have a lot of these things we think of as "string" functions available on arrays, like sub-sequence matching, replacement, trimming, even something like regex. Then strings can be "all that, plus unicode."

coldtea 11y ago

>I think that having different conceptual kinds of strings be distinct types would eliminate a lot of bugs. I'm not aware of any language or standard library that does this, though.) Rebol does that AFAIK.

mpweiher 11y ago

> Human-readable text, file paths, SQL statements, and others are all conceptually different, and this should be represented as different types at the language level. Objective-Smalltalk[1] has Polymorphic Identifiers[2][3], which are URIs used as identifiers in the language. So that handles file paths and web addresses. SQL is not solved directly, but XPath can be encoded in the URI and there are mappings from relational DB APIs to URIs[4][5][6]. I hope we can get rid of identifiers encoded as strings once and for all. [1] http://objective.st/URIs http://objective.st/URIs [2] http://dl.acm.org/citation.cfm?id=2508169 http://dl.acm.org/citation.cfm?id=2508169 [3] https://www.hpi.uni-potsdam.de/hirschfeld/publications/media/WeiherHirschfeld_2013_PolymorphicIdentifiersUniformResourceAccessInObjectiveSmalltalk_AcmDL.pdf https://www.hpi.uni-potsdam.de/hirschfeld/publications/media... [4] http://blog.dreamfactory.com/add-a-rest-api-to-any-sql-db-in-minutes http://blog.dreamfactory.com/add-a-rest-api-to-any-sql-db-in... [5] http://restsql.org/doc/Overview.html http://restsql.org/doc/Overview.html [6] http://www.slashdb.com http://www.slashdb.com