5 ms·
Unfortunately, since Swift's String implementation operates on grapheme clusters by default, a lot of parsing code people write is actually subtly broken in the
by eridius 8y ago
Unfortunately, since Swift's String implementation operates on grapheme clusters by default, a lot of parsing code people write is actually subtly broken in the presence of combining characters. As a trivial example, let's say the input is a comma-delimited string (say, a line of CSV without quotes). The obvious way to split this is
let fields = line.split(separator: ",")
But given the input "foo,\u{301}bar" (which looks like foo,́bar), this won't split correctly and you'll end up with a single field that contains a comma. The correct way to split this is
let fields = line.unicodeScalars.split(separator: ",").map(Substring.init)
This will get you the correct 2 fields (at the cost of an intermediate array, as there is no lazy split).
- Twisell 8y agoHave you opened a ticket about that? This is more due to the split implementation than to the format itself. Room for improvement.
- deleted 8y ago[deleted]
- eridius 8y agoThis is not an issue with the split implementation. Split is behaving correctly (according to the defined semantics of String). You'd get the exact same issue if you said str.index(of: ",") instead; in both cases, the "," is a Character, not a UnicodeScalar, and "," != ",\u{301}".
- masklinn 8y agoThe original split looks correct to me. If you're assuming CSV is a textual format, a comma with a combining acute accent is not the same thing as a comma: if I'm asking for a split on "e", I don't want a split on "é" whether in its precomposed form or not, and I especially do not want lone combining characters floating around as a result. If you're assuming CSV is a binary format, you should split on code units before the textual decoding.
- eridius 8y agoCSV is a machine-readable format. I've never heard of a machine-readable format that treats combining characters as significant when parsing syntax. Certainly the CSV format doesn't. Trivial proof: If it did, then it would be literally impossible to represent a field starting with a combining character. Also, the set of combining characters has changed over time. Machine-readable formats in general do not change as Unicode does. A CSV that parses today should not fail to parse next year because a field starts with a codepoint that today is unused and next year has been assigned to a combining character.
- dbaupp 8y ago"Parsing" CSV using split (whether on graphemes or on scalars) isn't correct at all, though, due to quoting.
- eridius 8y agoThis is why I said "without quotes". The same issue happens if you write a parsing routine that handles quotes too of course, it was just a lot simpler to demonstrate it with a single split() line than a whole parsing loop.