10 ms·
This is really great to hear. The different internal string representations really age a language: - C sort of assumes all strings are ASCII - Java, C#, Obj-C
by josephg 8y ago
This is really great to hear. The different internal string representations really age a language:
- C sort of assumes all strings are ASCII
- Java, C#, Obj-C, Javascript and Python2 were all written when it was assumed Unicode would have no more than 65536 characters. They all use 2 byte encodings, which has become the worst of all worlds. ASCII text is twice the size it needs to be, you still need to handle the mismatch between array length and codepoint length, and errors are harder to find because there are very few common characters that actually cross the 2 byte boundary - although the popularity of emoji has changed this.
- Go, Dart, Rust and Zig all return to single byte encodings. They all use UTF-8 internally, and have different functions to interact with the string as a list of bytes, and the string as a list of unicode codepoints. (In Go's case its because Rob Pike was behind both Go and UTF8.)
Its delightful to see Swift stepping away from Obj-C's mistakes and improving things internally. Although in Obj-C's defence, NSString has always hidden its internal representation, which did a lot of work to protect developers from the footguns.
One of the big ironies of all this is how well C has stood the test of time. C strings interoperate beautifully with UTF8 - all thats missing is some libc helper methods to count & iterate through UTF8 codepoints. strlen / strncpy / strcmp / etc all work perfectly when dealing with UTF8. The only change is that you need to supply lengths in bytes not characters.
- paulddraper 8y ago> C strings interoperate beautifully with UTF8 C strings interoperate terribly with almost any encoding, including ASCII and UTF8. C strings cannot contain the NUL character.
- unwind 8y agoBut NUL is not used by UTF-8 (by design, of course) so that cannot be a problem.
- taejo 8y agoU+0000 is a unicode character, which is encoded as 0x00 in UTF-8
- masklinn 8y agoNot sure why you'd think that, NUL is a perfectly valid, supported and usable character in unicode, and thus UTF8.
- kibwen 8y agoAny implementation of strings that cannot contain 0x0 is not a conformant implementation of UTF-8. It's not a conformant implementation of ASCII either, for that matter. C strings are neither (not that they claim to be).
- paulddraper 8y agoIt's been a character since the earliest Unicode. https://www.fileformat.info/info/unicode/char/0000/index.htm https://www.fileformat.info/info/unicode/char/0000/index.htm
- Mindless2112 8y ago.NET is going to add UTF8 strings [1], though it remains to be seen how usable they will be given that all existing APIs take non-UTF8 strings. [1] https://github.com/dotnet/corefxlab/issues/2350 https://github.com/dotnet/corefxlab/issues/2350
- bluejekyll 8y ago> C strings interoperate beautifully with UTF8 Isn’t this exactly the design goal of UTF8? To be backward compatible with ASCII strings?
- kibwen 8y ago> C strings interoperate beautifully with UTF8 Reversing cause and effect, methinks: UTF-8 interoperates beautifully with ASCII. :P
- jstimpfle 8y ago> C sort of assumes all strings are ASCII How so? Concerning runtime values, you can do whatever you want in C, which means UTF-8 support in C is basically free. Note that UTF-8 was explicitly designed to have this kind of upward compatibility. Concerning string literals, just keep your source files encoded in UTF-8 and put in quotes the UTF-8 string you want to process at runtime. I'm not aware how the spec handles source code files that are not ASCII, but in practice it works, and furthermore you can always opt for hex escapes.
- deleted 8y ago[deleted]
- ken 8y ago> Although in Obj-C's defence, NSString has always hidden its internal representation, which did a lot of work to protect you from this. Technically the docs never said what the internal representation was, but it was not very well hidden at all. As the article points out, NSString presents an API with "constant-time access to UTF-16 code units".
- josephg 8y agoThe foot gun happens when you assume that the string representation's length is equal to the number of unicode characters. This is true in C if all your test data uses ASCII characters. Its true in Java / C# / etc if you only test using ASCII + any characters in the lower unicode planes. And thats most but not all of the characters used in the world today. In Javascript, all the standard APIs make the wrong assumptions about unicode. Unless you think about it and actually test using exotic characters, its very common to make mistakes where certain characters would crash your software. (I'm talking about you, iMessage). I ran into a bug just the other day where emoji in a comment in a rust file with RLS caused a weird rendering error in VSC. In javascript String#length and naive array indexing is basically always wrong, and as far as I know there's no official API that does the right thing. But because its mostly right, people use String#length and don't think about it. $ node > "𝄞".length 2 > "𝄞"[0] '�' // oops! In Objective-C, even though large NSStrings use a 2 byte encoding under the hood, the API that NSString provides encourages you to do the right thing by default a lot more than javascript does. We could talk about whether language or the documentation is responsible, but either way the difference translates into fewer bugs.
- dwaite 8y agoUnicode is complex enough it is often best just to not think of it in terms of a character count unless you are doing text layout (in which case you care about grapheme clusters, not characters).
- josephg 8y agoI take your point, but it really depends on the application. I care about this having worked on realtime collaborative editors. In this space it definitely does make sense to describe string offsets with unicode codepoints. The problem with counting the number of grapheme clusters is that that number can change with every revision of the unicode spec, and you can get different answers on different platforms. Or the same platform, but a different version of the OS. Does a zero width space offset the position count or not? Do you get the same answer on Windows, MacOS and through whatever unix you happen to use AWS? I shudder just thinking about it. In contrast, counting unicode codepoints is simple, consistent and well defined. Counting string lengths that way is also implemented in the standard library of just about all languages.
- kjeetgill 8y agoAs of Java 9 have uses one byte UTF-8 array instead UTF-16 chars internally. See: http://openjdk.java.net/jeps/254 http://openjdk.java.net/jeps/254
- dwaite 8y agoIt is a Latin-1 byte array, not a UTF-8 array. This is because Java exposes some details like the length of the UTF-16 character array through the string api.
- TazeTSchnitzel 8y ago(And Latin-1 is a strict subset of Unicode, which means all the direct code point lookups are superfast, literally just a zero extension.)
- kjeetgill 8y agoAh, my bad. I stand corrected.
- deathanatos 8y ago> One of the big ironies of all this is how well C has stood the test of time. C strings interoperate beautifully with UTF8 - all thats missing is some libc helper methods to count & iterate through UTF8 codepoints. strlen / strncpy / strcmp / etc all work perfectly when dealing with UTF8. The only change is that you need to supply lengths in bytes not characters. That's exaggerating the truth, I think. Programmers can and do presume strlen's result is things like the length of the string in characters (and the typename "char" does not help), or the number of terminal columns wide a string is. E.g., I'm pretty sure the mysql client can still, to this day, not properly format a table, and IIRC, it is written in C. strcmp() does a byte-for-byte comparison of strings, which in Unicode may very well be meaningless. strncpy() has its own footguns w/ leaving the buffer unterminated. (But yes, as long as your indexes into the string make "sense" — aligned to code points that will make sense at the destination — it works fine.) If you have to write any of the helper methods to count & iterate through UTF8 codepoints, you have to watch out for the footgun that char is sometimes signed, sometimes not, so attempting to get the high bits to determine things like "is this a continuation byte?" with >> or & is either implementation defined or UB, depending on the specifics of the platform. (>> is UB on negatives, which char can be, and often is; & on a signed int doesn't make much sense, and it's implementation defined what it does in C (it operates on the bitwise representation of whatever the underlying signed integer representation is, which is implementation defined. Yes, yes, all the world is two's complement. The earlier UB is much more likely to bite you.)) unsigned char * is better for both raw binary buffers and dealing w/ buffers of UTF-8 data. (Honestly, I'm not even sure that C requires the character literal 'A' to correspond to an ASCII "A". But all the world is ASCII…)
- kevin_thibedeau 8y ago> Honestly, I'm not even sure that C requires the character literal 'A' to correspond to an ASCII "A". It doesn't. Nor do chars have to be 8-bits.
- xeeeeeeeeeeenu 8y ago>(Honestly, I'm not even sure that C requires the character literal 'A' to correspond to an ASCII "A". But all the world is ASCII…) EBCDIC is still alive and well on IBM mainframes.
- kevin_thibedeau 8y agoPython 2.2 added a compile time option for using UCS-4 internally. https://www.python.org/dev/peps/pep-0261/ https://www.python.org/dev/peps/pep-0261/
- WalterBright 8y agoD has string, wstring, and dstring types, corresponding to UTF-8, UTF-16, and UCS-32 encodings. But people normally just stick with string. It's been a huge win for D.
- josteink 8y ago> One of the big ironies of all this is how well C has stood the test of time. C strings interoperate beautifully with UTF8 I wouldn't say that. I'd rather claim that UTF-8 has been engineered to be ASCII pass-through compatible because of the lacking Unicode support in most C-codebases out there. I mean... If you compare Unicode support in platforms and OSes where this was clearly a thought (Windows, C#, Java) to platforms with the more naive approach (Linux, C, PHP, etc), you will see a very clear picture of which side has the most unicode bugs and encoding errors. And I say that as a Linux-guy. C is terrible for text-processing. UTF-8 was designed as it was because of that. Trying to paint C as the best choice here because someone found a working solution they could also apply to terrible C-code is clearly seeing things backwards.
- yoklov 8y agoWhile I agree that it's not really a measure of C standing the test of time, I kind of doubt that an alternate universe where 'C interop' / 'ASCII compatibility' weren't a design consideration, we'd have ended up with something much better (or TBH much different) than UTF-8.
- the_mitsuhiko 8y ago> If you compare Unicode support in platforms and OSes where this was clearly a thought (Windows, C#, Java) Their insistence on UCS2 however gave us the worst outcome of all. We have forever lost all north of 23 bits for characters due to UTF16. The result of this misdesign creates bigger problem than C’s char type.
- mikeash 8y agoUCS-2 was all there was at the time, though. Blame Unicode for thinking 65,536 code points would be enough.
- gpderetta 8y agoThat's not a very good excuse though: UTF-8 was invented, by Thompson and Pike, specifically because existing solutions were not deemed adequate for their needs. "Reasonable people adapt themselves to the world. Unreasonable people attempt to adapt the world to themselves. All progress, therefore, depends on unreasonable people"
- needusername 8y ago> Java ... They all use 2 byte encodings No, not anything that is OpenJDK 9+ based which uses 1 byte where possible. > They all use UTF-8 internally Which means a lot of functions now have linear instead of constant asymptotic complexity.
- masklinn 8y ago> Which means a lot of functions now have linear instead of constant asymptotic complexity. They already do if they do proper text manipulation as unicode itself is variable length and has to be stream-processed. O(1) access to codepoints is not actually useful, and most languages don't even provide it since they don't internally encode to UTF-32.
- masklinn 8y ago> - Java, C#, Obj-C, Javascript and Python2 were all written when it was assumed Unicode would have no more than 65536 characters. Python 2 had "narrow builds" (2-byte unichar) and "wide builds" (4 byte unichar). Wide was generally the default on unices. > - Go, Dart, Rust and Zig all return to single byte encodings. They all use UTF-8 internally, and have different functions to interact with the string as a list of bytes, and the string as a list of unicode codepoints. (In Go's case its because Rob Pike was behind both Go and UTF8.) Dunno about Dart and Zig, but AFAIK Go is closer to Ruby's old model of "IDK lol". A number of string-processing functions assume (and sometimes even assert) UTF8, but that's not a guarantee at all[0] > It's important to state right up front that a string holds arbitrary bytes. It is not required to hold Unicode text, UTF-8 text, or any other predefined format. As far as the content of a string is concerned, it is exactly equivalent to a slice of bytes. For Rust on the other hand, having non-UTF8 byte sequences in an str is one of the language's UB (hence str::from_utf8_unchecked existing and being unsafe). > C strings interoperate beautifully with UTF8 - all thats missing is some libc helper methods to count & iterate through UTF8 codepoints. strlen / strncpy / strcmp / etc all work perfectly when dealing with UTF8. C strings don't "interoperate with UTF8": they treat everything as a nul-terminated bag of bytes and that's it. Which means they won't properly process valid UTF8 sequences containing NUL, and they will improperly process invalid UTF8 sequences. Ignoring the entire thing is not interoperability, you can not actually operate on text using the C stdlib. All you can do is treat it as an opaque blob and move it from one place to the other mostly unmolested. [0] https://blog.golang.org/strings https://blog.golang.org/strings
- sametmax 8y agoThats's one of main reason we got python 3. The string string story is not as good as the swift one, but in 3.6, it's one of the best i could work with. Espacially for a language that works on so many platforms, for so many use cases including web and scripting in i18n corporate environnements. Bottom line, don't promote python 2. At this stage, it's legacy. We maintain existing money makers with it, but all new projects have no reason to be in 2.
- jcelerier 8y ago> although the popularity of emoji has changed this. are they that popular outside of chat apps ? how many emojis on this very page ?
- benibela 8y agoDelphi/Pascal had the best strings Single byte storage, so it works well with utf-8. Byte length field, so it knows the length of every string and it can store null bytes. An additional null byte past the end and pointer shifting, so every Pascal string is also a C pchar string. Reference-counted copy-on-write, so you can copy it in constant time for reading and treat it as if the string was fully copied for writing. Low-level memory management, so it can store 2 GB long strings. And in most situations you do not need a string builder, because strings are freed immediately and very fast when the refcounts zeros without staying around as garbage. Behaves as a value type with the null pointer being the empty string, so you can never get a null pointer exception. Any string operation returns always a valid string. (Optionally) index checked with the length, so you can never get a buffer overflow. Unfortunately the newer FreePascal/Delphi versions made it all very confusing by adding an encoding field. In the past you could just assume all strings are UTF-8 as code style rule. Now each string has an individual encoding. It is converted automatically from one encoding, but not always, so in any function that needs to access the characters, you kind of need to check if it is called with an utf-8 string, or latin1 string, or some other encoding.
- vintagedave 8y ago> Unfortunately the newer FreePascal/Delphi versions made it all very confusing by adding an encoding field I can clear this up. This change is really useful and important if you are using strings. A string of bytes has attached metadata saying what it is. Is it ANSI of some sort, or UTF8, or...? Is it a specific encoding, such as Windows-1252? Without that data, all you have are bytes, and you don't know how to interpret them. Thus, RawByteString (bytes); UTF8String (UTF8); and ANSI strings with the encoding, plus UnicodeString which is native Unicode on whichever platform (eg, on Windows it matches Windows UTF16.) This data is essential to convert to and from different string types. I don't know where conversion does "not always" happen - can't think of anywhere offhand. But if you ever run into issues, there are RTL functions for conversion. Check out the TEncoding class: http://docwiki.embarcadero.com/Libraries/Tokyo/en/System.SysUtils.TEncoding http://docwiki.embarcadero.com/Libraries/Tokyo/en/System.Sys... > In the past you could just assume all strings are UTF-8 as code style rule. This was an incorrect assumption, because before the encoding metadata was added, you would have been using an AnsiString there, and that, by definition ('Ansi'), is not UTF8. These days, if you have a UTF8 string, you can place it in a UTF8String type. Correctness enforced by libraries is much better than a coding convention that a certain type contains a subtly different payload. That way lies horror. Metadata and strong typing is much safer. -- I do agree with you that Delphi has the best strings :) Copy on write and embedded length both seem a real win, after twenty years of use, not to mention great string twiddling methods. I'm looking at adding string_view support to the strings currently (for C++17 support); one thing it highlights is how much more powerful the inbuilt String types are, and how much string_view is a workaround for a problem in C++'s string design which other string libraries - not just ours, but ours is IMO very good - do not suffer from.
- mpweiher 8y ago> ..Obj-C... assumed Unicode...use 2 byte encodings That's actually not entirely accurate. NSString in Foundation was (and is, sort of) a class cluster that is quite flexible and agnostic about the actual encoding of the representations. I remember this quite clearly, because I fleshed out some of the missing NSString pieces in libFoundation, which was a fairly clean/simple implementation of the OPENSTEP Foundation spec. What locked in the representation was CoreFoundation, Apple's C re-implementation of Foundation for the Carbon crowd, who apparently wouldn't countenance even linking against Objective-C, even if hidden behind a C facade. So instead of taking the tried-and-true, fast and flexible Foundation and adding some wrappers, the existing Foundation had to be abandoned and rebuilt on top of a newly-written much inferior object-oriented C library. With that, we got much worse performance[1], much reduced flexibility in terms of representation and locking away of flexibility. Just as an example, the binary plist format is quite capable, but almost all of that capability is lost because it's hidden behind a monolithic C API. Now there are some issues in the API, for example just a single "length" attribute that clashes with NSData's length and makes it...challenging...to present both Byte and String faces at the same time, making for a lot of otherwise unnecessary copying and conversion. [1] https://groups.google.com/group/comp.sys.next.advocacy/browse_thread/thread/b8e4270b5c44d004/701f32da45fc84fe?lnk=st&q=Quartz+buck+group%3Acomp.sys.next.*# https://groups.google.com/group/comp.sys.next.advocacy/brows...
- markdog12 8y ago> Go, Dart, Rust and Zig all return to single byte encodings In Dart, Strings are UTF-16 because that's what they are in JavaScript. https://www.dartlang.org/guides/language/language-tour#strings https://www.dartlang.org/guides/language/language-tour#strin... "A Dart string is a sequence of UTF-16 code units" You can get them as "runes": https://www.dartlang.org/guides/language/language-tour#runes https://www.dartlang.org/guides/language/language-tour#runes
- jstimpfle 8y agohttps://utf8everywhere.org https://utf8everywhere.org
- nwellnhof 8y ago> Java, C#, Obj-C, Javascript and Python2 were all written when it was assumed Unicode would have no more than 65536 characters. That's not true for C# and Python2. While both were first released in 2000 and Unicode code points beyond the BMP first appeared with Unicode 3.1.0 in 2001, the foundation for additional Unicode planes was laid as early as 1996 with Unicode 2.0.0.