3 ms·
Yes UTF-16 / UCS-2 is a silly encoding and in retrospect it was a mistake. It matters because for awhile it was believed that 2 bytes would be enough to encode
by josephg 6y ago
Yes UTF-16 / UCS-2 is a silly encoding and in retrospect it was a mistake. It matters because for awhile it was believed that 2 bytes would be enough to encode any Unicode character. During this time, lots of important languages appeared which figured a 2 byte fixed size encoding was better than a variable length encoding. So Java, javascript, C# and others all use UCS-2. Now they suffer the downsides of both using a variable length encodings and being memory inefficient. The string.length property in all these languages is almost totally meaningless.
UTF-8 is the encoding you should generally always reach for when designing new systems, or when implementing a network protocol. Rust, Go and other newer languages all use UTF-8 internally because it’s better. Well, and in Go’s case because it’s author, Rob Pike also had a hand in inventing UTF-8.
Ironically C and UNIX, which (mostly) stubbornly stuck with single byte character encodings generally works better with UTF-8 than a lot of newer languages.