3 ms·

Hey Bob, I super love this series. Everyone I know who's into PL likes it, so great work :) A little thing in your section about the different UTFs: I've hear

Hey Bob, I super love this series. Everyone I know who's into PL likes it, so great work :) A little thing in your section about the different UTFs: I've heard UTF-16 is a lot better than UTF-8 for East Asian languages because pretty much all those codepoints are multibyte. I don't _necessarily_ think anything you wrote was invalid, but just FYI.

munificent 8y ago
Hmm, interesting. I guess they'd likely end up three bytes in UTF-8. Do you have any details you can point me at?
- camgunz 8y ago
  Nothing like a paper or anything, but just kind of anecdotally I downloaded 西遊記 from Project Gutenberg [1] and measured it. In UTF-8 it's 2,236,564 bytes, in UTF-16 it's 1,554,998 bytes. [1] https://www.gutenberg.org/files/23962/23962-0.txt https://www.gutenberg.org/files/23962/23962-0.txt
ufo 8y ago
However, this is only the case if you have just a big blob of text and UTF-8 might be better if the east asian text is interspersed with ASCII characters. For example, HTML with east asian text often is smaller when encoded with UTF-8, because a significant portion of the content is HTML markup.
- camgunz 8y ago
  Right well again, if your average East Asian codepoint is 3 bytes in UTF-8, and your average markup codepoint is one byte, then as the percentage of markup in your corpus rises you'll grow asymptotically towards 1. But not all text contains markup ;) Consider any database storage for example, text in a game, e-books, visual novels. I guess anything that isn't the web or JSON is what I'm saying -- which is a lot.
- a1369209993 8y ago
  Also bear in mind that gzip (or any non-toy compression algorithm) will do much better than either on any non-contrived input.
mwcampbell 8y ago
Also on UTF-16: > Alas, if your language needs to run on or interoperate with the browser, the JVM, or the CLR, you might be stuck with [UTF-16] And Windows. Win32 and COM APIs use UTF-16 natively, so if you want to avoid lots of conversions on Windows, UTF-16 is the way to go. Maybe the best solution would be an abstraction that lets you choose the internal string encoding at compile time.

munificent 8y ago

Hmm, interesting. I guess they'd likely end up three bytes in UTF-8. Do you have any details you can point me at?

camgunz 8y ago
Nothing like a paper or anything, but just kind of anecdotally I downloaded 西遊記 from Project Gutenberg [1] and measured it. In UTF-8 it's 2,236,564 bytes, in UTF-16 it's 1,554,998 bytes. [1] https://www.gutenberg.org/files/23962/23962-0.txt https://www.gutenberg.org/files/23962/23962-0.txt

ufo 8y ago

However, this is only the case if you have just a big blob of text and UTF-8 might be better if the east asian text is interspersed with ASCII characters. For example, HTML with east asian text often is smaller when encoded with UTF-8, because a significant portion of the content is HTML markup.

camgunz 8y ago
Right well again, if your average East Asian codepoint is 3 bytes in UTF-8, and your average markup codepoint is one byte, then as the percentage of markup in your corpus rises you'll grow asymptotically towards 1. But not all text contains markup ;) Consider any database storage for example, text in a game, e-books, visual novels. I guess anything that isn't the web or JSON is what I'm saying -- which is a lot.
a1369209993 8y ago
Also bear in mind that gzip (or any non-toy compression algorithm) will do much better than either on any non-contrived input.

mwcampbell 8y ago

Also on UTF-16: > Alas, if your language needs to run on or interoperate with the browser, the JVM, or the CLR, you might be stuck with [UTF-16] And Windows. Win32 and COM APIs use UTF-16 natively, so if you want to avoid lots of conversions on Windows, UTF-16 is the way to go. Maybe the best solution would be an abstraction that lets you choose the internal string encoding at compile time.