3 ms·

Hacker News is a super-text-heavy site, but even with this extreme example, the favicon alone means that 20% of the page weight is binary data. For any normal s

Hacker News is a super-text-heavy site, but even with this extreme example, the favicon alone means that 20% of the page weight is binary data. For any normal site, binary data outweighs ASCII by several orders of magnitude. In either case though, we’re talking about a few kilobytes, which I wouldn’t consider “big” — like even if you wanted to write an HN reader app for your esp32 microcontroller, HN being served in UTF32 instead of UTF8 probably wouldn’t be the biggest obstacle :P

vlovich123 4y ago
> This however is not the case when Japanese text is mixed with ASCII control structures. For instance XML or HTML documents include enough in-line control data that is in the ASCII range that UTF-8 becomes more efficient as a format compared to UTF-16 (before compression). For instance the front page of the Japanese Wikipedia is 92KB in UTF-8 and 166KB in UTF-16. https://lucumr.pocoo.org/2014/1/9/ucs-vs-utf8/ https://lucumr.pocoo.org/2014/1/9/ucs-vs-utf8/ The favicon btw is cached and amortized across all HN pages whereas the text is not. I forget where I read this but about a decade or so I remember reading a paper or watching a video (maybe from the Azul folks?) that looked into JVM memory usage and a good chunk of it was strings and the ucs2 encoding was a problem. That’s why even languages that are nominally utf16/32 as the native type will frequently auto detect and special cases latin1 strings (python, js, etc). The other piece of it is that strings are copied around more and processed differently from images. The knock on effects of utf32 can be quite unfortunate (ie rendering your html document is meaningfully slower which you care about even though by weight your images take longer to transfer and show)
- kristianp 4y ago
  > That’s why even languages that are nominally utf16/32 as the native type will frequently auto detect and special cases latin1 strings (python, js, etc). I feel that this is worth a blog post in itself. I remember years ago comparing Go and C# when processing some large mostly-ascii files. The C# program was faster, much to my surprise, despite storing strings natively in UTF-16. (I don't remember the implementation details, so it may have been an artifact of my implementations).

vlovich123 4y ago

> This however is not the case when Japanese text is mixed with ASCII control structures. For instance XML or HTML documents include enough in-line control data that is in the ASCII range that UTF-8 becomes more efficient as a format compared to UTF-16 (before compression). For instance the front page of the Japanese Wikipedia is 92KB in UTF-8 and 166KB in UTF-16. https://lucumr.pocoo.org/2014/1/9/ucs-vs-utf8/ https://lucumr.pocoo.org/2014/1/9/ucs-vs-utf8/ The favicon btw is cached and amortized across all HN pages whereas the text is not. I forget where I read this but about a decade or so I remember reading a paper or watching a video (maybe from the Azul folks?) that looked into JVM memory usage and a good chunk of it was strings and the ucs2 encoding was a problem. That’s why even languages that are nominally utf16/32 as the native type will frequently auto detect and special cases latin1 strings (python, js, etc). The other piece of it is that strings are copied around more and processed differently from images. The knock on effects of utf32 can be quite unfortunate (ie rendering your html document is meaningfully slower which you care about even though by weight your images take longer to transfer and show)

kristianp 4y ago
> That’s why even languages that are nominally utf16/32 as the native type will frequently auto detect and special cases latin1 strings (python, js, etc). I feel that this is worth a blog post in itself. I remember years ago comparing Go and C# when processing some large mostly-ascii files. The C# program was faster, much to my surprise, despite storing strings natively in UTF-16. (I don't remember the implementation details, so it may have been an artifact of my implementations).