4 ms·
Definitely. I would be interested to know of any good statistics on data size of real-world non-European language webpages in UTF-8 vs. UTF-16 and also with/wi
by ssalazar 10y ago
Definitely.
I would be interested to know of any good statistics on data size of real-world non-European language webpages in UTF-8 vs. UTF-16 and also with/without compression.
Much of the markup will be smaller in UTF-8 but actual text content would be smaller in UTF-16.
- creshal 10y agoAnd both end up being fed to optimized-deflate-encoder-of-the-week and have negligible size differences once compressed.
- dom0 10y agoSee comment above parent: > This happens for pure text[nb 2] but actual documents often contain enough spaces and line terminators, numbers (digits 0–9), and HTML or XML or wiki markup characters, that they are shorter in UTF-8. For example, both the Japanese UTF-8 and the Hindi Unicode articles on Wikipedia take more space in UTF-16 than in UTF-8.[nb 3]
- jcranmer 10y agoThere was an experiment (see https://bug416411.bmoattachments.org/attachment.cgi?id=303074 https://bug416411.bmoattachments.org/attachment.cgi?id=30307... for the results). that switched all the UTF-16 internal strings in Firefox to UTF-8. It found that UTF-8 lowered memory usage, even on text-heavy East Asian pages. Keep in mind that things like tag names or attribute names are all interned, so there's no space savings from, say, compressing <img src="">.