5 ms·
Sadly, the pervasiveness of JavaScript means that UTF-16 interoperability will be needed as least as long as the Web is alive. JavaScript strings are fundamenta
by pcwalton 15y ago
Sadly, the pervasiveness of JavaScript means that UTF-16 interoperability will be needed as least as long as the Web is alive. JavaScript strings are fundamentally UTF-16. This is why we've tentatively decided to go with UTF-16 in Servo (the experimental browser engine) -- converting to UTF-8 every time text needed to go through the layout engine would kill us in benchmarks.
For new APIs in which legacy interoperability isn't needed, I completely approve of this document.
- nknight 15y ago> converting to UTF-8 every time text needed to go through the layout engine would kill us in benchmarks So what? Is your goal to create useful software, or win at worthless benchmarks?
- pcwalton 15y agoWell, we're talking about DOM manipulation performance here. Pages that use DOM manipulation heavily will see a potentially-unacceptable performance loss if text always has to be converted to UTF-8. Is fast DOM manipulation important? Given that the only way for the sole scripting language on the Web to display anything or interact with the user is through DOM manipulation, I think it's worth optimizing every cycle...
- ori_b 15y agohttp://www.utf8everywhere.org/#faq.cvt.perf http://www.utf8everywhere.org/#faq.cvt.perf If the function you're calling with UTF8 is non-trivial, converting a few dozen bytes is unlikely to make a significant difference. Benchmark it, of course, but don't be surprised if you don't need to care. Modifying the DOM is probably going to be non-trivial.
- pcwalton 15y agoSee my comment above; I can construct cases that make this sort of conversion have unacceptable overhead. Would it matter in a real-world setting? I can't say for sure, because nobody I know of has tried making a production-quality UTF-8 web layout engine. But, in my mind, none of the benefits of UTF-8 (memory usage being the main one in a browser [1]) outweigh the performance risks of doing conversion. And the risk is real. [1]: Note that you still need UTF-16 anyway, for interoperability with JavaScript. So using UTF-8 might even lead to worse memory usage, due to the necessity of duplicating strings, than a careful UTF-16-everywhere scheme that takes advantage of string buffer sharing between the JS heap and the layout engine heap would.
- ori_b 15y agoIs string storage really that big a portion of the browser's memory usage? I find it hard to believe that my browser is currently storing nearly a gigabyte of text right now I'm not saying that there wouldn't be performance overhead, but would it be significant? I would be surprised if it made a big difference. For legacy reasons, there probably isn't a point in changing it. But I'd be surprised if performance reasons turned out to be gating.
- khuey 15y agoUTF-8 -> UTF-16 conversion was at one point a noticeable fraction of Firefox's startup time. https://bugzilla.mozilla.org/show_bug.cgi?id=506431 https://bugzilla.mozilla.org/show_bug.cgi?id=506431 Since then we've done things such as fast-path ASCII -> UTF-16 conversion with SSE2 instructions. Converting a few dozen bytes is unlikely to make a significant difference, but often one needs to deal with more than a few dozen bytes.
- ybungalobill 15y agoI believe the authors wrote it clear enough that they don't rule out UTF-16 completely: > We believe that all other encodings of Unicode (or text, in general) belong to rare edge-cases of optimization and should be avoided by mainstream users. So if you're writing a browser that must use UTF-16 in its Javascript engine due to dumb standards... it is a reasonable performance optimizations to use UTF-16 for your strings. But how many people write Javascript engines?
- twoodfin 15y agoIt's not the conversion that kills you, it's the memory allocation. If you're trying to interact with a "chatty" UTF-16 API, the difference between using UTF-8 and UTF-16 in your implementation could the difference between passing a pointer (a cycle or two) and allocating/deallocating a buffer (potentially hundreds of cycles) per call. There are obviously ways around this. The API could be rewritten to exchange strings less frequently or use static strings that could be replaced with handles. You can try to be clever about your buffer allocation and share one amongst all calls (but watch out for threading issues!) You could write your own allocator. But all this plumbing just increases complexity and the risk of bugs, along with adding its own performance cost. I'm not arguing against UTF-8 as the preferred encoding for many future applications, but the "minimal overhead" example given in the manifesto isn't particularly convincing.
- ybungalobill 15y agoI agree in principle. But "chatty" APIs usually work with short strings, in which case you can use stack allocations in those performance critical calls.
- gavanwoolery 15y agoI agree with you in concept, but the Web is actually not strictly tied to javascript. In the short term (say, next decade), JS is probably not going anywhere, but in the longer term I hope someone creates a more well-thought-out replacement. (And for the record, I kind of like javascript, just a few things I would change with it.)
- daeken 15y agoThe likelihood of Javascript becoming non-dominant in the next 20 years is pretty low, I think. Rather than replacing it wholesale, you'll simply see it turn into something better over time.
- danssig 15y agoYes, like a proper VM that we can compile our preferred language to. It is ridiculous that the "language of the web" is something that is forced upon us instead of chosen. The number of languages that compile to JS illustrate the point nicely.
- jacobolus 15y agoAs long as you don’t care about errors, converting to UTF-8 is quite fast. Just use native calls: var decode = function (bytes) { return decodeURIComponent(escape(bytes)); } var encode = function (string) { return unescape(encodeURIComponent(string)); } (Definitely test it before wailing about benchmarks. My guess is that whatever else you’re doing is likely much slower.) If you do care about errors, or especially if you need to deal w/ UTF-8 streams that might be chopped mid-character, use something like https://github.com/gameclosure/js.io/blob/master/packages/std/utf8.js https://github.com/gameclosure/js.io/blob/master/packages/st...
- pcwalton 15y agoI think you misunderstand -- I'm referring to the actual systems-level implementation of the browser engine itself. I'm not talking about the implementation of web apps. Consider a pattern like this: A page calls document.createElement(), adds a large text node (say, the collected works of Shakespeare in text form) to it, calls window.getComputedStyle() on that element, then throws the element away. This series of DOM manipulations must go through the layout engine. If the layout engine knows only UTF-8, then the layout engine has to convert the collected works of William Shakespeare from UTF-16 to UTF-8 for no reason (as it needs an up-to-date DOM to perform CSS selector matching for the getComputedStyle() call). There is no reason to do that when it could just use UTF-16 instead and save itself the trouble.
- JoshTriplett 15y agoWhat part of that process requires UTF-16? JavaScript doesn't require UTF-16; it just requires Unicode. You could use UTF-8 in your JavaScript implementation as well.
- khuey 15y agoJS does require UTF-16, because the surrogate pairs of non-BMP characters are separable in JS strings. '<non-BMP character>'.length == 2 '<non-BMP character>'[0] == first codepoint in the surrogate pair '<non-BMP character>'[1] == second codepoint in the surrogate pair Any JS implementation using UTF-8 would have to convert to UTF-16 for proper answers to .length and array indexing on strings.
- lambda 15y agoYeah, it's really sad the number of legacy APIs which have standardized on UTF-16. The Windows API calles UTF-16 "Unicode". Most Mac OS X APIs use UTF-16. JavaScript and Java both use UTF-16. ICU uses UTF-16. So while UTF-8 is technically superior in almost every way, it's going to be an uphill battle to standardize on it. I appreciate that new languages like Rust and Go made the choice of UTF-8 as their native text encoding. But there's a lot of inertia for UTF-16, and I'm not sure it'll be easy to ever get free of it.
- antidoh 15y agoSome old languages have also made that choice, albeit recently in Python's case: http://www.python.org/dev/peps/pep-3120/ http://www.python.org/dev/peps/pep-3120/
- sho_hn 15y agoHowever, when it comes to the internal representation of text, things are quite complex as of Python 3.2: http://www.python.org/dev/peps/pep-0393/ http://www.python.org/dev/peps/pep-0393/
- ruediger 15y agoICU has some UTF-8 functionality. But ICU is really ugly. http://userguide.icu-project.org/strings/utf-8 http://userguide.icu-project.org/strings/utf-8
- brigade 15y agoOS X APIs generally use NSString/CFString, which hide the actual encoding of the string; they can be any encoding at all internally.
- lambda 15y agoWhile they could, in theory, hide the actual encoding, the APIs all refer to UTF-16 code units as "characters;" so while they could use UTF-8 as the internal encoding, you need to use and understand UTF-16 in order to interact with them properly. When you ask for the "length" of a string, you are told the number of UTF-16 code units. When you get a character at an index, you get the UTF-16 code unit. That's what I mean when I say the APIs use UTF-16; everything in the API that deals with individual "chracters" is actually referring to UTF-16 code units. The same is true of JavaScript; while you could technically implement the strings however you want, the APIs are all oriented around UTF-16 code units. And the Windows API as well, is all built around UTF-16 code units. The problem with all of these APIs is that they make the mistake of conflating characters and code units. They all make the assumption that a character consists of a single, fixed width integer, of some given size (16 bits in the case of UTF-16). It is better to distinguish between indexing in code units (such as bytes in UTF-8 or 16 bit integers in UTF-16) and indexing in code points, or glyphs, or whatever higher level concept you are talking about. Really, for anything higher than the code unit level, you should be dealing with variable-length strings, and not try to force that into fixed length units. With UTF-8, there's no temptation to treat a single code unit as being an independently meaningful entity, as that assumption breaks down as soon as you get past the ASCII range; while with UTF-16, it's easy to make that mistake, since it holds true for everything in the Basic Multilingual Plane, which contains most characters you're likely to encounter on a day to day basis.
- petercooper 15y agoA recent discussion about this involving Brendan Eich and a few other JS folk: https://gist.github.com/1850768 https://gist.github.com/1850768
- soc88 15y agoClaims like this are exactly the reason why we will still be stuck with multiple useless encodings in 2030. I wonder how long it will take until people find their balls and decide to move towards the right direction.