5 ms·

Requiem for hacks upon hacks all the way down the stack to make historic brain farts work. A sensible way forward would be to deprecate APIs for direct UTF-16

Requiem for hacks upon hacks all the way down the stack to make historic brain farts work. A sensible way forward would be to deprecate APIs for direct UTF-16 code unit access, but implement them for backward compatibility on top of an internal UTF-8 representation. On both sides. We've lived with JS bloat for two decades, you think a few string copies/conversions are gonna kill us? Any non-toy uses of WebAssembly are gonna be new developments. Old shit that nobody is gonna run on WebAssembly other than to go "yeah, huh, it runs on WebAssembly" and then never use it again, doesn't need to run great.

alexvitkov 3y ago
A few copies aren't gonna kill us, but this may very well might, if charAt() is now O(n): for (int i = 0; i < str.length(); i++) doSomethingWith(str.charAt(i)); UTF-8 just a text encoding. If you're making something new, yeah, it's the obvious choice, but it's not better enough to justify breaking all sorts of shit just to switch over.
- davexunit 3y ago
  A copy for every string passed to the DOM API, to name just one thing, will be a significant limiting factor.
  - alexvitkov 3y ago
    There are some Rust front-end web frameworks that presumably manipulate the DOM, and in C++/Rust to pass a string to JS you need to run a TextDecoder over your WASM memory, so it's probably not a deal breaker. But like... if you're writing a website, just use JavaScript.
    - davexunit 3y ago
      C++/Rust use WASM linear memory, but this article is about reference types via WASM GC. UTF-8 data in an (array i8) or UTF-16 data in an (array i16) are opaque to the host.
      - alexvitkov 3y ago
        Yeah, and you still have to marshal strings at the JS/WASM boundry, same as if you used (array i8/16) over JS strings in Java. In the case of non-managed strings, this overhead hasn't been big enough to stop people from writing fairly fast (by Web standards) frontend frameworks in Rust.
- Ygg2 3y ago
  That code makes zero sense in Unicode. First question how are representing your umlauts, followed by Zero Join Width Characters. You never work on characters, you work on grapheme clusters or whatnot but never characters.
  - alexvitkov 3y ago
    I'm not advocating you write this, I'm saying people have written it, probably hundreds of thousands of times, and if charAt() becomes O(n) instead of O(1), this code suddenly hangs your CPU for 10 seconds on a long string, thus you can't really swap out UTF-16 for UTF-8 transparently.
    - Ygg2 3y ago
      Your point doesn't stand for UTF-16 either. It's not a fixed length encoding either. It's broken in UTF-16 as well. It was always O(n). Of course assuming you aren't using UTF-32, which has its own set of problems (BE or LE), and sees little usage outside of China.
      - taway1237 3y ago
        ...it's not O(n). Many languages, JS, Java and C# included, have O(1) access to a character at a given position. You correctly note that it won't work well with international strings, but GP is right that A LOT of code like this was written by western ASCII-brained developers.
      - alexvitkov 3y ago
        Haven't used Java in a while but I believe charAt() returns a UTF-16 codepoint and is constant time access. So something like the above works not only for ASCII, as well as for the majority of Western languages and special characters you may encounter on a day to day basis.
        Ygg2 3y ago
        It's constant time iff you ignore surrogate pairs and Unicode. By that logic UTF8 is constant time if you ignore anything not ASCII because most text is in English. Saying it works fine if you ignore errors and avoid edge cases is just a clever rephrashing of it worked on my machine. Plus Emojis are Unicode U+1F600 and above, so even in Western language you are bound to find such "exceptions" .
kevingadd 3y ago
The amount of memory and CPU overhead involved in sending strings across the wasm/JS boundary to do something like put text in a textarea is a lot bigger than you might think. It's really severe.
azakai 3y ago
> Any non-toy uses of WebAssembly are gonna be new developments. Major uses of WebAssembly include things like Photoshop, Unity, and many other large existing codebases.

alexvitkov 3y ago

A few copies aren't gonna kill us, but this may very well might, if charAt() is now O(n): for (int i = 0; i < str.length(); i++) doSomethingWith(str.charAt(i)); UTF-8 just a text encoding. If you're making something new, yeah, it's the obvious choice, but it's not better enough to justify breaking all sorts of shit just to switch over.

davexunit 3y ago
A copy for every string passed to the DOM API, to name just one thing, will be a significant limiting factor.
- alexvitkov 3y ago
  There are some Rust front-end web frameworks that presumably manipulate the DOM, and in C++/Rust to pass a string to JS you need to run a TextDecoder over your WASM memory, so it's probably not a deal breaker. But like... if you're writing a website, just use JavaScript.
  - davexunit 3y ago
    C++/Rust use WASM linear memory, but this article is about reference types via WASM GC. UTF-8 data in an (array i8) or UTF-16 data in an (array i16) are opaque to the host.
    - alexvitkov 3y ago
      Yeah, and you still have to marshal strings at the JS/WASM boundry, same as if you used (array i8/16) over JS strings in Java. In the case of non-managed strings, this overhead hasn't been big enough to stop people from writing fairly fast (by Web standards) frontend frameworks in Rust.
Ygg2 3y ago
That code makes zero sense in Unicode. First question how are representing your umlauts, followed by Zero Join Width Characters. You never work on characters, you work on grapheme clusters or whatnot but never characters.
- alexvitkov 3y ago
  I'm not advocating you write this, I'm saying people have written it, probably hundreds of thousands of times, and if charAt() becomes O(n) instead of O(1), this code suddenly hangs your CPU for 10 seconds on a long string, thus you can't really swap out UTF-16 for UTF-8 transparently.
  - Ygg2 3y ago
    Your point doesn't stand for UTF-16 either. It's not a fixed length encoding either. It's broken in UTF-16 as well. It was always O(n). Of course assuming you aren't using UTF-32, which has its own set of problems (BE or LE), and sees little usage outside of China.
    - taway1237 3y ago
      ...it's not O(n). Many languages, JS, Java and C# included, have O(1) access to a character at a given position. You correctly note that it won't work well with international strings, but GP is right that A LOT of code like this was written by western ASCII-brained developers.
    - alexvitkov 3y ago
      Haven't used Java in a while but I believe charAt() returns a UTF-16 codepoint and is constant time access. So something like the above works not only for ASCII, as well as for the majority of Western languages and special characters you may encounter on a day to day basis.
      - Ygg2 3y ago
        It's constant time iff you ignore surrogate pairs and Unicode. By that logic UTF8 is constant time if you ignore anything not ASCII because most text is in English. Saying it works fine if you ignore errors and avoid edge cases is just a clever rephrashing of it worked on my machine. Plus Emojis are Unicode U+1F600 and above, so even in Western language you are bound to find such "exceptions" .

kevingadd 3y ago

The amount of memory and CPU overhead involved in sending strings across the wasm/JS boundary to do something like put text in a textarea is a lot bigger than you might think. It's really severe.

azakai 3y ago

> Any non-toy uses of WebAssembly are gonna be new developments. Major uses of WebAssembly include things like Photoshop, Unity, and many other large existing codebases.