4 ms·

> I do not think a suggestion to "encourage UTF-8 for new formats and APIs" can be said to "literally breaking the Web Platform" Perhaps you don’t see it that

> I do not think a suggestion to "encourage UTF-8 for new formats and APIs" can be said to "literally breaking the Web Platform" Perhaps you don’t see it that way, but we have to respect when others do. From the way I understand things, because JS strings (DOMString [1]) are already represented as UTF-16/WTF-16 internally, making the component model strings only UTF-8 puts certain languages in disadvantage against others specially regarding speed (because they’ll need to do extra effort when processing strings in order to be fully compliant, while the UTF-8 ones wouldn’t need to do it). This was the case for AssemblyScript, Javascript, Java and other JVM-related languages (Kotlin, Scala, ...). Specifically for a language such as AssemblyScript, for which its aim is to be completely complaint with JS, tiny and fast to execute I can certainly understand why that was a concern and why being dismissed could cause certain frustration. [1] https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String#utf-16_characters_unicode_codepoints_and_grapheme_clusters https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...

IshKebab 4y ago
Right but it's pretty clear that UTF-8 is the better, de facto choice. JS strings only use UTF-16 for historic reasons. If designed today they would definitely use UTF-8. Since some languages use UTF-8 and some use UTF-16 you're always going to have some languages doing conversions so it makes sense just to pick the best option surely?
- dcode 4y ago
  Except that picking the most restrictive option prevents two components that are using the same less restrictive semantics to communicate with each other securely (without throwing exceptions or silently mutating data), e.g. Java <-> Java, AssemblyScript <-> JavaScript. To illustrate, if one would design a Component Model for Java, restricting to UTF-8 would make the Component Model for Java a hazard for Java. The same effect happens in a multi-language Component Model, where some languages then work and others don't. Hence "pick the best option" falls short. The argument in all these questionably stonewalled discussions is basically to allow these languages and use cases to exist, which could be as trivial as to make UTF-8 the default if the WASI folks so wish, but also have a Boolean flag for "don't eagerly mutate on WTF-16 pass-through". Yet, even though trivial and rather obvious, this has been fought relentlessly since 2017, and surely one has to wonder why this vehemence.
  - IshKebab 4y ago
    UTF-8 is not "more restrictive". I'm not sure what you're talking about.
    - dcode 4y ago
      The value spaces are in fact asymmetric. In Unicode jargon, UTF-8 is a "list of Unicode scalar values" while WTF-16, i.e. UTF-16 as seen in practice, is a "list of Unicode code points (except surrogate pairs)". Unicode code points are a superset of Unicode scalar values, aka "list of Unicode Code Points except surrogate code points", hence conversion from WTF-16 to UTF-8 is lossy. In WTF-16 languages, this is not a problem because the system is designed for WTF-16, but it becomes one if the value space is restricted further, here to UTF-8. That's why, in such mixed systems, https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/ is typically used. Not UTF-8. And Wasm is such a mixed system.
Jasper_ 4y ago
Conversions like this are basically as fast as a memcpy, which will always be required in this system. Did either you or dcode try actually profiling the extra time spent?
- dcode 4y ago
  Despite that speed is not the critical concern (asymmetric value spaces are), UTF-8 must guard against invalid byte sequences, while WTF-16, where all possible values are valid as long as byte length is a multiple of 2, does not. In practice, the guard in UTF-8 is part of the copy loop over a boundary, typically from untyped memory to untyped memory, while WTF-16 can indeed just memcpy. Unless SIMD can be utilized, the difference is about that of a loop over a load into a branch vs. a memcpy, in case this helps to quantify. Expect an additional final memcpy if the UTF-8 copy should fail before storing anything to the receiver's memory.

IshKebab 4y ago

Right but it's pretty clear that UTF-8 is the better, de facto choice. JS strings only use UTF-16 for historic reasons. If designed today they would definitely use UTF-8. Since some languages use UTF-8 and some use UTF-16 you're always going to have some languages doing conversions so it makes sense just to pick the best option surely?

dcode 4y ago
Except that picking the most restrictive option prevents two components that are using the same less restrictive semantics to communicate with each other securely (without throwing exceptions or silently mutating data), e.g. Java <-> Java, AssemblyScript <-> JavaScript. To illustrate, if one would design a Component Model for Java, restricting to UTF-8 would make the Component Model for Java a hazard for Java. The same effect happens in a multi-language Component Model, where some languages then work and others don't. Hence "pick the best option" falls short. The argument in all these questionably stonewalled discussions is basically to allow these languages and use cases to exist, which could be as trivial as to make UTF-8 the default if the WASI folks so wish, but also have a Boolean flag for "don't eagerly mutate on WTF-16 pass-through". Yet, even though trivial and rather obvious, this has been fought relentlessly since 2017, and surely one has to wonder why this vehemence.
- IshKebab 4y ago
  UTF-8 is not "more restrictive". I'm not sure what you're talking about.
  - dcode 4y ago
    The value spaces are in fact asymmetric. In Unicode jargon, UTF-8 is a "list of Unicode scalar values" while WTF-16, i.e. UTF-16 as seen in practice, is a "list of Unicode code points (except surrogate pairs)". Unicode code points are a superset of Unicode scalar values, aka "list of Unicode Code Points except surrogate code points", hence conversion from WTF-16 to UTF-8 is lossy. In WTF-16 languages, this is not a problem because the system is designed for WTF-16, but it becomes one if the value space is restricted further, here to UTF-8. That's why, in such mixed systems, https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/ is typically used. Not UTF-8. And Wasm is such a mixed system.

Jasper_ 4y ago

Conversions like this are basically as fast as a memcpy, which will always be required in this system. Did either you or dcode try actually profiling the extra time spent?

dcode 4y ago
Despite that speed is not the critical concern (asymmetric value spaces are), UTF-8 must guard against invalid byte sequences, while WTF-16, where all possible values are valid as long as byte length is a multiple of 2, does not. In practice, the guard in UTF-8 is part of the copy loop over a boundary, typically from untyped memory to untyped memory, while WTF-16 can indeed just memcpy. Unless SIMD can be utilized, the difference is about that of a loop over a load into a branch vs. a memcpy, in case this helps to quantify. Expect an additional final memcpy if the UTF-8 copy should fail before storing anything to the receiver's memory.