5 ms·

Documentation is sparse on why emscripten compiled wasms are incompatible. Please provide more information

kentonv 8y ago
It's a very minor incompatibility: Traditionally, all WebAssembly modules are essentially eval()ed. You need JavaScript to download the module into an ArrayBuffer or the like, then pass that to the WebAssembly API to compile it. However, in Cloudflare Workers, we didn't want you to have to fetch WebAssembly remotely at startup. Instead, you upload your WASM module to the Cloudflare configuration UI/API together with your JavaScript code. At startup, the WASM is compiled and the resulting `WebAssembly.Module` appears as a global variable in your script, which you can then instantiate. Emscripten normally automatically generates JavaScript for you to load your WASM. But Emscripten's generated script doesn't understand this delivery model where the module shows up as a global variable. It should be trivial to add support, but it would be awkward for us to try to submit a patch upstream without the functionality being public yet.
- writepub 8y ago
  Understood. While the patch is being mainlined, it would benefit emscripten users targeting CloudFlare to use whatever you have working with emscripten's build workflow. Please consider releasing this repo for the interim
  - kentonv 8y ago
    We haven't actually tried making any changes to Emscripten yet. We've only used lower-level tools to build things more manually. There is a working example of this here: https://github.com/cloudflare/cloudflare-workers-wasm-demo https://github.com/cloudflare/cloudflare-workers-wasm-demo
- buu700 8y ago
  Actually, emscripten has a built-in flag (SINGLE_FILE) to embed the wasm code as base64 (which I'm the author of, incidentally).
  - hackcasual 8y ago
    May I recommend also supporting hex encoding the WASM? I think you'll find that while your file size is larger, it's actually significantly smaller once gzipped. Edit: Just tested this on https://public.tableau.com/vizql/v_public-release1809140800/javascripts/runtimeweb.wasm https://public.tableau.com/vizql/v_public-release1809140800/... runtimeweb.wasm 2,484,043 runtimewebwasm.b64 3,355,640 runtimwebwasm.hex 4,968,086 runtimewebwasm.b64.gz 974,065 runtimewebwasm.hex.gz 701,052 runtimewebwasm.b64.br 718,918 runtimewebwasm.hex.br 466,221 So with gzip -9, hex encoding is 72% of the size, with brotli (defaults) size is 65% of the size.
    - buu700 8y ago
      Hmm, thanks for the tip, and thanks for testing that out so I don't have to! It hadn't occurred to me that that would be the case, but it makes some sense after reading through https://stackoverflow.com/q/38124361/459881 https://stackoverflow.com/q/38124361/459881. I'll go ahead and open an issue with emscripten about this.
      - hackcasual 8y ago
        Yep, it's one of those counter intuitive things. Another benefit from using hex is the decoder is a lot simpler and easier for the JIT to vectorize.
  - kentonv 8y ago
    That actually doesn't quite solve the problem as we disallow dynamic code evaluation. This is for a couple reasons: * Security: In case of an attack we need to be able to call up the code to do forensics, which is hard if it was downloaded dynamically at runtime. * Optimizations: We want to keep the ability to pre-compile code e.g. into V8 code cache format before distributing it to the edge.
    - buu700 8y ago
      Ah, I thought that might've been part of it since you brought up the comparison to eval; just wanted to clarify that there are other options for avoiding reliance on external subresources. which is hard if it was downloaded dynamically at runtime Not sure if you mean something other than what this sounds like (I haven't used CF Workers), but just to clarify further, the option I mentioned causes emscripten to emit one single JS file with no external code downloaded at run time. (In any case, that performance optimization alone certainly sounds like a good reason for the separation.)
- curiousdannii 8y ago
  What about Module.instantiateWasm()? That should let you provide a pre-compiled or even pre-instantiated module.
deleted 8y ago
[deleted]

It's a very minor incompatibility: Traditionally, all WebAssembly modules are essentially eval()ed. You need JavaScript to download the module into an ArrayBuffer or the like, then pass that to the WebAssembly API to compile it. However, in Cloudflare Workers, we didn't want you to have to fetch WebAssembly remotely at startup. Instead, you upload your WASM module to the Cloudflare configuration UI/API together with your JavaScript code. At startup, the WASM is compiled and the resulting `WebAssembly.Module` appears as a global variable in your script, which you can then instantiate. Emscripten normally automatically generates JavaScript for you to load your WASM. But Emscripten's generated script doesn't understand this delivery model where the module shows up as a global variable. It should be trivial to add support, but it would be awkward for us to try to submit a patch upstream without the functionality being public yet.

writepub 8y ago
Understood. While the patch is being mainlined, it would benefit emscripten users targeting CloudFlare to use whatever you have working with emscripten's build workflow. Please consider releasing this repo for the interim
- kentonv 8y ago
  We haven't actually tried making any changes to Emscripten yet. We've only used lower-level tools to build things more manually. There is a working example of this here: https://github.com/cloudflare/cloudflare-workers-wasm-demo https://github.com/cloudflare/cloudflare-workers-wasm-demo
buu700 8y ago
Actually, emscripten has a built-in flag (SINGLE_FILE) to embed the wasm code as base64 (which I'm the author of, incidentally).
- hackcasual 8y ago
  May I recommend also supporting hex encoding the WASM? I think you'll find that while your file size is larger, it's actually significantly smaller once gzipped. Edit: Just tested this on https://public.tableau.com/vizql/v_public-release1809140800/javascripts/runtimeweb.wasm https://public.tableau.com/vizql/v_public-release1809140800/... runtimeweb.wasm 2,484,043 runtimewebwasm.b64 3,355,640 runtimwebwasm.hex 4,968,086 runtimewebwasm.b64.gz 974,065 runtimewebwasm.hex.gz 701,052 runtimewebwasm.b64.br 718,918 runtimewebwasm.hex.br 466,221 So with gzip -9, hex encoding is 72% of the size, with brotli (defaults) size is 65% of the size.
  - buu700 8y ago
    Hmm, thanks for the tip, and thanks for testing that out so I don't have to! It hadn't occurred to me that that would be the case, but it makes some sense after reading through https://stackoverflow.com/q/38124361/459881 https://stackoverflow.com/q/38124361/459881. I'll go ahead and open an issue with emscripten about this.
    - hackcasual 8y ago
      Yep, it's one of those counter intuitive things. Another benefit from using hex is the decoder is a lot simpler and easier for the JIT to vectorize.
- kentonv 8y ago
  That actually doesn't quite solve the problem as we disallow dynamic code evaluation. This is for a couple reasons: * Security: In case of an attack we need to be able to call up the code to do forensics, which is hard if it was downloaded dynamically at runtime. * Optimizations: We want to keep the ability to pre-compile code e.g. into V8 code cache format before distributing it to the edge.
  - buu700 8y ago
    Ah, I thought that might've been part of it since you brought up the comparison to eval; just wanted to clarify that there are other options for avoiding reliance on external subresources. which is hard if it was downloaded dynamically at runtime Not sure if you mean something other than what this sounds like (I haven't used CF Workers), but just to clarify further, the option I mentioned causes emscripten to emit one single JS file with no external code downloaded at run time. (In any case, that performance optimization alone certainly sounds like a good reason for the separation.)
curiousdannii 8y ago
What about Module.instantiateWasm()? That should let you provide a pre-compiled or even pre-instantiated module.

deleted 8y ago

[deleted]