6 ms·

Wasmer: A Python library for executing WebAssembly binaries

167 points by pplonski86 8y ago

KloudTrader 8y ago
This looks really fun to use for algorithmic trading (we have to deal with and sandbox client code not written by us here at KloudTrader). Wonder how hard it is to integrate with numpy etc.
- btown 8y ago
  I'm sure there are better ways to do this, but if you really want Python client code -> WASM -> Python host, I'd imagine you could probably run https://github.com/iodide-project/pyodide https://github.com/iodide-project/pyodide within Wasmer. But you'd likely lose a lot of the performance benefits of numpy etc. being compiled for the exact intrinsics supported by your host CPU. Container-level sandboxing still sounds like a superior bet.
  - sansnomme 8y ago
    Container level sandboxing is pretty difficult to do well without hypervisor/VM support.
indentit 8y ago
This looks great, I like the simple examples which show how to get a string out from the web assembly module. One thing I didn't see in the readme or examples, however, is a way to import python functions to the web assembly modules. Is it possible, and just missing from the documentation atm?
- djrenren 8y ago
  Looks like the current code doesn't support it. https://github.com/wasmerio/python-ext-wasm/blob/master/src/instance.rs#L133-L134 https://github.com/wasmerio/python-ext-wasm/blob/master/src/... but in theory it shouldn't be too hard™
- Hywan 8y ago
  We don't support that yet, but we are working on it. It's not that easy even if the theory is kind of simple to explain. The implementation in the different backends is mostly the challenge. The Python extension uses the Wasmer runtime (https://github.com/wasmerio/wasmer https://github.com/wasmerio/wasmer) which supports 3 backends to generate executable code: Singlepass, Cranelift, and LLVM. Cranelift is the easiest one to embed, and it brings a great balance between compilation time and execution time. That's the one used in the Python extension right now. The “import functions” feature is also missing in the PHP extension (https://github.com/wasmerio/php-ext-wasm https://github.com/wasmerio/php-ext-wasm), but yeah, as I said, we are working on it. It's coming soon!
  - indentit 8y ago
    understood, thanks for the explanation - keep up the great work! is there anything external contributors can do to help? :) I didn't find any issues labelled "good first issue" but I did find a related issue[0] [0]: https://github.com/wasmerio/wasmer/issues/267 https://github.com/wasmerio/wasmer/issues/267
    - Hywan 8y ago
      The best way to help right now is to test it, and report any issues! I've been able to try it on Windows so far for instance. Also the API isn't complete. We need to be able to grow the memory for instance. We need to understand the community needs to define a roadmap for the community itself.
  - jononor 8y ago
    Once you allow to call Python, keeping things sandboxed becomes pretty hard, doesn't it?
    - Hywan 8y ago
      That's true. However, it allows to integrate your Wasm program to your favorite framework more easily. It's likely to be a small surface of code to review carefully. It's always a balance :-).
galacticdessert 8y ago
Looks good! Just for my understanding, what would be the advantage of working with wasm binaries instead of, let's say, C external libraries or DLLs via ctypes? I can think of: 1. Portability of the wasm binaries Is easier interoperability between data types an advantage as well? Ctypes can be a nightmare when passing around structs and pointers (and pointers of pointers, etc.)
- make3 8y ago
  My first impression is that the goal of this project is to run binaries that were likely built for other purposes like the web, not that they actually think that making libraries into webasm and running them into python has any particular advantages vs other options that already exist
- sanxiyn 8y ago
  There's also sandboxing. You shouldn't load untrusted DLL, but loading potentially untrusted WASM is a design goal of WebAssembly.
- flohofwoe 8y ago
  I have an actual real-world use case for this :) Shader-cross-compilation for 3D frameworks: https://floooh.github.io/2017/05/15/oryol-spirv.html https://floooh.github.io/2017/05/15/oryol-spirv.html I'm hooking python scripts as custom-build-jobs into C/C++ projects which (for instance) takes care of compiling GLSL shader code into HLSL (D3D), MSL (Metal) and various GLSL versions (for GLES2, GLES3 and desktop GL). Khronos provides C++ projects which help with this stuff: https://github.com/KhronosGroup/glslang https://github.com/KhronosGroup/glslang https://github.com/KhronosGroup/SPIRV-Cross https://github.com/KhronosGroup/SPIRV-Cross So I'm using Python to hook my custom build jobs into the C/C++ build system (via cmake), but I still need to invoke native command line tools (or DLLs) which wrap around those Khronos libraries. Currently this means building (at least) 3 executables for Windows, macOS and Linux, and "distribute" those precompiled exes with my python code. It looks like with this python+wasmer solution, I can instead compile the Khronos libraries into WASM modules, and the result runs "anywhere" where wasmer runs (e.g. also Raspberry Pi, BSDs etc...). That's really cool stuff :)
- 8y ago
the_mitsuhiko 8y ago
This is awesome! I wanted this so much. Need to play with this asap.
devwastaken 8y ago
"Safe: All calls to WebAssembly will be fast, but more importantly, completely safe and sandboxed." That's a very bold statement, I see no backup of this in the readme.
- snek 8y ago
  It's a side effect of the design of WebAssembly, as long as wasmer itself has a conformant implementation (the WASM spec itself is already formally verified).
- Hywan 8y ago
  I recommend this excellent reading, https://00f.net/2018/11/25/webassembly-doesnt-make-unsafe-languages-safe/ https://00f.net/2018/11/25/webassembly-doesnt-make-unsafe-la.... It clarifies what “safe” means in this context.

167 points by pplonski86 8y ago

This looks really fun to use for algorithmic trading (we have to deal with and sandbox client code not written by us here at KloudTrader). Wonder how hard it is to integrate with numpy etc.

btown 8y ago
I'm sure there are better ways to do this, but if you really want Python client code -> WASM -> Python host, I'd imagine you could probably run https://github.com/iodide-project/pyodide https://github.com/iodide-project/pyodide within Wasmer. But you'd likely lose a lot of the performance benefits of numpy etc. being compiled for the exact intrinsics supported by your host CPU. Container-level sandboxing still sounds like a superior bet.
- sansnomme 8y ago
  Container level sandboxing is pretty difficult to do well without hypervisor/VM support.

indentit 8y ago

This looks great, I like the simple examples which show how to get a string out from the web assembly module. One thing I didn't see in the readme or examples, however, is a way to import python functions to the web assembly modules. Is it possible, and just missing from the documentation atm?

djrenren 8y ago
Looks like the current code doesn't support it. https://github.com/wasmerio/python-ext-wasm/blob/master/src/instance.rs#L133-L134 https://github.com/wasmerio/python-ext-wasm/blob/master/src/... but in theory it shouldn't be too hard™
Hywan 8y ago
We don't support that yet, but we are working on it. It's not that easy even if the theory is kind of simple to explain. The implementation in the different backends is mostly the challenge. The Python extension uses the Wasmer runtime (https://github.com/wasmerio/wasmer https://github.com/wasmerio/wasmer) which supports 3 backends to generate executable code: Singlepass, Cranelift, and LLVM. Cranelift is the easiest one to embed, and it brings a great balance between compilation time and execution time. That's the one used in the Python extension right now. The “import functions” feature is also missing in the PHP extension (https://github.com/wasmerio/php-ext-wasm https://github.com/wasmerio/php-ext-wasm), but yeah, as I said, we are working on it. It's coming soon!
- indentit 8y ago
  understood, thanks for the explanation - keep up the great work! is there anything external contributors can do to help? :) I didn't find any issues labelled "good first issue" but I did find a related issue[0] [0]: https://github.com/wasmerio/wasmer/issues/267 https://github.com/wasmerio/wasmer/issues/267
  - Hywan 8y ago
    The best way to help right now is to test it, and report any issues! I've been able to try it on Windows so far for instance. Also the API isn't complete. We need to be able to grow the memory for instance. We need to understand the community needs to define a roadmap for the community itself.
- jononor 8y ago
  Once you allow to call Python, keeping things sandboxed becomes pretty hard, doesn't it?
  - Hywan 8y ago
    That's true. However, it allows to integrate your Wasm program to your favorite framework more easily. It's likely to be a small surface of code to review carefully. It's always a balance :-).

galacticdessert 8y ago

Looks good! Just for my understanding, what would be the advantage of working with wasm binaries instead of, let's say, C external libraries or DLLs via ctypes? I can think of: 1. Portability of the wasm binaries Is easier interoperability between data types an advantage as well? Ctypes can be a nightmare when passing around structs and pointers (and pointers of pointers, etc.)

make3 8y ago
My first impression is that the goal of this project is to run binaries that were likely built for other purposes like the web, not that they actually think that making libraries into webasm and running them into python has any particular advantages vs other options that already exist
sanxiyn 8y ago
There's also sandboxing. You shouldn't load untrusted DLL, but loading potentially untrusted WASM is a design goal of WebAssembly.
flohofwoe 8y ago
I have an actual real-world use case for this :) Shader-cross-compilation for 3D frameworks: https://floooh.github.io/2017/05/15/oryol-spirv.html https://floooh.github.io/2017/05/15/oryol-spirv.html I'm hooking python scripts as custom-build-jobs into C/C++ projects which (for instance) takes care of compiling GLSL shader code into HLSL (D3D), MSL (Metal) and various GLSL versions (for GLES2, GLES3 and desktop GL). Khronos provides C++ projects which help with this stuff: https://github.com/KhronosGroup/glslang https://github.com/KhronosGroup/glslang https://github.com/KhronosGroup/SPIRV-Cross https://github.com/KhronosGroup/SPIRV-Cross So I'm using Python to hook my custom build jobs into the C/C++ build system (via cmake), but I still need to invoke native command line tools (or DLLs) which wrap around those Khronos libraries. Currently this means building (at least) 3 executables for Windows, macOS and Linux, and "distribute" those precompiled exes with my python code. It looks like with this python+wasmer solution, I can instead compile the Khronos libraries into WASM modules, and the result runs "anywhere" where wasmer runs (e.g. also Raspberry Pi, BSDs etc...). That's really cool stuff :)
8y ago

the_mitsuhiko 8y ago

This is awesome! I wanted this so much. Need to play with this asap.

devwastaken 8y ago

"Safe: All calls to WebAssembly will be fast, but more importantly, completely safe and sandboxed." That's a very bold statement, I see no backup of this in the readme.

snek 8y ago
It's a side effect of the design of WebAssembly, as long as wasmer itself has a conformant implementation (the WASM spec itself is already formally verified).
Hywan 8y ago
I recommend this excellent reading, https://00f.net/2018/11/25/webassembly-doesnt-make-unsafe-languages-safe/ https://00f.net/2018/11/25/webassembly-doesnt-make-unsafe-la.... It clarifies what “safe” means in this context.