5 ms·

While Mistral's Docker image provides a practical solution, it's important to note that Docker isn't truly cross-platform. A Docker image compiled for x86 archi

While Mistral's Docker image provides a practical solution, it's important to note that Docker isn't truly cross-platform. A Docker image compiled for x86 architecture won't run on ARM, which can be a limitation for diverse hardware environments. On the other hand, Rust + WebAssembly (Wasm) offers genuine cross-platform capabilities. Wasm is not only faster to start and more resource-efficient but also more secure. This makes it exceptionally suitable for handling large language models, ensuring high performance while being cost-effective. In scenarios where diverse hardware support and efficiency are crucial, Rust + Wasm emerges as a superior choice over Docker.

hobofan 3y ago
Is there any data to back up the "faster to start and more resource-efficient", especially if you compare it to the native non-Python solutions that most people are using to run LLMs on local machines? I'm as big of a fan of Rust and WASM as the next person, but throwing around claims like that without benchmarks is one of the quickest ways to get your product dismissed.
- 3Sophons 3y ago
  Faster is compared to Python. Portable, more secure and lightweight are compared with Python and other native solutions. In terms of benchmarks, Rust / C++ is 50,000x faster than Python; WasmEdge runtime + portable app is 30M compared with 4G Python and 300MB llama.cpp Docker image that is NOT portable across CPU or GPU; Wasm sandbox is more secure than native binary.
  - hobofan 3y ago
    Thanks for clarifying!
- 3Sophons 3y ago
  Details https://www.cncf.io/blog/2023/10/30/rust-webassembly-building-infrastructure-for-large-language-model-ecosystems/ https://www.cncf.io/blog/2023/10/30/rust-webassembly-buildin...
smcleod 3y ago
Usually when you’re building container images at any scale you’d use something like buildx, kaniko or buildah which allow you to easily set multiple target architectures (AMD64, ARM64) for your images. There’s hardly any / basically no overhead to running an application in a container if that’s what you mean by overhead. If you mean the image size - well you only need to add the things you need - the problem is people tend to abuse images and install a lot of packages in the final image which absolutely aren’t required. Rust or golang at the core would be nice as Python can be slow at times. I do wish more folks would give Tauri a go for GUI apps.

hobofan 3y ago

Is there any data to back up the "faster to start and more resource-efficient", especially if you compare it to the native non-Python solutions that most people are using to run LLMs on local machines? I'm as big of a fan of Rust and WASM as the next person, but throwing around claims like that without benchmarks is one of the quickest ways to get your product dismissed.

3Sophons 3y ago
Faster is compared to Python. Portable, more secure and lightweight are compared with Python and other native solutions. In terms of benchmarks, Rust / C++ is 50,000x faster than Python; WasmEdge runtime + portable app is 30M compared with 4G Python and 300MB llama.cpp Docker image that is NOT portable across CPU or GPU; Wasm sandbox is more secure than native binary.
- hobofan 3y ago
  Thanks for clarifying!
3Sophons 3y ago
Details https://www.cncf.io/blog/2023/10/30/rust-webassembly-building-infrastructure-for-large-language-model-ecosystems/ https://www.cncf.io/blog/2023/10/30/rust-webassembly-buildin...

smcleod 3y ago

Usually when you’re building container images at any scale you’d use something like buildx, kaniko or buildah which allow you to easily set multiple target architectures (AMD64, ARM64) for your images. There’s hardly any / basically no overhead to running an application in a container if that’s what you mean by overhead. If you mean the image size - well you only need to add the things you need - the problem is people tend to abuse images and install a lot of packages in the final image which absolutely aren’t required. Rust or golang at the core would be nice as Python can be slow at times. I do wish more folks would give Tauri a go for GUI apps.