3 ms·
Is there any data to back up the "faster to start and more resource-efficient", especially if you compare it to the native non-Python solutions that most people
by hobofan 3y ago
Is there any data to back up the "faster to start and more resource-efficient", especially if you compare it to the native non-Python solutions that most people are using to run LLMs on local machines?
I'm as big of a fan of Rust and WASM as the next person, but throwing around claims like that without benchmarks is one of the quickest ways to get your product dismissed.
- 3Sophons 3y agoFaster is compared to Python. Portable, more secure and lightweight are compared with Python and other native solutions. In terms of benchmarks, Rust / C++ is 50,000x faster than Python; WasmEdge runtime + portable app is 30M compared with 4G Python and 300MB llama.cpp Docker image that is NOT portable across CPU or GPU; Wasm sandbox is more secure than native binary.
- hobofan 3y agoThanks for clarifying!
- 3Sophons 3y agoDetails https://www.cncf.io/blog/2023/10/30/rust-webassembly-building-infrastructure-for-large-language-model-ecosystems/ https://www.cncf.io/blog/2023/10/30/rust-webassembly-buildin...