3Sophons

23 ms·

151.
by 3Sophons 3y ago
hmmm. This bot would need some karma first?
152.
Getting Started with Orca-2-13B(secondstate.io)
68 points by 3Sophons 3y ago|17 comments
153.
by 3Sophons 3y ago
https://www.secondstate.io/articles/zephyr-7b/
154.
by 3Sophons 3y ago
Will Satya be accused of stock price manipulation? Any legal professional knows?
155.
by 3Sophons 3y ago
https://www.secondstate.io/articles/causallm-14b/
156.
by 3Sophons 3y ago
The demos are as fast as the ChatGPT, they just look slower. -First. It might seem slow at first because it's loading the model. But once that's done, it gets much quicker for any follow-up requests. Second, Turning on streaming o
157.
by 3Sophons 3y ago
For Qwen there is this the CausualLM 14B model, based on llama2 architecture but with Qwen 14B model weights.
158.
by 3Sophons 3y ago
I think you have to do similar things with Baichuan too. Which is also a bit pseudo open source
159.
by 3Sophons 3y ago
emmm do you mean not truly open source
160.
by 3Sophons 3y ago
https://github.com/second-state/WasmEdge-WASINN-examples/tre... These are the speed of different hardwares. You can rent GPU, which will be much faster
161.
by 3Sophons 3y ago
Ollama's approach with Docker is indeed user-friendly, but Rust + WebAssembly (Wasm) offers some cool perks that are worth considering. With Wasm, you get true cross-platform support, meaning it runs smoothly on various devices, automa
162.
by 3Sophons 3y ago
Details https://www.cncf.io/blog/2023/10/30/rust-webassembly-buildin...
163.
by 3Sophons 3y ago
Faster is compared to Python. Portable, more secure and lightweight are compared with Python and other native solutions. In terms of benchmarks, Rust / C++ is 50,000x faster than Python; WasmEdge runtime + portable app is 30M compared
164.
by 3Sophons 3y ago
For Chinese, there are this Yi Model by 0.1 AI Yi 34B https://www.secondstate.io/articles/yi-34b/ Uncencored Yi: https://www.secondstate.io/articles/dolphin-2.2-yi-34b/ and also Baichua
165.
by 3Sophons 3y ago
While Mistral's Docker image provides a practical solution, it's important to note that Docker isn't truly cross-platform. A Docker image compiled for x86 architecture won't run on ARM, which can be a limitation for dive
166.
4 Command lines to run open source LLMs across devices with 2MB inference app(secondstate.io)
1 points by 3Sophons 3y ago|0 comments
167.
Getting Started with Mistral-7b-Instruct-v0.1(secondstate.io)
140 points by 3Sophons 3y ago|54 comments
168.
by 3Sophons 3y ago
'portable' in the article refers to the software's ability to run across various operating systems or environments, rather than its hardware dependencies? This means while the software can be installed and run on different OS
169.
by 3Sophons 3y ago
ok.. should be run LLMs on my own devices with a 2MB portable app then?
170.
by 3Sophons 3y ago
llama.cpp typically needs to be compiled separately for each operating system and architecture (Windows, macOS, Linux, etc.), which is less portable. Also, the article mentions the use of hardware acceleration on devices with heterogeneous
171.
by 3Sophons 3y ago
You can run it on a variety of Linux, Mac and Windows based devices, including the Raspberry Pi and most laptops / servers you might have. But you still need a few GBs of memory in order to fit the model itself.
172.
by 3Sophons 3y ago
yeah excited to see how this will evolve. BTW, maybe give it a try on your Mac and see how it performs.
173.
by 3Sophons 3y ago
I guess making it portable is still quite important?
174.
by 3Sophons 3y ago
Hello you might be talking about reducing the size of the model itself (i.e., the trained weights) by orders of magnitude without losing accuracy, that's indeed a different challenge. But the article discusses reducing the inference ap
175.
Fast and Portable Llama2 Inference on the Heterogeneous Edge(secondstate.io)
313 points by 3Sophons 3y ago|98 comments
176.
by 3Sophons 3y ago
1. Get started with running LLMs, including Llama2 7b chat and a various of others, on your daily used macOS, or virtually any platform 2. Build a lightweight AI agent in several minutes 3. Use WasmEdge on AI beyond LLMs
177.
Live from KubeCon: Deploy lightweight AI with a few MBs and run across platforms(secondstate.io)
1 points by 3Sophons 3y ago|1 comments
178.
New Release: WasmEdge 0.13.5 Launched(github.com)
1 points by 3Sophons 3y ago|1 comments
179.
by 3Sophons 3y ago
WasmEdge 0.13.5 is out. CUDA enabled on ggml/llama.cpp plugin on arm64/x86_64 Linux. The following video works on the Jetson AGX Orin 64GB developer kit. https://x.com/hydai_tw/status/1720682591790211412?s
180.
Security Slam 2023: Contribute to WasmEdge and Elevate Open Source Security(github.com)
2 points by 3Sophons 3y ago|0 comments

More ›