8 ms·
Lm.rs: Minimal CPU LLM inference in Rust with no dependency
- fuddle 2y agoNice work, it would be great to see some benchmarks comparing it to llm.c.
- littlestymaar 2y agoI doubt it would compare favorably at the moment, I don't think it's particularly well optimized besides using rayon to get CPU parallelism and wide for a bit of SIMD. It's good enough to get pretty good performance for little effort, but I don't think it would win a benchmark race either.
- echelon 2y agoThis is really cool. It's already using Dioxus (neat). I wonder if WASM could be put on the roadmap. If this could run a lightweight LLM like RWKV in the browser, then the browser unlocks a whole class of new capabilities without calling any SaaS APIs.
- marmaduke 2y agoI was poking at this a bit here https://github.com/maedoc/rwkv.js https://github.com/maedoc/rwkv.js using the Rwkv.cpp compiled with emscripten, but I didn’t quite figure out the tokenizers part (yet, only spent about an hour on it) Nevertheless I am pretty sure the 1.6b rwkv6 would be totally usable offline browser only. It’s not capable enough for general chat but for rag etc it could be quite enough
- littlestymaar 2y ago> I wonder if WASM could be put on the roadmap. The library itself should be able to compile to WASM with very little change: rayon and wide the only mandatory dependencies support wasm out of the box, and to get rid of memmap2 by replacing the `Mmap` type in transformer.rs with `&[u8]`. That being said, RWKV is a completely different architecture so it should be reimplemented entierly and is not likely to be part of the roadmap ever (not the main author so I can't say for sure, but I really doubt it).
- lucgagan 2y agoCorrect me if I am wrong, but these implementations are all CPU bound?, i.e. if I have a good GPU, I should look for alternatives.
- deleted 2y ago[deleted]
- littlestymaar 2y agoIt's all implemented on the CPU, yes, there's no GPU acceleration whatsoever (at the moment at least). > if I have a good GPU, I should look for alternatives. If you actually want to run it, even just on the CPU, you should look for an alternative (and the alternative is called llama.cpp) this is more of an educational resource about how things work when you remove all the layers of complexity in the ecosystem. LLM are somewhat magic in how effective they can be, but in terms of code it's really simple.
- bt1a 2y agoYou are correct. This project is "on the CPU", so it will not utilize your GPU for computation. If you would like to try out a Rust framework that does support GPUs, Candle https://github.com/huggingface/candle/tree/main https://github.com/huggingface/candle/tree/main may be worth exploring
- J_Shelby_J 2y agoYes. Depending on gpu 10-20x difference. For rust you have the llama.cpp wrappers like llm_client (mine), and the candle based projects mistral.rs, and Kalosm. Although, my project does try and provide an implementation of mistral.rs, I haven’t fully migrated from llama.cpp. A full rust implementation would be nice for quick install times (among other reasons). Right now my crate has to clone and build. It’s automated for mac, pc, and Linux but it adds about a minute of build time.
- kkielhofner 2y agoCPU, yes, but more importantly memory bandwidth. An RTX 3090 (as one example) has nearly 1TB/s of memory bandwidth. You'd need at least 12 channels of the fastest proof-of-concept DDR5 on the planet to equal that. If you have a discrete GPU, use an implementation that utilizes it because it's a completely different story. Apple Silicon boasts impressive numbers on LLM inference because it has a unified CPU-GPU high-bandwidth (400GB/s IIRC) memory architecture.
- wyldfire 2y agoThe title is less clear than it could be IMO. When I saw "no dependency" I thought maybe it could be no_std (llama.c is relatively lightweight in this regard). But it's definitely not `no_std` and in fact seems like it has several dependencies. Perhaps all of them are rust dependencies?
- saghm 2y agoThe readme seems to indicate that it expects pytorch alongside several other Python dependencies in a requirements.txt file (which is the only place I can find any form of the word "dependency" on the page). I'm very confused by the characterization in the title here given that it doesn't seem to be claimed at all by the project itself (which simple has the subtitle "Minimal LLM inference in Rust"). From the git history, it looks like the username of the person who posted this here is someone who has contributed to the project but isn't the primary author. If they could elaborate on what exactly they mean by saying this has "zero dependencies", that might be helpful.
- littlestymaar 2y ago> The readme seems to indicate that it expects pytorch alongside several other Python dependencies in a requirements.txt file That's only if you want to convert the model yourself, you don't need that if you use the converted weights on the author's huggingface page (in “prepared-models” table of the README). > From the git history, it looks like the username of the person who posted this here is someone who has contributed to the project but isn't the primary author. Yup that's correct, so far I've only authored the dioxus GUI app. > If they could elaborate on what exactly they mean by saying this has "zero dependencies", that might be helpful. See my other response: https://news.ycombinator.com/item?id=41812665 https://news.ycombinator.com/item?id=41812665
- J_Shelby_J 2y agoWhat do you think about implementing your gui for other rust LLM projects? I’m looking for a front end for my project: https://github.com/ShelbyJenkins/llm_client https://github.com/ShelbyJenkins/llm_client
- J_Shelby_J 2y agoNeat. FYI I have a whole bunch of rust tools[0] for loading models and other LLM tasks. For example auto selecting the largest quant based on memory available, extracting a tokenizer from a gguf, prompting, etc. You could use this to remove some of the python dependencies you have. Currently to support llama.cpp, but this is pretty neat too. Any plans to support grammars? [0] https://github.com/ShelbyJenkins/llm_client https://github.com/ShelbyJenkins/llm_client
- gip 2y agoGreat! Did something similar some time ago [0] but the performance was underwhelming compared to C/C++ code running on CPU (which points to my lack of understanding of how to make Rust fast). Would be nice to have some benchmarks of the different Rust implementations. Implementing LLM inference should/could really become the new "hello world!" for serious programmers out there :) [0] https://github.com/gip/yllama.rs https://github.com/gip/yllama.rs
- flaneur2020 2y agoi also had a similar 'hello world' experience some time ago with [0] :). i manually used some SIMD instructions, and it seems the performance could align with llama.cpp. it appears that the key to performance is: 1. using SIMD on quantized matrix multiplication 2. using a busy loop instead of condition variables when splitting work among threads. (however, i haven't had more free time to continue working on inferencing quantized models on GPU (with Vulkan), and it hasn't been updated for a long time since then.) [0] https://github.com/crabml/crabml https://github.com/crabml/crabml
- simonw 2y agoThis is impressive. I just ran the 1.2G llama3.2-1b-it-q80.lmrs on a M2 64GB MacBook and it felt speedy and used 1000% of CPU across 13 threads (according to Activity Monitor). cd /tmp git clone https://github.com/samuel-vitorino/lm.rs cd lm.rs RUSTFLAGS="-C target-cpu=native" cargo build --release --bin chat curl -LO 'https://huggingface.co/samuel-vitorino/Llama-3.2-1B-Instruct-Q8_0-LMRS/resolve/main/tokenizer.bin?download=true' curl -LO 'https://huggingface.co/samuel-vitorino/Llama-3.2-1B-Instruct-Q8_0-LMRS/resolve/main/llama3.2-1b-it-q80.lmrs?download=true' ./target/release/chat --model llama3.2-1b-it-q80.lmrs
- littlestymaar 2y agoCould you try with ./target/release/chat --model llama3.2-1b-it-q80.lmrs --show-metrics To know how many token/s you get?
- simonw 2y agoNice, just tried that with "tell me a long tall tale" as the prompt and got: Speed: 26.41 tok/s Full output: https://gist.github.com/simonw/6f25fca5c664b84fdd4b72b0918546e0 https://gist.github.com/simonw/6f25fca5c664b84fdd4b72b091854...
- jodleif 2y agoHow much with llama.cpp? A 1b model should be a lot faster on a m2
- hobofan 2y agoGiven the fact that this at the core relies on the `rayon` and `wide` libraries, which are decently baseline optimized but quite a bit away from what llama.cpp can do when being specialized on such a specific use-case, I think the speed is about what I would expect. So yeah, I think there is a lot of room for optimization, and the only reason one would use this today is if they want to have a "simple" implementation that doesn't have any C/C++ dependencies for build tooling reasons.
- jll29 2y agoThis is beautifully written, thanks for sharing. I could see myself using some of the source code in the classroom to explain how transformers "really" work; code is more concrete/detailed than all those pictures of attention heads etc. Two points of minor criticism/suggestions for improvement: - libraries should not print to stdout, as that output may detroy application output (imagine I want to use the library in a text editor to offer style checking). So best to write to a string buffer owned by a logging class instance associated with a lm.rs object. - Is it possible to do all this without "unsafe" without twisting one's arm? I see there are uses of "unsafe" e.g. to force data alignment in the model reader. Again, thanks and very impressive!
- LoganDark 2y ago> best to write to a string buffer It's best to call a user callback. That way logs can be, for example, displayed in a GUI.
- azinman2 2y agoA good logging framework has all the hooks you need
- amelius 2y agoDoesn't rust have a standard solution for that? If I use 10 libraries and they all use a different logging framework then that's ... not convenient.
- sapiogram 2y agoIt does, everyone uses the `log` crate. But then it wouldn't be zero-dependencies anymore.
- littlestymaar 2y agoIn fairness it's already not really “zero dependency” since it uses rayon (for easy multithreading) and wide (for easy SIMD), using log would make total sense I think (not the main author, just a contributor).
- marques576 2y agoSuch a talented guy!
- dcreater 2y agoWhat's the value of this compared to llama.cpp?
- kvakkefly 2y agoWould love to see a wasm version of this!
- dboreham 2y agoQuite curious to hear: why? Asking because this program isn't useful without 3G of model data, and WASM isn't useful outside of the browser (and perhaps some blockchain applications), where 3G of data isn't going to be practically available.
- nikolayasdf123 2y agohow does this compare to https://github.com/EricLBuehler/mistral.rs https://github.com/EricLBuehler/mistral.rs ?
- littlestymaar 2y agoMuch simpler codebase because it has much less features. It doesn't aim to be a llama.cpp competitor AFAIK.
- aravindputrevu 2y agoInteresting, I appreciate the rust community‘s enthu to rewrite most the stuff.
- dvt 2y agoThis is cool (and congrats on writing your first Rust lib!), but Metal/Cuda support is a must for serious local usage.
- littlestymaar 2y agoUsing Cuda is a non starter because it would go against the purpose of this project, but I (not the main author but contributor) am experimenting with wgpu to get some kind of GPU acceleration. I'm not sure it goes anywhere though, because the main author want to keep the complexity under control.
- tormeh 2y agowgpu would be awesome. Too little ML software out there is hardware-agnostic.
- littlestymaar 2y agoThat's exactly my feeling and that's why I started working on it.
- vietvu 2y agoAnother llama.cpp and mistral.rs? If it support vision models then fine, I will try it. EDIT: Looks like no L3.2 11B yet.
- littlestymaar 2y agoIt supports the PHI 3.5 vision model since yesterday actually. I think a 11B model would be way too slow in its current shape though.