3 ms·
> Keeping in mind that the i7 runs at nearly six times the wattage, this is pretty good3, and the key point here is that the Orange Pi 5+ generated the output a
by user_7832 3y ago
> Keeping in mind that the i7 runs at nearly six times the wattage, this is pretty good3, and the key point here is that the Orange Pi 5+ generated the output at a usable pace–slower than what you’d get online from ChatGPT, but still fast enough that it is close to “reading speed”–whereas the Raspberry Pi 4 was too slow to be usable.
> This also means I could probably (if I could find a suitable model that fit into 4GB RAM) use the Orange Pi 5+ as a back-end for a “smart speaker” for my home automation setup (and I will probably try that in the future).
This is pretty interesting for me. I had (wrongly, I suppose) assumed that hardware requirements for LLMs were “have a recent NVidia GPU” but this proves otherwise.
- rcarmo 3y agoHi! Author here. Mind that I had to test with relatively small, "dumb" LLMs. I have no doubt I can run whisper.cpp on the RK3588 _and_ a tuned LLM to handle intents, but it won't be a very smart one (I am hoping to find a good way to run quantized Mixtral, but given the RAM constraints on the 4GB board I didn't even try). Edited to add: I'm looking for something like https://news.ycombinator.com/item?id=38704982 https://news.ycombinator.com/item?id=38704982 (LLM in a Flash) even if I find something with 16/32GB of RAM, which is why I looked at OnnxStream as well (but of course the inference in LLMs is different, so I can't leverage the NVMe just yet).
- user_7832 3y agoThanks! How "bad" are these LLMs though, especially for smart home-esque basic tasks? I'd imagine them to be alright-ish?
- rcarmo 3y agoWell, the ones I tried that would fit into 4GB RAM have trouble following directions and inconsistent output. I can't really tell you (yet) which would fit into, say, 16GB RAM and work consistently (and fast enough to, say, turn a light off faster than it would take you to get up and reach for the switch), but I'll eventually get to it...
- user_7832 3y agoThank you, that makes sense. I think I'll stick with my google hub for now haha
- FergusArgyll 3y agoDolphin-phi (uncensored tweaked version of msft phi) has been pretty good, I've been testing for only a couple days. It's 2.7B params, so depending on how much RAM the os is using you might be able to run that on 4GB. I run it on 8GB windows using wsl2/ollama and the OS takes up around 5GB (I think), so maybe....
- Havoc 3y agoI’ve run larger models on mine - work but quite slow. You can get a 32 orange off aliexpress today already
- timnetworks 3y agoInferencing can be done in software entirely (e.g. INT8) but it's very slow compared to GPU or APU. nVidia cornered the market because everything (tensorflow and everything after) is optimized for it, but you can get good results on AMD now, and on ARC too in some cases. And slow results entirely in software (CPU-RAM), which for personal and non-constant use may be just fine too.
- user_7832 3y agoThanks! Do you have any guides/websites/github repos for running these models on CPUs?
- rcarmo 3y agoOllama will (nearly always) work provided you have enough RAM. I was actually pretty surprised that it didn't work on my N5105 (which has 16GB) because it relies on AVX instructions...
- user_7832 3y agoThanks! Someone else mentioned llama.cpp but it appears that ollama is just a gui frontend for llama (which is good because I find guis easier). I'll hopefully set it up soon!
- FergusArgyll 3y agoIt's not a GUI it's a cli but very easy to use "ollama run {model}". you can also `ollama serve` which serves an api, and then you can use or build a simple gui.
- user_7832 3y agoThanks, I’ll keep that in mind!
- jmorgan 3y ago
- nabakin 3y agoOver the past year or so various projects have made it possible to run LLMs on just about anything. Some GPUs are still better than others like Nvidia GPUs are still the best for token throughput (via TensorRT-LLM), but AMD GPUs are competitive (via vLLM) and even CPUs can run LLMs at decent speeds (via Llama.cpp).
- user_7832 3y agoThank you!