3 ms·
Hi! Author here. Mind that I had to test with relatively small, "dumb" LLMs. I have no doubt I can run whisper.cpp on the RK3588 _and_ a tuned LLM to handle int
by rcarmo 3y ago
Hi! Author here. Mind that I had to test with relatively small, "dumb" LLMs. I have no doubt I can run whisper.cpp on the RK3588 _and_ a tuned LLM to handle intents, but it won't be a very smart one (I am hoping to find a good way to run quantized Mixtral, but given the RAM constraints on the 4GB board I didn't even try).
Edited to add: I'm looking for something like https://news.ycombinator.com/item?id=38704982 https://news.ycombinator.com/item?id=38704982 (LLM in a Flash) even if I find something with 16/32GB of RAM, which is why I looked at OnnxStream as well (but of course the inference in LLMs is different, so I can't leverage the NVMe just yet).
- user_7832 3y agoThanks! How "bad" are these LLMs though, especially for smart home-esque basic tasks? I'd imagine them to be alright-ish?
- rcarmo 3y agoWell, the ones I tried that would fit into 4GB RAM have trouble following directions and inconsistent output. I can't really tell you (yet) which would fit into, say, 16GB RAM and work consistently (and fast enough to, say, turn a light off faster than it would take you to get up and reach for the switch), but I'll eventually get to it...
- user_7832 3y agoThank you, that makes sense. I think I'll stick with my google hub for now haha
- FergusArgyll 3y agoDolphin-phi (uncensored tweaked version of msft phi) has been pretty good, I've been testing for only a couple days. It's 2.7B params, so depending on how much RAM the os is using you might be able to run that on 4GB. I run it on 8GB windows using wsl2/ollama and the OS takes up around 5GB (I think), so maybe....
- Havoc 3y agoI’ve run larger models on mine - work but quite slow. You can get a 32 orange off aliexpress today already