3 ms·
Can it really run local LLMs at any decent size? I find it hard to believe for the price it can run anything more than 7b or 8b models slowly.
by kingkongjaffa 10mo ago
Can it really run local LLMs at any decent size? I find it hard to believe for the price it can run anything more than 7b or 8b models slowly.
- sosodev 10mo agoIt’s all subjective. Personally I think it would border on useless for local inference but maybe some people are happy with low quality models at slow speeds.
- jazzyjackson 10mo agoWow that's wild that they advertise "Run Deepseek-R1 locally" when the screenshot in the app store refers to "DeepSeek-R1-0528-Qwen3-8B"
- evil-olive 10mo agoaccording to [0] it looks like the "Umbrel Home" device they sell (with 16GB RAM and an N150 CPU) can run a 7B model at 2.7 tokens/sec, or a 13B model at 1.5 t/s. especially when they seem to be aiming for a not-terribly-technical market segment, there seems to be a pretty big mismatch between that performance and their website claims: > The most transformative technology of our generation shouldn't be confined to corporate data centers. Umbrel Home democratizes access to AI, allowing you to run powerful models on a device you own and control. 0: https://github.com/getumbrel/llama-gpt?tab=readme-ov-file#benchmarks https://github.com/getumbrel/llama-gpt?tab=readme-ov-file#be...