3 ms·

What's the best way to use Ollama with a GUI, just OpenWebUI? Any options as well for mobile platforms like Android (or, I don't even know if we can run LLMs on

What's the best way to use Ollama with a GUI, just OpenWebUI? Any options as well for mobile platforms like Android (or, I don't even know if we can run LLMs on the phone in the first place).

sadeshmukh 2y ago
A lot of the UIs, including OpenWebUI have the feature to expose over LAN with users - that's what I did to use my GPU while still being on my phone. Not entirely sure about native UIs though. Also, I normally use Groq's (with a q) API since it's really cheap with no upfront billing info required - it's a whole order of magnitude cheaper iirc than OpenAI/Claude. They literally have a /openai endpoint if you need compatibility. You can look in the direction of Google's Gemma if you need a lightweight open weights LLM - there was something there that I forgot.
smcleod 2y ago
I personally use a mix of Open WebUI, Big AGI, BoltAI, AnythingLLM on the desktop. The mobile space is where things are really lacking at the moment, really I just end up browsing to Open WebUI but that's not ideal. I'd love a iOS native client that's well integrated into Siri, Shortcuts, Sharing etc...
qudat 2y ago
For hosting a web gui for ollama I use https://tuns.sh https://tuns.sh It really convenient because it's just an SSH tunnel and then you get automatic TLS and it protects your home IP. With that you can access it from your mobile phone, just gotta require a password to access it.
- satvikpendem 2y ago
  I'm running OpenWebUI via Docker via OrbStack, it also automatically provides TLS and works pretty well.
- minwidth0px 2y ago
  I wrote my own UI[0] that connects over WebRTC using Smoke.[1] [0] https://github.com/minwidth0px/gpt-playground https://github.com/minwidth0px/gpt-playground and https://github.com/minwidth0px/Webrtc-NAT-Traversal-Proxy-Server https://github.com/minwidth0px/Webrtc-NAT-Traversal-Proxy-Se... [1] https://github.com/sinclairzx81/smoke https://github.com/sinclairzx81/smoke
deleted 2y ago
[deleted]
huijzer 2y ago
I have Open WebUI on a Hetzner instance connected to Deep Infra. Works on mobile by turning the web page into an app. I find the web framework that WebUI uses quite bloated/slow, but apart from that it does work reliably. Price at Deep Infra is typically about $0.04 per month even when actively asking lots of questions during programming.
paradite 2y ago
I built a custom GUI for coding tasks specifically, with built-in code context management and workspaces: https://prompt.16x.engineer/ https://prompt.16x.engineer/ Should work well if you have 64G vRAM to run SOTA models locally.
- deleted 2y ago
  [deleted]
- throwaway314155 2y ago
  > Should work well if you have 64G vRAM to run SOTA models locally. Does anyone have this? edit: Ah, it's a Mac app.
  - paradite 2y ago
    Yeah Mac eats Windows on running LLMs. My app does support Windows though, you can connect to OpenAI, Claude, OpenRouter, Azure and other 3rd party providers. Just running SOTA LLMs locally can be challenging.
    - throwaway314155 2y ago
      I'm pretty satisfied with my linux nvidia gpu setup. I may not have as much memory on my card, but the speed is almost certainly competitive if not outright faster. Further there are lots of techniques that mitigate this issue like offloading/streaming layers in as needed, quantizing, etc. It also handles actual training/finetuning better.
  - gzer0 2y ago
    M4 Max with 128 GB RAM here. ;) Love it. A very expensive early Christmas present.
- accrual 2y ago
  Great looking GUI, I find simple black/white/boxy/monospace UIs very effective.
rkwz 2y ago
If you’re using a Mac, I’ve built a lightweight native app - https://github.com/sheshbabu/Chital https://github.com/sheshbabu/Chital
- antirez 2y ago
  That's very cool, finally a native app that runs fast. Thanks.
  - rkwz 2y ago
    Thanks for the kind words :)
magicalhippo 2y ago
Aa a Windows user, who just wanted something bare bones for playing, I found this[1] small project useful. It does support multi-modal models which is nice. [1]: https://github.com/jakobhoeg/nextjs-ollama-llm-ui https://github.com/jakobhoeg/nextjs-ollama-llm-ui
seb314 2y ago
For running llms _locally_ on Android, there's "pocketpal" (~7tok/s on a pixel 7 pro for some quant of llama 3.2 3B). (Not sure if it uses ollama though)
vunderba 2y ago
As far as open source goes, I'd probably recommend LibreChat. It has connections for ollama, openai, anthropic, etc. It let's you setup auth so you can theoretically use it from anywhere (phone, etc.). Fair warning, it's relatively heavyweight in so far as it has to spin up a number of docker instances but works very well. https://github.com/danny-avila/LibreChat https://github.com/danny-avila/LibreChat
zerop 2y ago
Many are there, apart from what others mentioned I am exploring Anything LLM - https://anythingllm.com/ https://anythingllm.com/. Liked the workspace concept in it. We can club documents in workspaces and RAG scope is managed.

What's the best way to use Ollama with a GUI, just OpenWebUI? Any options as well for mobile platforms like Android (or, I don't even know if we can run LLMs on the phone in the first place).

sadeshmukh 2y ago

A lot of the UIs, including OpenWebUI have the feature to expose over LAN with users - that's what I did to use my GPU while still being on my phone. Not entirely sure about native UIs though. Also, I normally use Groq's (with a q) API since it's really cheap with no upfront billing info required - it's a whole order of magnitude cheaper iirc than OpenAI/Claude. They literally have a /openai endpoint if you need compatibility. You can look in the direction of Google's Gemma if you need a lightweight open weights LLM - there was something there that I forgot.

smcleod 2y ago

I personally use a mix of Open WebUI, Big AGI, BoltAI, AnythingLLM on the desktop. The mobile space is where things are really lacking at the moment, really I just end up browsing to Open WebUI but that's not ideal. I'd love a iOS native client that's well integrated into Siri, Shortcuts, Sharing etc...

qudat 2y ago

For hosting a web gui for ollama I use https://tuns.sh https://tuns.sh It really convenient because it's just an SSH tunnel and then you get automatic TLS and it protects your home IP. With that you can access it from your mobile phone, just gotta require a password to access it.

satvikpendem 2y ago
I'm running OpenWebUI via Docker via OrbStack, it also automatically provides TLS and works pretty well.
minwidth0px 2y ago
I wrote my own UI[0] that connects over WebRTC using Smoke.[1] [0] https://github.com/minwidth0px/gpt-playground https://github.com/minwidth0px/gpt-playground and https://github.com/minwidth0px/Webrtc-NAT-Traversal-Proxy-Server https://github.com/minwidth0px/Webrtc-NAT-Traversal-Proxy-Se... [1] https://github.com/sinclairzx81/smoke https://github.com/sinclairzx81/smoke

deleted 2y ago

[deleted]

huijzer 2y ago

I have Open WebUI on a Hetzner instance connected to Deep Infra. Works on mobile by turning the web page into an app. I find the web framework that WebUI uses quite bloated/slow, but apart from that it does work reliably. Price at Deep Infra is typically about $0.04 per month even when actively asking lots of questions during programming.

paradite 2y ago

I built a custom GUI for coding tasks specifically, with built-in code context management and workspaces: https://prompt.16x.engineer/ https://prompt.16x.engineer/ Should work well if you have 64G vRAM to run SOTA models locally.

deleted 2y ago
[deleted]
throwaway314155 2y ago
> Should work well if you have 64G vRAM to run SOTA models locally. Does anyone have this? edit: Ah, it's a Mac app.
- paradite 2y ago
  Yeah Mac eats Windows on running LLMs. My app does support Windows though, you can connect to OpenAI, Claude, OpenRouter, Azure and other 3rd party providers. Just running SOTA LLMs locally can be challenging.
  - throwaway314155 2y ago
    I'm pretty satisfied with my linux nvidia gpu setup. I may not have as much memory on my card, but the speed is almost certainly competitive if not outright faster. Further there are lots of techniques that mitigate this issue like offloading/streaming layers in as needed, quantizing, etc. It also handles actual training/finetuning better.
- gzer0 2y ago
  M4 Max with 128 GB RAM here. ;) Love it. A very expensive early Christmas present.
accrual 2y ago
Great looking GUI, I find simple black/white/boxy/monospace UIs very effective.

rkwz 2y ago

If you’re using a Mac, I’ve built a lightweight native app - https://github.com/sheshbabu/Chital https://github.com/sheshbabu/Chital

antirez 2y ago
That's very cool, finally a native app that runs fast. Thanks.
- rkwz 2y ago
  Thanks for the kind words :)

magicalhippo 2y ago

Aa a Windows user, who just wanted something bare bones for playing, I found this[1] small project useful. It does support multi-modal models which is nice. [1]: https://github.com/jakobhoeg/nextjs-ollama-llm-ui https://github.com/jakobhoeg/nextjs-ollama-llm-ui

seb314 2y ago

For running llms _locally_ on Android, there's "pocketpal" (~7tok/s on a pixel 7 pro for some quant of llama 3.2 3B). (Not sure if it uses ollama though)

vunderba 2y ago

As far as open source goes, I'd probably recommend LibreChat. It has connections for ollama, openai, anthropic, etc. It let's you setup auth so you can theoretically use it from anywhere (phone, etc.). Fair warning, it's relatively heavyweight in so far as it has to spin up a number of docker instances but works very well. https://github.com/danny-avila/LibreChat https://github.com/danny-avila/LibreChat

zerop 2y ago

Many are there, apart from what others mentioned I am exploring Anything LLM - https://anythingllm.com/ https://anythingllm.com/. Liked the workspace concept in it. We can club documents in workspaces and RAG scope is managed.