5 ms·

Show HN: macOS GUI for running LLMs locally

58 points by cztomsik 3y ago

Hello HN, I've been working on this project for a while, and it has been in an "open" beta for some time. I finally believe it's ready for its first release. I hope you like it. Here are some potential questions that may arise: 1. How does it compare to LM Studio? It's likely that if you're already using LM Studio, you'll continue to do so. This project is designed to be more user-friendly. 2. Is it open-source? No, it is not. 3. Does it use any open-source libraries? Yes, it uses llama.cpp and a few others, as indicated in the license information included with the application. 4. Why is not using electronjs? Two reasons, I wanted total control over the whole tech-stack and second, I wanted to be able to send this to my friends over iMessage. 5. Does it support Intel macs? It should, but I couldn't test it. 6. Does it support older macOS? 12.6 is the lowest version at the moment. 7. Is XXX a bug? Probably :)

syntaxing 3y ago
Do the models run in a docker container? Probably one of my favorite things about Ollama.
- mchiang 3y ago
  disclaimer: I'm one of the maintainers working on Ollama. I would love to hear how you are using Ollama. One of the upcoming releases will involve an official release of Ollama on Linux with CUDA support of the box. From there, we will publish Ollama Docker images to enable GPU support as well.
  - syntaxing 3y ago
    Thank you for your amazing work! I more or less use it as a llama cpp replacement cause I honestly can’t figure out a good prompt structure so I get bad results. On my M1 MBP, I use it for stuff that can’t go to GPT but can only use upto 13B. I also run a 34B model with CUBLAS enabled on my Linux server by modifying the go files mentioned in one of the issue tickets.
- cztomsik 3y ago
  Sorry, I thought I have answered already. No, everything runs as one native-like (mix of native and webview) application.
swyx 3y ago
very cool. why did you work on it? curious on motivations. congrats on shipping.
- cztomsik 3y ago
  Thanks :) love your podcast BTW. See https://news.ycombinator.com/item?id=37562287 https://news.ycombinator.com/item?id=37562287 - I can't tell much more yet, but I feel like LLMs are way more useful than for a chat-bot or a vscode extension.
  - swyx 3y ago
    thanks very much! will keep a look out as you build your app.
alexstore06 3y ago
How does this compare to GUIs like llamero (https://github.com/mpociot/llamero https://github.com/mpociot/llamero) and gpt4all (https://gpt4all.io/index.html https://gpt4all.io/index.html)?
- cztomsik 3y ago
  It has playground with prompt saving and (simple) templates, chat saving, builtin model downloader. It's also very small, most people don't care about that, but I do.
deleted 3y ago
[deleted]
krm01 3y ago
Is there a central place where most downloadable models are available for quick download?
- SparkyMcUnicorn 3y ago
  https://huggingface.co/TheBloke https://huggingface.co/TheBloke
- cztomsik 3y ago
  It works with current version of GGUF models for llama.cpp - you can find them on huggingface, or you can convert them manually. Only a few download links are baked-in at the moment but whatever *.gguf file you put in your Downloads folder should appear in the dropdown.
marcellus23 3y ago
> What is a language server? > A language server is a specialized program that processes language-related tasks. This includes activities like text generation, grammar correction, rephrasing, summarization, data extraction, and more. This confused me initially. I've never heard language server in this context. To me, it has always been in the context of the LSP[0]. But maybe this is a common usage of the term and I've just missed it? Even if so, it seems like an odd term, since this is a self-contained desktop application and not really a server, right? 0: https://en.wikipedia.org/wiki/Language_Server_Protocol https://en.wikipedia.org/wiki/Language_Server_Protocol
- cztomsik 3y ago
  No, you are right, it's a new term I made up. The idea is that I see this as a RAD tool (for non-programmers). So you would have this app open all the time and it would serve different tasks you have prepared before. But that part is not finished yet and it will be limited in the free version.
  - CharlesW 3y ago
    > …it's a new term I made up. On the bright side, now that you know what a language server is, you can rename your thing so it can be searched for.
- duskwuff 3y ago
  Now I'm imagining a natural-language language server -- a plugin for text editors which can provide services like text completion and transformations in plain text documents based on an underlying LLM. Why let programming languages have all the fun? :)
achrono 3y ago
>Is it open-source? No, it is not. Why not? As to Why --> this one actually has a decent GUI and one that's not (visibly) based on a locally running webserver -- so it seems like it's got a shot at getting 100x popular with just a YouTube video (or gasp Tiktok). With the popularity of open-source today I find it a little hard to trust that something is "ensuring maximum privacy" as you say without having the open-source badge (yes, I know I shouldn't be so naive, but no, I am not going to look at network level logs to find out if the app is truly contained to local).
- cztomsik 3y ago
  Fair point, but if this was open-source, would you really go and read all of the source code in several programming languages and also audit all of the dependencies? IMHO, at some point, it's just about trust. Now, to be 100% honest, I'd love to release this as OSS, but my current business model is based on freemium. And, sell it as white-label for companies. EDIT: if you have better idea, I'd love to hear about it.
  - lagniappe 3y ago
    I resent having to step in and say something here, but the way you describe things, I effectively don't exist. I read the app code. I write stdlib code. I grokk every line of every library I use. I share alike. I'm not saying change your ways, just consider being a bit less flippant when you say things like "IMHO, at some point, it's just about trust". As someone who seeks to profit from this, that's about the worst example you can give a prospective user like me. My face is hot just having to type this.
    - unshavedyak 3y ago
      Also, FOSS has a lot more chances to have eyes on it. I definitely don't read every line, and i definitely don't expect anyone else to - but i do expect more eyes to be on it, to raise questions, spot interesting behavior, etc. If you see a questionable network IO you can look in the source to investigate, etc. It has value even if every person doesn't individually read every line.
BrutalCoding 3y ago
Good to see that other devs share the same passion by making AI accessible to everyone. I’ve got a similar app, with the same goals and eventually same supported platforms. Slight difference is that it’s open source: https://github.com/BrutalCoding/shady.ai https://github.com/BrutalCoding/shady.ai
- aaomidi 3y ago
  Open source when running LLMs locally is...kinda the entire point. Thank you for this!
  - BrutalCoding 3y ago
    Cheers for your nice words! I still salute products like Avapls. Hope you succeed @cztomsik. It’s a good initiative to make offline models accessible.
    - cztomsik 3y ago
      Thank you, and I wish you luck with your project too, of course :)
- syncbehind 3y ago
  Yours looks ... shady. Jokes aside, thanks for building this! It looks swell and I really enjoy your sense of humor on your readme as well. Looking forward to poking around tonight!
  - BrutalCoding 3y ago
    If you're on macOS ARM64, give it a go. I have the hardware from the README but need more time. I often switch between languages and systems. Kudos to the OP for their work.
deleted 3y ago
[deleted]
carlosjobim 3y ago
> 6. Does it support older macOS? 12.6 is the lowest version at the moment. Do you have any interest in supporting older versions of OS X?
- cztomsik 3y ago
  It depends, what version do you need?
  - carlosjobim 3y ago
    I'm on Catalina, but to be fair I haven't used LLMs for more than toying, so I wouldn't call it a "need", just something cool to try out.
    - cztomsik 3y ago
      I can't promise anything but there's apparently some problem with intel macs and I only have older machine here (with catalina) so I might give it a shot and if it works I can publish the build. EDIT: it's too old, sorry.
neontomo 3y ago
I like this GUI. Notes: - I was missing file sizes on the models list, and even in the installer window there's only a percentage indicator. Blurring the window while installing also stops me from visually exploring the app while waiting for the install to finish (maybe just a lil' bit of blur is enough to show that the rest of the window is inactive?). - Scrolling up should stop auto-scroll in the chat, I couldn't even reach the Stop generation button while it was generating. Maybe put that button in a static location. - Maybe an uninstall button for models is appropriate.
- cztomsik 3y ago
  Thanks for the feedback, this is what I was hoping for. > Scrolling up should stop auto-scroll in the chat This is at the top-of the list for today, check the website/discord/twitter for updates. I want to do both - pin the button at the bottom, and also make sure that if you scroll, it will stay there. Currently, it's one component which contains the button and also scrolls and it was a bad idea. > Blurring the window while installing Agreed 100%, I definitely want to improve downloader a lot more but this was good enough for first release. > Maybe an uninstall button for models is appropriate. It should be there (in the upper area), I hope the 24h cache in cloudfront is not too much but the link should be pointing to 09-18 version which has Delete button.
pedalpete 3y ago
I'm tempted to try this, not because I want a locally running version of openAI, but just because the openAI website is so painful to use. I'm currently on an intel mac mini with almost no memory, so I can't even download to test this (I have to close all other programs just so I can build an app in xcode). I've got a razer blade as well, but obviously mac only doesn't help there.
walth 3y ago
How does this compare with https://github.com/getumbrel/llama-gpt https://github.com/getumbrel/llama-gpt ?
- cztomsik 3y ago
  This is macos application, you download it, double click it and it should work.

58 points by cztomsik 3y ago

syntaxing 3y ago

Do the models run in a docker container? Probably one of my favorite things about Ollama.

mchiang 3y ago
disclaimer: I'm one of the maintainers working on Ollama. I would love to hear how you are using Ollama. One of the upcoming releases will involve an official release of Ollama on Linux with CUDA support of the box. From there, we will publish Ollama Docker images to enable GPU support as well.
- syntaxing 3y ago
  Thank you for your amazing work! I more or less use it as a llama cpp replacement cause I honestly can’t figure out a good prompt structure so I get bad results. On my M1 MBP, I use it for stuff that can’t go to GPT but can only use upto 13B. I also run a 34B model with CUBLAS enabled on my Linux server by modifying the go files mentioned in one of the issue tickets.
cztomsik 3y ago
Sorry, I thought I have answered already. No, everything runs as one native-like (mix of native and webview) application.

swyx 3y ago

very cool. why did you work on it? curious on motivations. congrats on shipping.

cztomsik 3y ago
Thanks :) love your podcast BTW. See https://news.ycombinator.com/item?id=37562287 https://news.ycombinator.com/item?id=37562287 - I can't tell much more yet, but I feel like LLMs are way more useful than for a chat-bot or a vscode extension.
- swyx 3y ago
  thanks very much! will keep a look out as you build your app.

alexstore06 3y ago

How does this compare to GUIs like llamero (https://github.com/mpociot/llamero https://github.com/mpociot/llamero) and gpt4all (https://gpt4all.io/index.html https://gpt4all.io/index.html)?

cztomsik 3y ago
It has playground with prompt saving and (simple) templates, chat saving, builtin model downloader. It's also very small, most people don't care about that, but I do.

deleted 3y ago

[deleted]

krm01 3y ago

Is there a central place where most downloadable models are available for quick download?

SparkyMcUnicorn 3y ago
https://huggingface.co/TheBloke https://huggingface.co/TheBloke
cztomsik 3y ago
It works with current version of GGUF models for llama.cpp - you can find them on huggingface, or you can convert them manually. Only a few download links are baked-in at the moment but whatever *.gguf file you put in your Downloads folder should appear in the dropdown.

marcellus23 3y ago

> What is a language server? > A language server is a specialized program that processes language-related tasks. This includes activities like text generation, grammar correction, rephrasing, summarization, data extraction, and more. This confused me initially. I've never heard language server in this context. To me, it has always been in the context of the LSP[0]. But maybe this is a common usage of the term and I've just missed it? Even if so, it seems like an odd term, since this is a self-contained desktop application and not really a server, right? 0: https://en.wikipedia.org/wiki/Language_Server_Protocol https://en.wikipedia.org/wiki/Language_Server_Protocol

cztomsik 3y ago
No, you are right, it's a new term I made up. The idea is that I see this as a RAD tool (for non-programmers). So you would have this app open all the time and it would serve different tasks you have prepared before. But that part is not finished yet and it will be limited in the free version.
- CharlesW 3y ago
  > …it's a new term I made up. On the bright side, now that you know what a language server is, you can rename your thing so it can be searched for.
duskwuff 3y ago
Now I'm imagining a natural-language language server -- a plugin for text editors which can provide services like text completion and transformations in plain text documents based on an underlying LLM. Why let programming languages have all the fun? :)

achrono 3y ago

>Is it open-source? No, it is not. Why not? As to Why --> this one actually has a decent GUI and one that's not (visibly) based on a locally running webserver -- so it seems like it's got a shot at getting 100x popular with just a YouTube video (or gasp Tiktok). With the popularity of open-source today I find it a little hard to trust that something is "ensuring maximum privacy" as you say without having the open-source badge (yes, I know I shouldn't be so naive, but no, I am not going to look at network level logs to find out if the app is truly contained to local).

cztomsik 3y ago
Fair point, but if this was open-source, would you really go and read all of the source code in several programming languages and also audit all of the dependencies? IMHO, at some point, it's just about trust. Now, to be 100% honest, I'd love to release this as OSS, but my current business model is based on freemium. And, sell it as white-label for companies. EDIT: if you have better idea, I'd love to hear about it.
- lagniappe 3y ago
  I resent having to step in and say something here, but the way you describe things, I effectively don't exist. I read the app code. I write stdlib code. I grokk every line of every library I use. I share alike. I'm not saying change your ways, just consider being a bit less flippant when you say things like "IMHO, at some point, it's just about trust". As someone who seeks to profit from this, that's about the worst example you can give a prospective user like me. My face is hot just having to type this.
  - unshavedyak 3y ago
    Also, FOSS has a lot more chances to have eyes on it. I definitely don't read every line, and i definitely don't expect anyone else to - but i do expect more eyes to be on it, to raise questions, spot interesting behavior, etc. If you see a questionable network IO you can look in the source to investigate, etc. It has value even if every person doesn't individually read every line.

BrutalCoding 3y ago

Good to see that other devs share the same passion by making AI accessible to everyone. I’ve got a similar app, with the same goals and eventually same supported platforms. Slight difference is that it’s open source: https://github.com/BrutalCoding/shady.ai https://github.com/BrutalCoding/shady.ai

aaomidi 3y ago
Open source when running LLMs locally is...kinda the entire point. Thank you for this!
- BrutalCoding 3y ago
  Cheers for your nice words! I still salute products like Avapls. Hope you succeed @cztomsik. It’s a good initiative to make offline models accessible.
  - cztomsik 3y ago
    Thank you, and I wish you luck with your project too, of course :)
syncbehind 3y ago
Yours looks ... shady. Jokes aside, thanks for building this! It looks swell and I really enjoy your sense of humor on your readme as well. Looking forward to poking around tonight!
- BrutalCoding 3y ago
  If you're on macOS ARM64, give it a go. I have the hardware from the README but need more time. I often switch between languages and systems. Kudos to the OP for their work.

deleted 3y ago

[deleted]

carlosjobim 3y ago

> 6. Does it support older macOS? 12.6 is the lowest version at the moment. Do you have any interest in supporting older versions of OS X?

cztomsik 3y ago
It depends, what version do you need?
- carlosjobim 3y ago
  I'm on Catalina, but to be fair I haven't used LLMs for more than toying, so I wouldn't call it a "need", just something cool to try out.
  - cztomsik 3y ago
    I can't promise anything but there's apparently some problem with intel macs and I only have older machine here (with catalina) so I might give it a shot and if it works I can publish the build. EDIT: it's too old, sorry.

neontomo 3y ago

I like this GUI. Notes: - I was missing file sizes on the models list, and even in the installer window there's only a percentage indicator. Blurring the window while installing also stops me from visually exploring the app while waiting for the install to finish (maybe just a lil' bit of blur is enough to show that the rest of the window is inactive?). - Scrolling up should stop auto-scroll in the chat, I couldn't even reach the Stop generation button while it was generating. Maybe put that button in a static location. - Maybe an uninstall button for models is appropriate.

cztomsik 3y ago
Thanks for the feedback, this is what I was hoping for. > Scrolling up should stop auto-scroll in the chat This is at the top-of the list for today, check the website/discord/twitter for updates. I want to do both - pin the button at the bottom, and also make sure that if you scroll, it will stay there. Currently, it's one component which contains the button and also scrolls and it was a bad idea. > Blurring the window while installing Agreed 100%, I definitely want to improve downloader a lot more but this was good enough for first release. > Maybe an uninstall button for models is appropriate. It should be there (in the upper area), I hope the 24h cache in cloudfront is not too much but the link should be pointing to 09-18 version which has Delete button.

pedalpete 3y ago

I'm tempted to try this, not because I want a locally running version of openAI, but just because the openAI website is so painful to use. I'm currently on an intel mac mini with almost no memory, so I can't even download to test this (I have to close all other programs just so I can build an app in xcode). I've got a razer blade as well, but obviously mac only doesn't help there.

walth 3y ago

How does this compare with https://github.com/getumbrel/llama-gpt https://github.com/getumbrel/llama-gpt ?

cztomsik 3y ago
This is macos application, you download it, double click it and it should work.