3 ms·

Show HN: FreeChat – local LLM chat for macOS

4 points by rgbrgb 3y ago

Hey HN, I made a tiny website for my llama.cpp-powered mac app. You can load in any GGUF (or it'll download the nice synthia mistral 7B fine-tune by default). I'm excited about local AI because of the privacy characteristics (you can run FreeChat without even connecting to the internet). I think that to amplify our abilities to the fullest extent we'll want information to flow fluidly back and forth between our brain and the AI. I'm pretty wary of what I'd plug my uncensored brain into, but keeping everything on-device feels most trustworthy to me. I was blown away by the llama.cpp demos and want to 1) make something useful for myself and 2) push more people towards open models and local AI. This is my first foray into packaging a local AI product for less technical users. The UX is basically cloned from other chat apps out there. Please join the TestFlight if you're a mac user interested in local AI and want to see how many tokens/second you can get on your machine.

rgbrgb 3y ago
Of course the other thing that's fun about local AI is that it's free if you already bought an expensive laptop. With the default model, I get ~32 tokens / second on my 2021 m1 ultra MacBook pro with 64GB RAM.
smcleod 3y ago
It would be nice to see a native and open source app like this catch up to (the fantastic) LM Studio!
- rgbrgb 3y ago
  thanks for checking it out. What’s the killer LMStudio feature for you? I think we probably have slightly different goals with our apps but would love to know what you’re liking regardless.
  - smcleod 3y ago
    For me it's a few things, I like LM studio because it's an app that's got lots of fine grained controls for tech-focused folks but is simple enough for people that just want to try LLMs to use. The customisability is great - the ability to have all your configuration/settings in the right panel is really useful, prompt templates per model, the inbuilt model gallery / downloader and more recently the local inference server that exposes an API is really handy. But... it's not open source and it's Electron...
  - smcleod 3y ago
    If you were to ask me what feature did I wish LMStudio had that if FreeChat had - would make me use it it would probably be really tight macOS integration. Perhaps sort of similar to how something like https://www.finito.ai https://www.finito.ai looks to be positioning itself but OSS and local first. It would be awesome to be able to have your local lly complete text in any application, input code in any editor, pass any image or graphic on the screen etc.... I think this would require using the accessibility / screen recording APIs so you'd want to trust whatever model you were running.
    - rgbrgb 3y ago
      Thanks, this and your other comment are very useful. I agree that finito ux is really cool and I had the same thought about it feeling sketchy to enable. They’re definitely more aligned with the use cases I’m excited about (more task oriented / simple for non-technical people).

4 points by rgbrgb 3y ago

rgbrgb 3y ago

Of course the other thing that's fun about local AI is that it's free if you already bought an expensive laptop. With the default model, I get ~32 tokens / second on my 2021 m1 ultra MacBook pro with 64GB RAM.

smcleod 3y ago

It would be nice to see a native and open source app like this catch up to (the fantastic) LM Studio!

rgbrgb 3y ago
thanks for checking it out. What’s the killer LMStudio feature for you? I think we probably have slightly different goals with our apps but would love to know what you’re liking regardless.
- smcleod 3y ago
  For me it's a few things, I like LM studio because it's an app that's got lots of fine grained controls for tech-focused folks but is simple enough for people that just want to try LLMs to use. The customisability is great - the ability to have all your configuration/settings in the right panel is really useful, prompt templates per model, the inbuilt model gallery / downloader and more recently the local inference server that exposes an API is really handy. But... it's not open source and it's Electron...
- smcleod 3y ago
  If you were to ask me what feature did I wish LMStudio had that if FreeChat had - would make me use it it would probably be really tight macOS integration. Perhaps sort of similar to how something like https://www.finito.ai https://www.finito.ai looks to be positioning itself but OSS and local first. It would be awesome to be able to have your local lly complete text in any application, input code in any editor, pass any image or graphic on the screen etc.... I think this would require using the accessibility / screen recording APIs so you'd want to trust whatever model you were running.
  - rgbrgb 3y ago
    Thanks, this and your other comment are very useful. I agree that finito ux is really cool and I had the same thought about it feeling sketchy to enable. They’re definitely more aligned with the use cases I’m excited about (more task oriented / simple for non-technical people).