5 ms·
Ava PLS: open-source application for running language models locally
- iFire 3y agoLICENSE - MIT https://github.com/cztomsik/ava/blob/main/LICENSE.md https://github.com/cztomsik/ava/blob/main/LICENSE.md
- deleted 3y ago[deleted]
- codetrotter 3y agoIn other words, the best family of licenses in the world
- rgbrgb 3y agolooks nice, congrats on open sourcing it cztomsik! Good to have another reference implementation, excited to dig through and check out how you're integrating llama.cpp and SwiftUI. I went with an architecture based on hitting examples/server on localhost. Would you be interested in adding with otherbrain [0] to help build an open human feedback data set? Happy to help with PRs if so. If you're interested in supporting open model training, we'd love to have you! [0]: https://www.otherbrain.world/human-feedback https://www.otherbrain.world/human-feedback
- cztomsik 3y agoIt goes a bit against the original idea, this should be 100% air-gapped. No phoning home, no data-collection, everything is private.
- rgbrgb 3y agoMakes sense, thanks for considering (and fwiw I'm also building with a private AI / offline philosophy). In my implementation, the compromise is 1) being super loud with an alert warning the user when they're sharing feedback and 2) providing a setting where the user can turn off feedback buttons. I had gotten user feedback around wanting to tell the AI when it was doing a good/bad job and this seemed like a privacy preserving compromise that could actually improve models. Anyway, good luck and congrats again!
- jart 3y agoKudos for saying no. I can't believe we're living in an age where open source developers are being asked openly to leak the private conversations of their users to the public.
- haolez 3y agoI'll give it a try. I hope it's better than gpt4all, which I found out to be very buggy.
- cztomsik 3y agoIt should work fine if you're on Apple silicon, everything else is unfortunately second-class citizen, because I don't have any other devices and it's currently just me working on it. But if you'd like to maintain the windows, linux or intel-based mac releases, ping me on discord and I'd be happy to discuss and to accept PRs.
- deleted 3y ago[deleted]
- rgbrgb 3y agoedit: [deleted]
- cztomsik 3y agoIntel works in Ava, it's just slow, for some reason.
- rgbrgb 3y agoah, got it. yeah the llama.cpp speeds I've seen are like 5-10 tokens/second on intel but I've been impressed with the relatively ancient hardware llama.cpp works on (2017 iMac with 8GB RAM at 5.5 t/s with a 7B model!).
- cztomsik 3y agoit's ok, you can put the link back I don't mind the competition :) yes, I know it definitely works, so it has to be some stupid mistake but it's impossible to reproduce it without actual intel mac :-/ I had enough trouble with supporting monterey, which also does not work 100% AFAIK
- dang 3y agoWe changed the URL from https://tomsik.cz/posts/ava-oss/ https://tomsik.cz/posts/ava-oss/ to the project page, because it doesn't look like this project has been discussed on HN before. Readers might want to look at both links!
- cztomsik 3y agoActually, it was, here's the initial thread: https://news.ycombinator.com/item?id=37561085 https://news.ycombinator.com/item?id=37561085 The big news here is that it's now open-source: https://github.com/cztomsik/ava https://github.com/cztomsik/ava
- deleted 3y ago[deleted]
- nipperkinfeet 3y agoI couldn't get it working. Its stuck on Waiting for the model for all the models I downloaded.
- cztomsik 3y agoHow long did you wait, depending on the model size, it can take a while. Can you check logs? It's in the Settings -> System tab.
- canadiantim 3y agoWhich models do you currently support or have plans for? Looks cool tho
- cztomsik 3y agoeverything what llama.cpp supports
- vulture916 3y agoDoes an API exist that I may have overlooked? In the video @ https://github.com/cztomsik/ava https://github.com/cztomsik/ava, I see "Workflows" but not in the app. What are these, and are they coming to the app? Are quick tools basically the same as saving a playground prompt with variables?
- cztomsik 3y agoYes, it's at localhost:3002/api/ (and api/*.zig files respectively) The Workflows screen is feature-flagged for now and only visible in the DEV build. Yes, that's the whole idea behind QuickTools - to make repetitive tasks available quickly. It's also supposed to be in the main menu (drag-drop, reorderable, etc.) but I didn't have time to work on that yet.
- ranguna 3y agoAny benefit over text generation webui?
- cztomsik 3y agoIt's small, all-in-one binary. You only need to download model, and you can use builtin downloader to do that.