5 ms·
Show HN: LLaVaVision: An AI "Be My Eyes"-like web app with a llama.cpp backend
A simple mobile web app inspired by Fuzzy-Search/realtime-bakllava that uses llama.cpp server backend with multimodal mode to describe and narrate what the phone camera sees.
I built this thing in a few hours using a single ChatGPT thread to generate most things for me and iterate on this project. Here's the workflow: https://chat.openai.com/share/ea84ec69-5617-45e8-8772-ac2dcfa2e1b1 https://chat.openai.com/share/ea84ec69-5617-45e8-8772-ac2dcf...
- yousnail 3y agoThis called my garage ‘fairly run-down’. Needs tuning.
- lxe 3y agoI noticed that it "hallucinates" in the most direct sense of the word as the description goes on.
- ggerganov 3y agoI've found lowering the temperature and disabling the repetition penalty can help [0]. My explanation is that the repetition penalty penalizes the end of sentences and sort of forces the generation to go on instead of stopping. [0] https://old.reddit.com/r/LocalLLaMA/comments/17e855d/llamacpp_server_now_supports_multimodal/k66jpb5/ https://old.reddit.com/r/LocalLLaMA/comments/17e855d/llamacp...
- DANmode 3y agoHuman perception of how weathered or safe their surroundings are seems to vary extraordinarily. Posting a (cropped?) picture would make this comment way more interesting.
- tsunamifury 3y agoOr…
- darylteo 3y agoThe model or the garage?
- birdyrooster 3y agoWatching the chat gpt workflow is cool because it captures your intent along with the code change without need for a commit comment
- lxe 3y agoIt’s been nothing short of “Jarvis”. You do have to ask things correctly to get correct answers, but it accelerates pretty much everything I do on a computer by a factor of 100.
- enoch2090 3y agoThis is cool! One thing that can be easily improved is that the narrator is still trying to describe everything, while people using Be My Eyes they usually got specific purposes (or "attention"). Would be nice to also implement whisper.cpp and use query inputs with some CoT or prompting to refine the description.
- lxe 3y agoThis is an awesome idea! Yeah refining and asking more questions about the image could make it viable. The BakLlava model is pretty impressive, but I do need to tune the prompts and the hyperparams as well. Or even do a small finetune. It’s a fun space to dive into.
- umtksa 3y agoawesome, love the idea and maybe putting a default prompt like: "this images comes from the persons POV that ask you questions" instead of "a person holding something" "you are holding xyz" wold be better
- FrenchDevRemote 3y agoDo you know how the model accuracy compares to GPT-4 vision?
- lxe 3y agoLet me see if I can find existing benchmark results. My gut feeling is that GPT4 is better overall... But BakLLaVA model is just incredibly small for how powerful it is.
- thih9 3y agoImpressive! Is the chatgpt transcript from the free chatgpt version (3.5) or the paid one (v4)?
- leodriesch 3y agoThe purple icon in the shared transcript indicates GPT-4.
- devinprater 3y agoA "Be My Eyes web app"? Well whatever. I guess Be My Eyes is just gonna be known for its Be My AI anyway. As soon as Llava is able to read text, this is gonna be pretty nice. I just wish it wasn't basically using Be My Eyes as a name to jump off of. This kind of local tech will be pretty amazing once it's fast enough to help with video games, or inaccessible apps. Add the ability for the AI to move the mouse in some of the GUI controller AI packages, and I could just say "click the next button and tell me what's on the screen afterwards" kind of thing.
- anon23432343 3y agoWait an actual cool peace tech on the HN front page? What happened here 0_0 Cool peace of tech!! keep on doing great work.
- anon23432343 3y ago[flagged]
- ReD_CoDE 3y agoIs there any solution that just identifies the object as accurate as possible? For instance when it sees a t-shirt, can be identify the exact brand and model?
- lxe 3y agoYes, GPT Vision / ChatGPT is very accurate. CogVLM is also very powerful as a self-hosted solution.