xyc

8 ms·

1. 61.▲
   by xyc 2y ago

   This is awesome! We used Qdrant vector DB for a local AI RAG app https://recurse.chat/blog/posts/local-docs#vector-database , but was eyeing up sqlite-vss as a lightweight embedded solution. Excited to see you are
2. 62.▲
   Chat with PDF locally using Llama 3(recurse.chat)

   4 points by xyc 2y ago|5 comments
3. 63.▲
   by xyc 2y ago

   Hi Hackers! A couple of months back, we received great feedback when launching RecurseChat, an app to help you use local AI as a daily driver. Recently we added features to chat with local documents / Llama 3 support and wrote about it
4. 64.▲
   by xyc 2y ago

   Andrej Karpathy's course is a good resource: https://www.youtube.com/playlist?list=PLAqhIrjkxbuWI23v9cThs...
5. 65.▲
   by xyc 2y ago

   Quite possible that llama.cpp already supports WizardLM 2: https://github.com/ggerganov/llama.cpp/issues/6691
6. 66.▲
   by xyc 2y ago

   What did they do to support WizardLM 2? It seems to work with an earlier llama.cpp version. (I have an app in production that uses a llama.cpp version before WizardLM 2 release)
7. 67.▲
   by xyc 2y ago

   Mistral-7B-Instruct-v0.2 is amazing. I've set it as the default model of RecurseChat (a local AI chat app). It works great, until recently WizardLM 2 came out and I tried it with local RAG. The difference is quite significant: https:&
8. 68.▲
   by xyc 2y ago

   We recently added support for local document chat in RecurseChat ( https://recurse.chat ), including chatting with PDFs and markdown. You can see a demo here: https://twitter.com/chxy/status/1777234458372
9. 69.▲
   by xyc 2y ago

   Tried 7b q5 with some RAG tasks. Seems quite impressive.
10. 70.▲
    by xyc 2y ago

    Also gguf files by abroxis: https://huggingface.co/ABX-AI/WizardLM-2-7B-GGUF-IQ-Imatrix
11. 71.▲
    by xyc 3y ago

    Thanks for the suggestion! I play with Apple Shortcuts sometimes. It's an exemplary example of how easy end user programming could be. Will keep this in mind.
12. 72.▲
    by xyc 3y ago

    Thank you! and thanks so much for the feature suggestions: - Make the system font (San Francisco) an option for the UI. Maybe even SF Mono as an option as well? Reasonable request! Won't be too hard to add - A little more help about wh
13. 73.▲
    by xyc 3y ago

    Thanks! honestly it's a quick hack together compared to the app. screenshots are from screen.studio. website is built with https://astro.build
14. 74.▲
    by xyc 3y ago

    Nice suggestion! Threading / branching won't be too crazy to support. I'll explore ChatGPT style branch or threads and see what'll work better.
15. 75.▲
    by xyc 3y ago

    Appreciate your support. Thank you so much!
16. 76.▲
    by xyc 3y ago

    Thank you for the support and the valuable feedback! Sorry about the response time, I haven't expected the incoming volume of requests. * For changing prompt in the middle - I'll take a crack at it this week. It's on top of m
17. 77.▲
    by xyc 3y ago

    Oh wow it's the goat himself, love how your work has democratized AI. Thanks so much for the encouragement. I'm mostly a UI/app engineer, total beginner when it comes to llama.cpp, would love to learn more and help along the
18. 78.▲
    by xyc 3y ago

    I think this is a good take. While there's big enough niche for personal data locally, I'd love if there's a way to solve for email/cloud data requiring API keys.
19. 79.▲
    by xyc 3y ago

    Unfortunately not now. If you are interested in email updates: https://tally.so/r/wzDvLM
20. 80.▲
    by xyc 3y ago

    Good to know. I've learned lots of things from Simon Willison's blog (datasette's author), so can't imagine llm being unuseful.
21. 81.▲
    by xyc 3y ago

    Thank you! I'd love to learn more about your use cases. Would you mind sending an email to feedback@recurse.chat or DM me on https://x.com/chxy to get the conversation started?
22. 82.▲
    by xyc 3y ago

    yes I wish it could talk. It's after other priorities though, but I might try something experimental.
23. 83.▲
    by xyc 3y ago

    Agree, there's a non real-time angle to this.
24. 84.▲
    by xyc 3y ago

    no this doesn't use ollama, just based on llama.cpp.
25. 85.▲
    by xyc 3y ago

    Appreciate the feedback! It works on mac with Apple Silicon only. I'll put some system requirements on the website.
26. 86.▲
    by xyc 3y ago

    Thank you!
27. 87.▲
    by xyc 3y ago

    Thanks! Sorry no immediate plan. People have recommended Chat with RTX so it might be worth checking out. https://www.nvidia.com/en-us/ai-on-rtx/chat-with-rtx-generat...
28. 88.▲
    by xyc 3y ago

    haven't tried Raindrop.io, looks neat! Saw some other posts mentioning bookmarks as well. I'll keep this in thought, but will have to try it out first to find out.
29. 89.▲
    by xyc 3y ago

    Good question, I'll put some system requirements on the website. It only supports mac with Apple Silicon now, if that's helpful.
30. 90.▲
    by xyc 3y ago

    Thanks so much for the kind words and giving it a spin! Feel free to send feedback, issues, feature suggestion as you use it more, I'm all ears. My twitter DM is also open: https://x.com/chxy .