13 ms·

Show HN: Khoj – Chat offline with your second brain using Llama 2

565 points by 110 3y ago

Hi folks, we're Debanjum and Saba. We created Khoj as a hobby project 2+ years ago because: (1) Search on the desktop sucked; we just had keyword search on the desktop vs google for the internet; and (2) Natural language search models had become good and easy to run on consumer hardware by this point. Once we made Khoj search incremental, I completely stopped using the default incremental search (C-s) in Emacs. Since then Khoj has grown to support more content types, deeper integrations and chat (using ChatGPT). With Llama 2 released last week, chat models are finally good and easy enough to use on consumer hardware for the chat with docs scenario. Khoj is a desktop application to search and chat with your personal notes, documents and images. It is accessible from within Emacs, Obsidian or your Web browser. It works with org-mode, markdown, pdf, jpeg files and notion, github repositories. It is open-source and can work without internet access (e.g on a plane). Our chat feature allows you to extract answers and create content from your existing knowledge base. Example: "What was that book Trillian mentioned at Zaphod's birthday last week". We personally use the chat feature regularly to find links, names and addresses (especially on mobile) and collate content across multiple, messy notes. It works online or offline: you can chat without internet using Llama 2 or with internet using GPT3.5+ depending on your requirements. Our search feature lets you quickly find relevant notes, documents or images using natural language. It does not use the internet. Example: Search for "bought flowers at grocery store" will find notes about "roses at wholefoods". Quickstart: pip install khoj-assistant && khoj See https://docs.khoj.dev/#/setup https://docs.khoj.dev/#/setup for detailed instructions We also have desktop apps (in beta) at https://github.com/khoj-ai/khoj/releases/tag/0.10.0 https://github.com/khoj-ai/khoj/releases/tag/0.10.0 if you want to try them out. Please do try out Khoj and let us know if it works for your use cases? Looking forward to the feedback!

marcus_holmes 3y ago
Any chance it can look at Gitlab as well? I like the idea but I'm not giving all my work to Microsoft.
Roark66 3y ago
From previous answers it appears you're using standard lama-7b (quantized to 4 bits). I suppose you're doing a search on the notes than you pass what you found with the original query to lama. This technique is cool, but there are many limitations. For example lama's content length. I can't wait for software that will take my notes each day and fine tune a LLM model on them so I can use entire context length for my question/answers.
- ankit219 3y ago
  > I can't wait for software that will take my notes each day and fine tune a LLM model on them so I can use entire context length for my question/answers Problem is finetuning does not work that way. Finetuning is useful when you want to teach a model about a certain pattern, not when you want it output it right. Eg: With enough finetuning and prompts, a model will be able to output the result in a certain format that you need, but it does not guarantee that it would not be hallucination prone. The best way to minimize hallucination is still embedding based retrieval passed along with the question/prompt. In future, there can be a system where you can build a knowledge base for LLMs, and tell it to access that for any knowledge, and finetune it for the patterns you want the output in.
Hypergraphe 3y ago
Hi, my dream app ! Will it work on non english sources ?
- 110 3y ago
  To use Chat with non-english sources you'll need to enable OpenAI. Offline chat with Llama 2 can't do that yet. And Search can be configured to work with 50+ languages. You'll just need to configure the asymmetric search model khoj uses to paraphrase-multilingual-MiniLM-L12-v2 in your ~/.khoj/khoj.yml config file For setup details see http://docs.khoj.dev/#/advanced?id=search-across-different-languages http://docs.khoj.dev/#/advanced?id=search-across-different-l...
  - Hypergraphe 3y ago
    Thank you for your reply. I was thinking of using a translation model to translate all of my documents to english before indexing them.
kljuka 3y ago
I tried the search using Slavic language (all my notes are in Slovene) - it performed very poorly: if the searched keyword was not directly in the note itself, the search results seemed to be more or less random.
- omniglottal 3y ago
  So you're saying you got no results when searching for patterns which did not exist in the dataset...?
  - kljuka 3y ago
    There was always full list of results. They just weren't relevant.
- tom910 3y ago
  Yes, I confirm. I have many articles in Russian language and the search can not find relative information, but if I try to search in English it works fine and can find documents that use English
- 110 3y ago
  Search should work with Slavic languages including Russian and 50+ other languages. You'll just need to configure the asymmetric search model khoj uses to paraphrase-multilingual-MiniLM-L12-v2 in your ~/.khoj/khoj.yml config file See http://docs.khoj.dev/#/advanced?id=search-across-different-languages http://docs.khoj.dev/#/advanced?id=search-across-different-l...
  - 110 3y ago
    Khoj chat with Llama2 will not work with non-english languages though. You'll have to enable OpenAI for that
pachico 3y ago
Would anybody be able to recommend any standalone solution (essentially data must not leave elsewhere) to chat with documents with a web interface? I tried privategpt but results were not great.
- 110 3y ago
  Khoj provides exactly that; it runs on your machine, none of your data leaves your machine and it has a web interface to chat
btbuildem 3y ago
Heads up, docker build fails with: #12 2.017 ERROR: Could not find a version that satisfies the requirement pyside6>=6.5.1 (from khoj-assistant) (from versions: none) #12 2.017 ERROR: No matching distribution found for pyside6>=6.5.1 ------ executor failed running [/bin/sh -c sed -i 's/dynamic = \["version"\]/version = "0.0.0"/' pyproject.toml && pip install --no-cache-dir .]: exit code: 1
- sabaimran 3y ago
  Darn, I've seen this error a couple of times. Can you drop a couple of details in this Github issue? https://github.com/khoj-ai/khoj/issues/391 https://github.com/khoj-ai/khoj/issues/391 I'm particularly interested in your OS/build environment.
  - btbuildem 3y ago
    Sure thing! I left a comment. The "buildx" flag gets me past that one, and to the next error: #0 12.37 ERROR: Could not find a version that satisfies the requirement gpt4all>=1.0.7 (from khoj-assistant) (from versions: 0.1.5, 0.1.6, 0.1.7) #0 12.37 ERROR: No matching distribution found for gpt4all>=1.0.7
blinkingled 3y ago
Somewhat unrelated but do people have links to share that walk you through taking Llama2 model and feeding it local data - confluence links, Google docs, plain text documents etc. I came across embeddings and langchain but was curious if people had thoughts on better ways to go about it as a newcomer experiment.
Crowberry 3y ago
It would be pretty awesome if this could be hooked up into Jira and Confluence as well!
bajirao 3y ago
What's the recommended 'size' of the machine to run this? I tried to run it on a pretty beefy machine (8 core cpu/32 GB RAM) to use with ~40 odd PDF documents. My observation is that the queries (chat) takes forever and also getting Segmentation fault (core dumped) for every other or so query.
- 110 3y ago
  Thanks for the feedback. Does your machine have a GPU? 32GB CPU RAM should be enough but GPU speeds up response time. We have fixes for the seg fault[1] and improvement to the query speed[2] that should be released by end of day today[3]. Update khoj to version 0.10.1 with pip install --upgrade khoj-assistant later today to see if that improves your experience. The number of documents/pages/entries doesn't scale memory utilization as quickly and doesn't affect the search, chat response time as much [1]: The seg fault would occur when folks sent multiple chat queries at the same time. A lock and some UX improvements fixed that [2]: The query time improvements are done by increasing batch size, to trade-off increased memory utilization for more speed [3]: The relevant pull request for reference: https://github.com/khoj-ai/khoj/pull/393 https://github.com/khoj-ai/khoj/pull/393
Bonapara 3y ago
Congrats guys!
- 110 3y ago
  Thanks! :)
smcleod 3y ago
I've been playing with Khoj for the past day - it's really neat, well done! A few observations: 1. Telemetry is enabled by default, and may contain the API and chat queries. I've logged an issue for this along with some suggestions here: https://github.com/khoj-ai/khoj/issues/389 https://github.com/khoj-ai/khoj/issues/389 2. It would be advantageous to have configuration in the UI rather than baking it's YAML into the container image. (added a note on that in the aforementioned issue on Github). 3. It's not clear if you can bring your own models, e.g. can I configure a model from huggingface/gpt4all? if so, will it be automatically downloaded based on the name or should I put the .bin (and yaml?) in a volume somewhere? 4. AMD GPU/APU acceleration (CLBLAS) would be really nice, I've logged an issue for this feature request as well. https://github.com/khoj-ai/khoj/issues/390 https://github.com/khoj-ai/khoj/issues/390
- sabaimran 3y ago
  Thanks for the feedback! Much appreciated. I responded in the issue, but I'll paste here as well for those also curious: Khoj does not collect any search or chat queries. As mentioned in the docs, you can see our telemetry server[1]. If you see anything amiss, point it out to me and I'll hotfix it right away. You can see all the telemetry metadata right here[2]. [1]: https://github.com/khoj-ai/khoj/tree/master/src/telemetry https://github.com/khoj-ai/khoj/tree/master/src/telemetry [2]: https://github.com/khoj-ai/khoj/blob/master/src/khoj/routers/helpers.py#L29 https://github.com/khoj-ai/khoj/blob/master/src/khoj/routers... Configuration with the `docker-compose` setup is a little bit particular, see the issue^ for details. Thanks for the reference points for GPU integration! Just to clarify, we do use GPU optimization for indexing, but not for local chat with Llama. We're looking into getting that working.
- tmzt 3y ago
  Would it be possible to support a custom URL for the local model, such as running ./server in ggml would give you? This may be more difficult if you are pre-tokenizing the search context. Very cool project.
jsdeveloper 3y ago
will it work on linux?(ubuntu)
- 110 3y ago
  Yes, of course
asynchronous 3y ago
This is very cool, the Obsidian integration is a neat feature. Please, someone make a home-assistant Alexa clone for this.
- 110 3y ago
  Thanks! We've just been testing integrating over voice, whatsapp over the last few days[1][2] :) [1]: https://github.com/khoj-ai/khoj/tree/khoj-chat-over-whatsapp-sms-with-twilio https://github.com/khoj-ai/khoj/tree/khoj-chat-over-whatsapp... [2]: https://github.com/khoj-ai/khoj/compare/master...features/whatsapp-anthropic-demo https://github.com/khoj-ai/khoj/compare/master...features/wh...
mmanfrin 3y ago
As someone who's been getting int o using Obsidian and messing around with chat ais, this is excellent, thank you!
- 110 3y ago
  Thanks! Do try it out and let us know if it works for your use-case?
- tarwin 3y ago
  Really encourages me to move to Obsidion :D
matmulbro 3y ago
[flagged]
- isoprophlex 3y ago
  Please don't post low effort, shallow dismissals; without substantiation you're not posting anything useful, you're just a loud asshole.
mlajtos 3y ago
Have anyone got something valuable from talking to your second brain? What kind of conversations are you trying to have?
- bozhark 3y ago
  Traumatic Brain Injury. I can’t remember yesterday. Would be hella nice to connect all the scattered lines of thoughts in various notes on a variety of platforms.
  - mlajtos 3y ago
    I am sorry. Would some summary of previous day would be helpful to you? Is your memory problem only episodic, or does it extend to factual and kinesthetic as well?
    - samstave 3y ago
      I want a body cam that I wear and it transcribes into something searchable from things I did... Basically like a gopro on steroids with searchable context - or even the ability for me to say outloud "KEEP A NOTE OF THIS" and it will keep a segment tagged and can give me summaries of moments I wanted particularly logged... I applied to YC with an idea 'sorta' like this almost a decade ago. The idea was to have a timeline of communications between all my contacts such that I could side-scroll a timeline with dots of actions such a "sent email" "made call" "sent text" received txt" and I could see all these in filters by contacts/day whatever... This was pre-snowden, so I didnt have confirmation that there were already people doing this for me, just not letting me browse my own data ;-)
      - erklik 3y ago
        > Basically like a gopro on steroids with searchable context - or even the ability for me to say outloud "KEEP A NOTE OF THIS" and it will keep a segment tagged and can give me summaries of moments I wanted particularly logged... This is generally called Lifelogging. https://roberdam.com/en/wisper.html https://roberdam.com/en/wisper.html - roberdam@ created basically what you just said, but focused on Audio, not Video. https://news.ycombinator.com/item?id=29692087 https://news.ycombinator.com/item?id=29692087 has some possible info too.
spdustin 3y ago
I see you’re using gpt4all; do you have a supported way to change the model being used for local inference? A number of apps that are designed for OpenAI’s completion/chat APIs can simply point to the endpoints served by llama-cpp-python [0], and function in (largely) the same way, while using the various models and quants supported by llama.cpp. That would allow folks to run larger models on the hardware of their choice (including Apple Silicon with Metal acceleration or NVIDIA GPUs) or using other proxies like openrouter.io. I enjoy openrouter.io myself because it supports Anthropic’s 100k models. [0]: https://github.com/abetlen/llama-cpp-python https://github.com/abetlen/llama-cpp-python
- 110 3y ago
  No, we don't yet. Lots of developer folks want to try different models, we want to provide simple to use, but deep assistance. Kind of unsure what to focus on given our limited resources.
  - vunderba 3y ago
    I really like the idea of running a dedicated server that serves up various large language models via a standardized API, and then Khoj could just be pointed at one. Depending on the notes and the type of conversation I want to have, that would even allow for Khoj to swap models on the fly.
    - sabaimran 3y ago
      We're planning to work on this!
- syntaxing 3y ago
  The point of gpt4all is that you can change the model with minimal breaking. You should be able to change this line https://github.com/khoj-ai/khoj/blob/master/src/khoj/processor/conversation/gpt4all/chat_model.py#L19 https://github.com/khoj-ai/khoj/blob/master/src/khoj/process... to the model you want. You'll need to build your own local image with docker-compose but should be relatively straight forward.
LanternLight83 3y ago
It's funny that you mention `C-s`, because `isearch-forward` is usually used for low-latency literal matches. In what workflow can Khoj offer acceptable latency or superior utility as a drop-in replacement for isearch? Is there an example of how you might use it to navigate a document?
- 110 3y ago
  That's (almost) exactly what khoj search provides a search-as-you-type experience but with a natural language (instead of keyword) search interface. My workflow looks like: 1. Search with Khoj search[1]: `C-c s s` <search-query> RET 2. Use speed key to jump to relevant entry[2]: with `n n o 2` [1]: `C-c s` is bound to `khoj` transient menu [2] https://orgmode.org/manual/Speed-Keys.html https://orgmode.org/manual/Speed-Keys.html
  - Blackthorn 3y ago
    Can you elaborate on that a little bit? When you search like that, what do you type in to find stuff in a programming language source file? I'd like to better understand this workflow, it seems interesting and I might be missing out.
    - 110 3y ago
      Firstly, my previous statement needs quite a lot of clarification: Khoj works more like an incremental, natural language version of org-agenda-search (or projectile) rather than isearch. You've to configure which files it should index first. You can then use natural language to search those files with a search-as-you-type experience. This is not the same as isearch that just searches the current file for keyword matches. Second, khoj doesn't index source code and the default search models don't work well with code files. But yeah someone should implement a natural language isearch as a standalone tool (as suggested by parent comment). It'd be super-useful
bozhark 3y ago
I’m not a software dev. Is there a way to have this bot read from a discord and google drive?
- syntaxing 3y ago
  gpt4all itself (the library on the backend for this) has a similar program [1]. You just need to put everything into a folder. This should be straight forward for google drive. Harder for discord though but I’m sure theres a bot online that can do the extraction. [1] https://gpt4all.io/index.html https://gpt4all.io/index.html
RomanHauksson 3y ago
Awesome work, I've been looking for something like this. Any plans to support Logseq in the future?
- 110 3y ago
  Yes, we hope to get to it soon! This has been an ask on our Github since a while[1] [1]: https://github.com/khoj-ai/khoj/issues/141 https://github.com/khoj-ai/khoj/issues/141
Ilnsk 3y ago
[flagged]
- 110 3y ago
  2.5 years! We're kind of slow :P
- tudorw 3y ago
  hi, you seem keen to share something neat you took less than 10 minutes to implement, I'd love to see that?
jigneshdarji91 3y ago
This would be even great if available as a Spotlight Search replacement (with some additional features that Spotlight supports).
- 110 3y ago
  Yeah, this would be ideal for Mac users. Just need to look into what is required and how much work it is
- tough 3y ago
  Should be easy to plug it in with a Raycast.app or Alfred.app plugin.
  - 110 3y ago
    Khoj exposes a local API. Hopefully that makes it easy to integrate with Raycast, Alfred (or even Spotlight)?
agg23 3y ago
Just a heads up, your landing page on your website doesn't seem to mention Llama/the offline usecase at all, only online via OpenAI. ---- What model size/particular fine-tuning are you using, and how have you observed it to perform for the usecase? I've only started playing with Llama 2 at 7B and 13B sizes, and I feel they're awfully RAM heavy for consumer machines, though I'm really excited by this possibility. How is the search implemented? Is it just an embedding and vector DB, plus some additional metadata filtering (the date commands)?
- rapnie 3y ago
  > Just a heads up, your landing page on your website doesn't seem to mention Llama/the offline usecase at all, only online via OpenAI. I am sufficiently uneducated on the ins and outs of AI integrations to always wonder if projects like this one can be used in local-only mode, i.e. when self-hosted ensuring me that never any of my personal information is sent to a remote service. So it would be very helpful to very explicitly give me that assurance of privacy, if that's the case.
  - 110 3y ago
    Yes, this seems to be a common concern. We're trying to see how best to crisply address it. But yes, Khoj can be used even with your internet turned off
- 110 3y ago
  Thanks for the pointer, yeah the website content has gone stale. I'll try update it by end of day Khoj is using the Llama 7B, 4bit quantized, GGML by TheBloke. It's actually the first offline chat model that gives coherent answers to user queries given notes as context. And it's interestingly more conversational than GPT3.5+, which is much more formal
  - moneywoes 3y ago
    Is a vector db used?
  - M4v3R 3y ago
coder543 3y ago
This seems like a cool project. It would be awesome if it could also index a directory of PDFs, and if it could do OCR on those PDFs to support indexing scanned documents. Probably outside of the scope of the project for now, but just the other day I was just thinking how nice it would be to have a tool like this.
- samstave 3y ago
  Ive wanted a crawler on my machine for auto-categorizing and organizing, tagging and moving ALL my files around based on all my machines - so the ability to crawl PDFs, downloads, screenshots, pictures, etc and give me a logical tree of the org of the files - and allow me to modify it by saying "add all PDF related to [subject] here and the organize by source/author etc... and then move all my screenshots, ordered by date here etc... I've wanted a "COMPUTER.", uh... I say "COMPUTER!", 'sir, you have to use the keyboard', ah a Keyboard, how quaint.... forever.
  - 110 3y ago
    That.would.be.awesome! Khoj isn't their yet, but that actually shouldn't be too far away if you give it a voice interface and terminal access. Of course, having it be stable enough to not `rm -rf /` soon after is definitely not part of the warranty
    - samstave 3y ago
      Copy first, move and delete after confirmation from user on big moves. - So aside from doing something such as this - whats the size of your current usecase - is it being able to search and org a large orge of devs through files and directories/repos/whatever? What if you had a global glob of data/code/files - and as an enterprise you can just give each new emp an employee role, and based on their job and purported skills, just throw them at the monolith, yet the AI will tailor the info they get out of the glob. "User10267A - you are chartered with working on projects XYZ, based on your skillset - here are a few assessments to see which sections of the codebase best suit you." [does work] "OK .U267A, here is your workspace with all auths to the libs and such youll need to work on [PROJECT FSPEZ]"
ramesh31 3y ago
Something I've noticed playing around with Llama 7b/13b on my Macbook is that it clearly points out just how little RAM 16GB really is these days. I've had a lot of trouble running both inference and a web UI together locally when browser tabs take up 5GB alone. Hopefully we will see a resurgence of lightweight native UIs for these things that don't hog resources from the model.
- Kwpolska 3y ago
  Or hopefully we will see an end of the LLM hype. Or at least models that don’t hog so much RAM.
  - ramesh31 3y ago
    >Or at least models that don’t hog so much RAM The RAM usage is kind of the point though; we're trading space for time. It's not a problem that the model is using it, it's just that with the default choice for UI being web based now, the unnecessary memory usage of browsers is actually starting to be a real pain point.
    - 110 3y ago
      1. I hear you on going back to lightweight native apps. Unfortunately the Python ecosystem is not great for this. We use pyinstaller to create the native desktop app but it's a pain to manage. 2. The web UI isn't required if you use Obsidian or Emacs. That's just a convenient, generic interface that everyone can use.
- thenickdude 3y ago
  The new Chrome "memory saver" feature that discards the contents of old tabs saves a lot of memory for me. Tabs get reloaded from the server if you revisit them.
- sabaimran 3y ago
  FWIW I've also had browser RAM consumption issues in life, but it's been mitigated by extensions like OneTab: https://chrome.google.com/webstore/detail/onetab/chphlpgkkbolifaimnlloiipkdnihall https://chrome.google.com/webstore/detail/onetab/chphlpgkkbo... For now, local LLMs take up an egregious about of RAM, totally agreed. But we trust the ecosystem is going to keep improving and growing and we'll be able to make improvements over time. They'll probably become efficient enough where we can run them on phones, which will unlock some cool scope for Khoj to integrate with on device, offline assistance.
wg0 3y ago
I have not tried it but something like this should exist. I don't think it is going to be as useable on consumer hardware as yet unless you have a good enough GPU but within couple of years (or less), we'll be there I am sure. Irrelevant opinion - The logo is beautiful, I like it and so are the colours used. Lastly, LLMA2 for such use cases, I think is capable enough that paying for ChatGPT won't be as lucrative especially when privacy is of concern. Keep it up. Good craftsmanship. :)
- sabaimran 3y ago
  Thanks! I do think Llama V2 is going to be a good enough replacement for ChatGPT (aka GPT3.5) for a lot of use cases.
School-Cotton 3y ago
Markdown doesn't work on HN...
IshKebab 3y ago
Interesting. The obvious question you haven't answered anywhere (as far as I can see) is what are the hardware requirements to run this locally?
- 110 3y ago
  Ah, you're right, forgot to mention that. We use the Llama 2 7B 4 bit quantized model. The machine requirements are: Ideal: 16Gb (GPU) RAM Less Ideal: 8GB RAM and CPU
  - HexDecOctBin 3y ago
    Sorry for the repetition, but do you mean 16 GB VRAM? That is a very high requirement, a RTX 4060 only has 8GB and even a RTX 4070 only ships with 12GB. Any upcoming further optimizations for reducing memory usage? PS. Nice to see an Hindi name for a software. For those who don't speak Hindi: https://en.m.wiktionary.org/wiki/%E0%A4%96%E0%A5%8B%E0%A4%9C#:~:text=%E0%A4%96%E0%A5%8B%E0%A4%9C%20%E2%80%A2%20(khoj)%20f%20(,finding%2C%20exploration https://en.m.wiktionary.org/wiki/%E0%A4%96%E0%A5%8B%E0%A4%9C...
    - intended 3y ago
      One of the reasons 4090 / 3090s are expensive these days. It’s an issue with the models, not with Khoj.
  - danparsonson 3y ago
    So just to clarify, is that: Ideal is running the model on a GPU (any brand? Nvidia, AMD, etc.?) with 16GB of GPU RAM, less ideal is running it on the CPU, for which it needs 8GB system RAM? Presumably it will occupy all that memory while it's running? What about if I have a GPU with 8GB?
    - 110 3y ago
      Khoj uses Llama 2 7B, 4bit quantized. So it just needs 3.5Gb of RAM (GPU or System) [1]. Khoj and your other apps need more RAM themselves, so practically 8GB of System or GPU RAM should suffice. Khoj has been tested with CUDA and Metal capable GPUs. So Nvidia and Mac M1+ GPUs should work. I'm think it'll work with AMD GPUs out of the box too but let me know if it doesn't for you? I can look into what needs to be done to get that to work. [1]: The calculation is [params] * [bytes] GB RAM, so 7 * 0.5 = 3.5Gb
calnayak 3y ago
How does one access this from a web browser?
- sabaimran 3y ago
  We have a cloud product you can sign up for, but it's more limited in what data sources it supports. It currently only works for Notion and Github indexing. If you're interested in that, send me a dm on Discord - https://discord.gg/BDgyabRM6e https://discord.gg/BDgyabRM6e But that would allow you to access Khoj from the web.
overnight5349 3y ago
Could this do something like take in the contents of my web history for the day and summarize notes on what I've been researching? This is getting very close to my ideal of a personal AI. It's only gonna be a few more years until I can have a digital brain filled with everything I know. I can't wait
- usehackernews 3y ago
  Interesting, this is the exact question that came to mind for me. This would address a pain point for me. Does anyone have recommendations for a tool that does it? Or, anyone want to build it together?
- 110 3y ago
  That would be pretty awesome. Building a daily web history summarizer as a browser extension shouldn't be too much work. I bet there's something like that already out there. Having something that indexes all your digital travels and makes it easily digestible will be gold. Hopefully Khoj can become that :)
  - reitanqild 3y ago
    > I bet there's something like that already out there. There was. It was called Google Desktop Search, it was awesome, and it was axed. That said, today I wouldn't use it anyway as both I and Google have changed a lot.
    - 110 3y ago
      Oh yeah, I used to use Google Desktop too. It was awesome for it's time
- porcc 3y ago
  https://www.rewind.ai https://www.rewind.ai
  - aliasxneo 3y ago
ubertaco 3y ago
Hey, I saw Khoj hit HN a few weeks ago and get slaughtered because the messaging didn't match the product. You've come a good way in both directions: the messaging is clearer about current state vs aspirations, and you've made good progress towards the aspirational parts. Really glad to see the warm reception you're getting now. Nice job, y'all.
- sabaimran 3y ago
  Hey ubertaco! I remember you. Appreciate the well-wishes. The landing page still needs some tweaking. It's kind of hard keeping what you're building in sync with what you're aspiring for, but we're definitely working towards it.
yberreby 3y ago
Cool project. I tried it last time this got posted, but it was still a bit buggy. Giving it another shot - I'm mainly interested in the local chat. Could you elaborate on the incremental search feature? How did you implement it? Don't you need to re-encode the full query through a SBERT or such as each token is written (perhaps with debouncing)? Also, having an easily-extended data connector interface would be awesome, to connect to custom data sources.
- 110 3y ago
  Buggy for setup? We've done some improvements and have desktop apps (in beta) too now to simplify this. Feel free to report any issues on the khoj github. I can have a look. Yes, we don't do optimizations on the query encoding yet. So SBERT just re-encodes the whole query every time. It gets results in <100ms which is good enough for incremental search. I did create a plugin system, so that a data plugin just has to convert the source data into a standardized intermeditate jsonl format. But this hasn't been documented or extensively tested yet.
mavsman 3y ago
Really cool to see this! Local is the real future of AI. I got really excited about this and fired it up on my petite little M2 Macbook Air only for it to grind it to a halt. Think the old days when you had a virus on your PC and you'd move the mouse then wait 45 seconds to see the cursor move. It honestly made me feel nostalgic. I guess I have to taper performance expectations with this Air, though this is the first time it's happened.
- quickthrower2 3y ago
  Just wait 10 years when computers have a dedicated AI-PU and you don't have to worry about freezing anything up to talk to your bot.
  - kridsdale1 3y ago
    The M1 line of Macs does have that unit.
  - joaogui1 3y ago
    I wonder if edge tpus (sold here https://coral.ai/products/ https://coral.ai/products/) could help, though I guess the community would have to optimize for them instead of for standard hardware
- jmorgan 3y ago
  How much memory do you have in your Macbook? The 7B models seem to work well with at least 16GB of unified memory, but I’ve seen Macs with 8GB really struggle.
  - mavsman 3y ago
    Indeed it's just a poor little 8GB of RAM.
throw03172019 3y ago
Feedback for landing page: use a fixed height container for the example prompts. Without it, it causes jumping while scrolling down the page making other sections hard to read. iOS Safari
- 110 3y ago
  Thanks for the feedback! Someone else mentioned this issue the other day as well. I'll fix this issue on the landing page soon
thangngoc89 3y ago
I’m in search of a new Macbook Mx. what is the requirements for running these model locally without breaking the bank? Would 32GB be enough?
- 110 3y ago
  You do not need to break the bank to use Khoj for local chat, 16Gb RAM should be good enough
  - DoctorOetker 3y ago
    How slow would that be on an old non-Apple laptop, but also 16Gb RAM? lscpu output: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 36 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Vendor ID: GenuineIntel Model name: Intel(R) Core(TM) i7-3630QM CPU @ 2.40GHz CPU family: 6 Model: 58 Thread(s) per core: 2 Core(s) per socket: 4 Socket(s): 1 Stepping: 9 CPU(s) scaling MHz: 35% CPU max MHz: 3400.0000 CPU min MHz: 1200.0000 BogoMIPS: 4791.90 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cp uid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm cpuid_fault epb pti ssbd ibrs ibpb stibp tpr _shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts md_clear flush_l1d L1d: 128 KiB (4 instances) L1i: 128 KiB (4 instances) L2: 1 MiB (4 instances) L3: 6 MiB (1 instance) NUMA: NUMA node(s): 1 NUMA node0 CPU(s): 0-7
    - 110 3y ago
      It should work but not sure how fast. Best way to find out would be to try it. We also have some fixes and perf improvements we plan to release later today in version 0.10.1
IAmNotACellist 3y ago
What's the posthog telemetry used for? Why is there nothing on it in the docs? Why no clear way to opt out?
- Kerbonut 3y ago
  It’s pretty easy to remove which is what I ended up doing. The project works remarkably well otherwise.
- sabaimran 3y ago
  Thanks for pointing that out! We use it for understanding usage -- like determining whether people are using markdown or org or more. Everything is collected entirely anonymized, and no identifiable information is ever sent to the telemetry server. To opt-out, you set the `should-log-telemetry` value in `khoj.yml` to false. Updated the docs to include these instructions and what we collect -- https://docs.khoj.dev/#/telemetry https://docs.khoj.dev/#/telemetry.
wanderingmind 3y ago
Two comments 1. If you want better adoption especially among corporations, GPL-3 wont cut it. Maybe think of some business friendly licenses (MIT etc) 2. I understand the excitement about llm's. But how about making something more accessible to people with regular machines and not state of art. I use rip-grep-all (rga) along with fzf [1] that can search all files including pdfs in a specific folders. However, I would like a GUI tool to (a) search across multiple folders, (b) provide priority of results across folders, filetypes and (c) store search histories where I can do a meta-search. This is sufficient for 95% of my usecases to search locally and I don't need LLM. If khoj can enable such search as default without LLM that will be a gamechanger for many people without a heavy compute machine or who dont want to use OpenAI. [1] https://github.com/phiresky/ripgrep-all/wiki/fzf-Integration https://github.com/phiresky/ripgrep-all/wiki/fzf-Integration
- pvh 3y ago
  Just a note to suggest that giving away your hard work to those who will profit from it in the hope that they will remember you later seems like a pretty dubious exchange. Have a look at how that worked out for the folks who built node and its libraries versus the ones who maintained control of their work (like npm).
  - quickthrower2 3y ago
    What happened there. Surely the people who built node (or the people building the most popular fork at least) get to define what the default package manager is etc. and get some BATNA against the likes of a 3rd party package manager profiting from their thing. I don't know what the node/npm relationship story is though.
- Zuiii 3y ago
  If corporations have no issue with using restrictive proprietary licenses, they should not have any issues with the GPL.
- trenchgun 3y ago
  That seems like a pretty trivial thing to implement. Why not do it yourself?

565 points by 110 3y ago

marcus_holmes 3y ago

Any chance it can look at Gitlab as well? I like the idea but I'm not giving all my work to Microsoft.

Roark66 3y ago

From previous answers it appears you're using standard lama-7b (quantized to 4 bits). I suppose you're doing a search on the notes than you pass what you found with the original query to lama. This technique is cool, but there are many limitations. For example lama's content length. I can't wait for software that will take my notes each day and fine tune a LLM model on them so I can use entire context length for my question/answers.

ankit219 3y ago
> I can't wait for software that will take my notes each day and fine tune a LLM model on them so I can use entire context length for my question/answers Problem is finetuning does not work that way. Finetuning is useful when you want to teach a model about a certain pattern, not when you want it output it right. Eg: With enough finetuning and prompts, a model will be able to output the result in a certain format that you need, but it does not guarantee that it would not be hallucination prone. The best way to minimize hallucination is still embedding based retrieval passed along with the question/prompt. In future, there can be a system where you can build a knowledge base for LLMs, and tell it to access that for any knowledge, and finetune it for the patterns you want the output in.

Hypergraphe 3y ago

Hi, my dream app ! Will it work on non english sources ?

110 3y ago
To use Chat with non-english sources you'll need to enable OpenAI. Offline chat with Llama 2 can't do that yet. And Search can be configured to work with 50+ languages. You'll just need to configure the asymmetric search model khoj uses to paraphrase-multilingual-MiniLM-L12-v2 in your ~/.khoj/khoj.yml config file For setup details see http://docs.khoj.dev/#/advanced?id=search-across-different-languages http://docs.khoj.dev/#/advanced?id=search-across-different-l...
- Hypergraphe 3y ago
  Thank you for your reply. I was thinking of using a translation model to translate all of my documents to english before indexing them.

kljuka 3y ago

I tried the search using Slavic language (all my notes are in Slovene) - it performed very poorly: if the searched keyword was not directly in the note itself, the search results seemed to be more or less random.

omniglottal 3y ago
So you're saying you got no results when searching for patterns which did not exist in the dataset...?
- kljuka 3y ago
  There was always full list of results. They just weren't relevant.
tom910 3y ago
Yes, I confirm. I have many articles in Russian language and the search can not find relative information, but if I try to search in English it works fine and can find documents that use English
110 3y ago
Search should work with Slavic languages including Russian and 50+ other languages. You'll just need to configure the asymmetric search model khoj uses to paraphrase-multilingual-MiniLM-L12-v2 in your ~/.khoj/khoj.yml config file See http://docs.khoj.dev/#/advanced?id=search-across-different-languages http://docs.khoj.dev/#/advanced?id=search-across-different-l...
- 110 3y ago
  Khoj chat with Llama2 will not work with non-english languages though. You'll have to enable OpenAI for that

pachico 3y ago

Would anybody be able to recommend any standalone solution (essentially data must not leave elsewhere) to chat with documents with a web interface? I tried privategpt but results were not great.

110 3y ago
Khoj provides exactly that; it runs on your machine, none of your data leaves your machine and it has a web interface to chat

btbuildem 3y ago

Heads up, docker build fails with: #12 2.017 ERROR: Could not find a version that satisfies the requirement pyside6>=6.5.1 (from khoj-assistant) (from versions: none) #12 2.017 ERROR: No matching distribution found for pyside6>=6.5.1 ------ executor failed running [/bin/sh -c sed -i 's/dynamic = \["version"\]/version = "0.0.0"/' pyproject.toml && pip install --no-cache-dir .]: exit code: 1

sabaimran 3y ago
Darn, I've seen this error a couple of times. Can you drop a couple of details in this Github issue? https://github.com/khoj-ai/khoj/issues/391 https://github.com/khoj-ai/khoj/issues/391 I'm particularly interested in your OS/build environment.
- btbuildem 3y ago
  Sure thing! I left a comment. The "buildx" flag gets me past that one, and to the next error: #0 12.37 ERROR: Could not find a version that satisfies the requirement gpt4all>=1.0.7 (from khoj-assistant) (from versions: 0.1.5, 0.1.6, 0.1.7) #0 12.37 ERROR: No matching distribution found for gpt4all>=1.0.7

blinkingled 3y ago

Somewhat unrelated but do people have links to share that walk you through taking Llama2 model and feeding it local data - confluence links, Google docs, plain text documents etc. I came across embeddings and langchain but was curious if people had thoughts on better ways to go about it as a newcomer experiment.

Crowberry 3y ago

It would be pretty awesome if this could be hooked up into Jira and Confluence as well!

bajirao 3y ago

What's the recommended 'size' of the machine to run this? I tried to run it on a pretty beefy machine (8 core cpu/32 GB RAM) to use with ~40 odd PDF documents. My observation is that the queries (chat) takes forever and also getting Segmentation fault (core dumped) for every other or so query.

110 3y ago
Thanks for the feedback. Does your machine have a GPU? 32GB CPU RAM should be enough but GPU speeds up response time. We have fixes for the seg fault[1] and improvement to the query speed[2] that should be released by end of day today[3]. Update khoj to version 0.10.1 with pip install --upgrade khoj-assistant later today to see if that improves your experience. The number of documents/pages/entries doesn't scale memory utilization as quickly and doesn't affect the search, chat response time as much [1]: The seg fault would occur when folks sent multiple chat queries at the same time. A lock and some UX improvements fixed that [2]: The query time improvements are done by increasing batch size, to trade-off increased memory utilization for more speed [3]: The relevant pull request for reference: https://github.com/khoj-ai/khoj/pull/393 https://github.com/khoj-ai/khoj/pull/393

Bonapara 3y ago

Congrats guys!

110 3y ago
Thanks! :)

smcleod 3y ago

I've been playing with Khoj for the past day - it's really neat, well done! A few observations: 1. Telemetry is enabled by default, and may contain the API and chat queries. I've logged an issue for this along with some suggestions here: https://github.com/khoj-ai/khoj/issues/389 https://github.com/khoj-ai/khoj/issues/389 2. It would be advantageous to have configuration in the UI rather than baking it's YAML into the container image. (added a note on that in the aforementioned issue on Github). 3. It's not clear if you can bring your own models, e.g. can I configure a model from huggingface/gpt4all? if so, will it be automatically downloaded based on the name or should I put the .bin (and yaml?) in a volume somewhere? 4. AMD GPU/APU acceleration (CLBLAS) would be really nice, I've logged an issue for this feature request as well. https://github.com/khoj-ai/khoj/issues/390 https://github.com/khoj-ai/khoj/issues/390

sabaimran 3y ago
Thanks for the feedback! Much appreciated. I responded in the issue, but I'll paste here as well for those also curious: Khoj does not collect any search or chat queries. As mentioned in the docs, you can see our telemetry server[1]. If you see anything amiss, point it out to me and I'll hotfix it right away. You can see all the telemetry metadata right here[2]. [1]: https://github.com/khoj-ai/khoj/tree/master/src/telemetry https://github.com/khoj-ai/khoj/tree/master/src/telemetry [2]: https://github.com/khoj-ai/khoj/blob/master/src/khoj/routers/helpers.py#L29 https://github.com/khoj-ai/khoj/blob/master/src/khoj/routers... Configuration with the `docker-compose` setup is a little bit particular, see the issue^ for details. Thanks for the reference points for GPU integration! Just to clarify, we do use GPU optimization for indexing, but not for local chat with Llama. We're looking into getting that working.
tmzt 3y ago
Would it be possible to support a custom URL for the local model, such as running ./server in ggml would give you? This may be more difficult if you are pre-tokenizing the search context. Very cool project.

jsdeveloper 3y ago

will it work on linux?(ubuntu)

110 3y ago
Yes, of course

asynchronous 3y ago

This is very cool, the Obsidian integration is a neat feature. Please, someone make a home-assistant Alexa clone for this.

110 3y ago
Thanks! We've just been testing integrating over voice, whatsapp over the last few days[1][2] :) [1]: https://github.com/khoj-ai/khoj/tree/khoj-chat-over-whatsapp-sms-with-twilio https://github.com/khoj-ai/khoj/tree/khoj-chat-over-whatsapp... [2]: https://github.com/khoj-ai/khoj/compare/master...features/whatsapp-anthropic-demo https://github.com/khoj-ai/khoj/compare/master...features/wh...

mmanfrin 3y ago

As someone who's been getting int o using Obsidian and messing around with chat ais, this is excellent, thank you!

110 3y ago
Thanks! Do try it out and let us know if it works for your use-case?
tarwin 3y ago
Really encourages me to move to Obsidion :D

matmulbro 3y ago

[flagged]

isoprophlex 3y ago
Please don't post low effort, shallow dismissals; without substantiation you're not posting anything useful, you're just a loud asshole.

mlajtos 3y ago

Have anyone got something valuable from talking to your second brain? What kind of conversations are you trying to have?

bozhark 3y ago
Traumatic Brain Injury. I can’t remember yesterday. Would be hella nice to connect all the scattered lines of thoughts in various notes on a variety of platforms.
- mlajtos 3y ago
  I am sorry. Would some summary of previous day would be helpful to you? Is your memory problem only episodic, or does it extend to factual and kinesthetic as well?
  - samstave 3y ago
    I want a body cam that I wear and it transcribes into something searchable from things I did... Basically like a gopro on steroids with searchable context - or even the ability for me to say outloud "KEEP A NOTE OF THIS" and it will keep a segment tagged and can give me summaries of moments I wanted particularly logged... I applied to YC with an idea 'sorta' like this almost a decade ago. The idea was to have a timeline of communications between all my contacts such that I could side-scroll a timeline with dots of actions such a "sent email" "made call" "sent text" received txt" and I could see all these in filters by contacts/day whatever... This was pre-snowden, so I didnt have confirmation that there were already people doing this for me, just not letting me browse my own data ;-)
    - erklik 3y ago
      > Basically like a gopro on steroids with searchable context - or even the ability for me to say outloud "KEEP A NOTE OF THIS" and it will keep a segment tagged and can give me summaries of moments I wanted particularly logged... This is generally called Lifelogging. https://roberdam.com/en/wisper.html https://roberdam.com/en/wisper.html - roberdam@ created basically what you just said, but focused on Audio, not Video. https://news.ycombinator.com/item?id=29692087 https://news.ycombinator.com/item?id=29692087 has some possible info too.

spdustin 3y ago

I see you’re using gpt4all; do you have a supported way to change the model being used for local inference? A number of apps that are designed for OpenAI’s completion/chat APIs can simply point to the endpoints served by llama-cpp-python [0], and function in (largely) the same way, while using the various models and quants supported by llama.cpp. That would allow folks to run larger models on the hardware of their choice (including Apple Silicon with Metal acceleration or NVIDIA GPUs) or using other proxies like openrouter.io. I enjoy openrouter.io myself because it supports Anthropic’s 100k models. [0]: https://github.com/abetlen/llama-cpp-python https://github.com/abetlen/llama-cpp-python

110 3y ago
No, we don't yet. Lots of developer folks want to try different models, we want to provide simple to use, but deep assistance. Kind of unsure what to focus on given our limited resources.
- vunderba 3y ago
  I really like the idea of running a dedicated server that serves up various large language models via a standardized API, and then Khoj could just be pointed at one. Depending on the notes and the type of conversation I want to have, that would even allow for Khoj to swap models on the fly.
  - sabaimran 3y ago
    We're planning to work on this!
syntaxing 3y ago
The point of gpt4all is that you can change the model with minimal breaking. You should be able to change this line https://github.com/khoj-ai/khoj/blob/master/src/khoj/processor/conversation/gpt4all/chat_model.py#L19 https://github.com/khoj-ai/khoj/blob/master/src/khoj/process... to the model you want. You'll need to build your own local image with docker-compose but should be relatively straight forward.

LanternLight83 3y ago

It's funny that you mention `C-s`, because `isearch-forward` is usually used for low-latency literal matches. In what workflow can Khoj offer acceptable latency or superior utility as a drop-in replacement for isearch? Is there an example of how you might use it to navigate a document?

110 3y ago
That's (almost) exactly what khoj search provides a search-as-you-type experience but with a natural language (instead of keyword) search interface. My workflow looks like: 1. Search with Khoj search[1]: `C-c s s` <search-query> RET 2. Use speed key to jump to relevant entry[2]: with `n n o 2` [1]: `C-c s` is bound to `khoj` transient menu [2] https://orgmode.org/manual/Speed-Keys.html https://orgmode.org/manual/Speed-Keys.html
- Blackthorn 3y ago
  Can you elaborate on that a little bit? When you search like that, what do you type in to find stuff in a programming language source file? I'd like to better understand this workflow, it seems interesting and I might be missing out.
  - 110 3y ago
    Firstly, my previous statement needs quite a lot of clarification: Khoj works more like an incremental, natural language version of org-agenda-search (or projectile) rather than isearch. You've to configure which files it should index first. You can then use natural language to search those files with a search-as-you-type experience. This is not the same as isearch that just searches the current file for keyword matches. Second, khoj doesn't index source code and the default search models don't work well with code files. But yeah someone should implement a natural language isearch as a standalone tool (as suggested by parent comment). It'd be super-useful

bozhark 3y ago

I’m not a software dev. Is there a way to have this bot read from a discord and google drive?

syntaxing 3y ago
gpt4all itself (the library on the backend for this) has a similar program [1]. You just need to put everything into a folder. This should be straight forward for google drive. Harder for discord though but I’m sure theres a bot online that can do the extraction. [1] https://gpt4all.io/index.html https://gpt4all.io/index.html

RomanHauksson 3y ago

Awesome work, I've been looking for something like this. Any plans to support Logseq in the future?

110 3y ago
Yes, we hope to get to it soon! This has been an ask on our Github since a while[1] [1]: https://github.com/khoj-ai/khoj/issues/141 https://github.com/khoj-ai/khoj/issues/141

Ilnsk 3y ago

[flagged]

110 3y ago
2.5 years! We're kind of slow :P
tudorw 3y ago
hi, you seem keen to share something neat you took less than 10 minutes to implement, I'd love to see that?

jigneshdarji91 3y ago

This would be even great if available as a Spotlight Search replacement (with some additional features that Spotlight supports).

110 3y ago
Yeah, this would be ideal for Mac users. Just need to look into what is required and how much work it is
tough 3y ago
Should be easy to plug it in with a Raycast.app or Alfred.app plugin.
- 110 3y ago
  Khoj exposes a local API. Hopefully that makes it easy to integrate with Raycast, Alfred (or even Spotlight)?

agg23 3y ago

Just a heads up, your landing page on your website doesn't seem to mention Llama/the offline usecase at all, only online via OpenAI. ---- What model size/particular fine-tuning are you using, and how have you observed it to perform for the usecase? I've only started playing with Llama 2 at 7B and 13B sizes, and I feel they're awfully RAM heavy for consumer machines, though I'm really excited by this possibility. How is the search implemented? Is it just an embedding and vector DB, plus some additional metadata filtering (the date commands)?

rapnie 3y ago
> Just a heads up, your landing page on your website doesn't seem to mention Llama/the offline usecase at all, only online via OpenAI. I am sufficiently uneducated on the ins and outs of AI integrations to always wonder if projects like this one can be used in local-only mode, i.e. when self-hosted ensuring me that never any of my personal information is sent to a remote service. So it would be very helpful to very explicitly give me that assurance of privacy, if that's the case.
- 110 3y ago
  Yes, this seems to be a common concern. We're trying to see how best to crisply address it. But yes, Khoj can be used even with your internet turned off
110 3y ago
Thanks for the pointer, yeah the website content has gone stale. I'll try update it by end of day Khoj is using the Llama 7B, 4bit quantized, GGML by TheBloke. It's actually the first offline chat model that gives coherent answers to user queries given notes as context. And it's interestingly more conversational than GPT3.5+, which is much more formal
- moneywoes 3y ago
  Is a vector db used?
- M4v3R 3y ago

coder543 3y ago

This seems like a cool project. It would be awesome if it could also index a directory of PDFs, and if it could do OCR on those PDFs to support indexing scanned documents. Probably outside of the scope of the project for now, but just the other day I was just thinking how nice it would be to have a tool like this.

samstave 3y ago
Ive wanted a crawler on my machine for auto-categorizing and organizing, tagging and moving ALL my files around based on all my machines - so the ability to crawl PDFs, downloads, screenshots, pictures, etc and give me a logical tree of the org of the files - and allow me to modify it by saying "add all PDF related to [subject] here and the organize by source/author etc... and then move all my screenshots, ordered by date here etc... I've wanted a "COMPUTER.", uh... I say "COMPUTER!", 'sir, you have to use the keyboard', ah a Keyboard, how quaint.... forever.
- 110 3y ago
  That.would.be.awesome! Khoj isn't their yet, but that actually shouldn't be too far away if you give it a voice interface and terminal access. Of course, having it be stable enough to not `rm -rf /` soon after is definitely not part of the warranty
  - samstave 3y ago
    Copy first, move and delete after confirmation from user on big moves. - So aside from doing something such as this - whats the size of your current usecase - is it being able to search and org a large orge of devs through files and directories/repos/whatever? What if you had a global glob of data/code/files - and as an enterprise you can just give each new emp an employee role, and based on their job and purported skills, just throw them at the monolith, yet the AI will tailor the info they get out of the glob. "User10267A - you are chartered with working on projects XYZ, based on your skillset - here are a few assessments to see which sections of the codebase best suit you." [does work] "OK .U267A, here is your workspace with all auths to the libs and such youll need to work on [PROJECT FSPEZ]"

ramesh31 3y ago

Something I've noticed playing around with Llama 7b/13b on my Macbook is that it clearly points out just how little RAM 16GB really is these days. I've had a lot of trouble running both inference and a web UI together locally when browser tabs take up 5GB alone. Hopefully we will see a resurgence of lightweight native UIs for these things that don't hog resources from the model.

Kwpolska 3y ago
Or hopefully we will see an end of the LLM hype. Or at least models that don’t hog so much RAM.
- ramesh31 3y ago
  >Or at least models that don’t hog so much RAM The RAM usage is kind of the point though; we're trading space for time. It's not a problem that the model is using it, it's just that with the default choice for UI being web based now, the unnecessary memory usage of browsers is actually starting to be a real pain point.
  - 110 3y ago
    1. I hear you on going back to lightweight native apps. Unfortunately the Python ecosystem is not great for this. We use pyinstaller to create the native desktop app but it's a pain to manage. 2. The web UI isn't required if you use Obsidian or Emacs. That's just a convenient, generic interface that everyone can use.
thenickdude 3y ago
The new Chrome "memory saver" feature that discards the contents of old tabs saves a lot of memory for me. Tabs get reloaded from the server if you revisit them.
sabaimran 3y ago
FWIW I've also had browser RAM consumption issues in life, but it's been mitigated by extensions like OneTab: https://chrome.google.com/webstore/detail/onetab/chphlpgkkbolifaimnlloiipkdnihall https://chrome.google.com/webstore/detail/onetab/chphlpgkkbo... For now, local LLMs take up an egregious about of RAM, totally agreed. But we trust the ecosystem is going to keep improving and growing and we'll be able to make improvements over time. They'll probably become efficient enough where we can run them on phones, which will unlock some cool scope for Khoj to integrate with on device, offline assistance.

wg0 3y ago

I have not tried it but something like this should exist. I don't think it is going to be as useable on consumer hardware as yet unless you have a good enough GPU but within couple of years (or less), we'll be there I am sure. Irrelevant opinion - The logo is beautiful, I like it and so are the colours used. Lastly, LLMA2 for such use cases, I think is capable enough that paying for ChatGPT won't be as lucrative especially when privacy is of concern. Keep it up. Good craftsmanship. :)

sabaimran 3y ago
Thanks! I do think Llama V2 is going to be a good enough replacement for ChatGPT (aka GPT3.5) for a lot of use cases.

School-Cotton 3y ago

Markdown doesn't work on HN...

IshKebab 3y ago

Interesting. The obvious question you haven't answered anywhere (as far as I can see) is what are the hardware requirements to run this locally?

110 3y ago
Ah, you're right, forgot to mention that. We use the Llama 2 7B 4 bit quantized model. The machine requirements are: Ideal: 16Gb (GPU) RAM Less Ideal: 8GB RAM and CPU
- HexDecOctBin 3y ago
  Sorry for the repetition, but do you mean 16 GB VRAM? That is a very high requirement, a RTX 4060 only has 8GB and even a RTX 4070 only ships with 12GB. Any upcoming further optimizations for reducing memory usage? PS. Nice to see an Hindi name for a software. For those who don't speak Hindi: https://en.m.wiktionary.org/wiki/%E0%A4%96%E0%A5%8B%E0%A4%9C#:~:text=%E0%A4%96%E0%A5%8B%E0%A4%9C%20%E2%80%A2%20(khoj)%20f%20(,finding%2C%20exploration https://en.m.wiktionary.org/wiki/%E0%A4%96%E0%A5%8B%E0%A4%9C...
  - intended 3y ago
    One of the reasons 4090 / 3090s are expensive these days. It’s an issue with the models, not with Khoj.
- danparsonson 3y ago
  So just to clarify, is that: Ideal is running the model on a GPU (any brand? Nvidia, AMD, etc.?) with 16GB of GPU RAM, less ideal is running it on the CPU, for which it needs 8GB system RAM? Presumably it will occupy all that memory while it's running? What about if I have a GPU with 8GB?
  - 110 3y ago
    Khoj uses Llama 2 7B, 4bit quantized. So it just needs 3.5Gb of RAM (GPU or System) [1]. Khoj and your other apps need more RAM themselves, so practically 8GB of System or GPU RAM should suffice. Khoj has been tested with CUDA and Metal capable GPUs. So Nvidia and Mac M1+ GPUs should work. I'm think it'll work with AMD GPUs out of the box too but let me know if it doesn't for you? I can look into what needs to be done to get that to work. [1]: The calculation is [params] * [bytes] GB RAM, so 7 * 0.5 = 3.5Gb

calnayak 3y ago

How does one access this from a web browser?

sabaimran 3y ago
We have a cloud product you can sign up for, but it's more limited in what data sources it supports. It currently only works for Notion and Github indexing. If you're interested in that, send me a dm on Discord - https://discord.gg/BDgyabRM6e https://discord.gg/BDgyabRM6e But that would allow you to access Khoj from the web.

overnight5349 3y ago

Could this do something like take in the contents of my web history for the day and summarize notes on what I've been researching? This is getting very close to my ideal of a personal AI. It's only gonna be a few more years until I can have a digital brain filled with everything I know. I can't wait

usehackernews 3y ago
Interesting, this is the exact question that came to mind for me. This would address a pain point for me. Does anyone have recommendations for a tool that does it? Or, anyone want to build it together?
110 3y ago
That would be pretty awesome. Building a daily web history summarizer as a browser extension shouldn't be too much work. I bet there's something like that already out there. Having something that indexes all your digital travels and makes it easily digestible will be gold. Hopefully Khoj can become that :)
- reitanqild 3y ago
  > I bet there's something like that already out there. There was. It was called Google Desktop Search, it was awesome, and it was axed. That said, today I wouldn't use it anyway as both I and Google have changed a lot.
  - 110 3y ago
    Oh yeah, I used to use Google Desktop too. It was awesome for it's time
porcc 3y ago
https://www.rewind.ai https://www.rewind.ai
- aliasxneo 3y ago

ubertaco 3y ago

Hey, I saw Khoj hit HN a few weeks ago and get slaughtered because the messaging didn't match the product. You've come a good way in both directions: the messaging is clearer about current state vs aspirations, and you've made good progress towards the aspirational parts. Really glad to see the warm reception you're getting now. Nice job, y'all.

sabaimran 3y ago
Hey ubertaco! I remember you. Appreciate the well-wishes. The landing page still needs some tweaking. It's kind of hard keeping what you're building in sync with what you're aspiring for, but we're definitely working towards it.

yberreby 3y ago

Cool project. I tried it last time this got posted, but it was still a bit buggy. Giving it another shot - I'm mainly interested in the local chat. Could you elaborate on the incremental search feature? How did you implement it? Don't you need to re-encode the full query through a SBERT or such as each token is written (perhaps with debouncing)? Also, having an easily-extended data connector interface would be awesome, to connect to custom data sources.

110 3y ago
Buggy for setup? We've done some improvements and have desktop apps (in beta) too now to simplify this. Feel free to report any issues on the khoj github. I can have a look. Yes, we don't do optimizations on the query encoding yet. So SBERT just re-encodes the whole query every time. It gets results in <100ms which is good enough for incremental search. I did create a plugin system, so that a data plugin just has to convert the source data into a standardized intermeditate jsonl format. But this hasn't been documented or extensively tested yet.

mavsman 3y ago

Really cool to see this! Local is the real future of AI. I got really excited about this and fired it up on my petite little M2 Macbook Air only for it to grind it to a halt. Think the old days when you had a virus on your PC and you'd move the mouse then wait 45 seconds to see the cursor move. It honestly made me feel nostalgic. I guess I have to taper performance expectations with this Air, though this is the first time it's happened.

quickthrower2 3y ago
Just wait 10 years when computers have a dedicated AI-PU and you don't have to worry about freezing anything up to talk to your bot.
- kridsdale1 3y ago
  The M1 line of Macs does have that unit.
- joaogui1 3y ago
  I wonder if edge tpus (sold here https://coral.ai/products/ https://coral.ai/products/) could help, though I guess the community would have to optimize for them instead of for standard hardware
jmorgan 3y ago
How much memory do you have in your Macbook? The 7B models seem to work well with at least 16GB of unified memory, but I’ve seen Macs with 8GB really struggle.
- mavsman 3y ago
  Indeed it's just a poor little 8GB of RAM.

throw03172019 3y ago

Feedback for landing page: use a fixed height container for the example prompts. Without it, it causes jumping while scrolling down the page making other sections hard to read. iOS Safari

110 3y ago
Thanks for the feedback! Someone else mentioned this issue the other day as well. I'll fix this issue on the landing page soon

thangngoc89 3y ago

I’m in search of a new Macbook Mx. what is the requirements for running these model locally without breaking the bank? Would 32GB be enough?

110 3y ago
You do not need to break the bank to use Khoj for local chat, 16Gb RAM should be good enough
- DoctorOetker 3y ago
  How slow would that be on an old non-Apple laptop, but also 16Gb RAM? lscpu output: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 36 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Vendor ID: GenuineIntel Model name: Intel(R) Core(TM) i7-3630QM CPU @ 2.40GHz CPU family: 6 Model: 58 Thread(s) per core: 2 Core(s) per socket: 4 Socket(s): 1 Stepping: 9 CPU(s) scaling MHz: 35% CPU max MHz: 3400.0000 CPU min MHz: 1200.0000 BogoMIPS: 4791.90 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cp uid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm cpuid_fault epb pti ssbd ibrs ibpb stibp tpr _shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts md_clear flush_l1d L1d: 128 KiB (4 instances) L1i: 128 KiB (4 instances) L2: 1 MiB (4 instances) L3: 6 MiB (1 instance) NUMA: NUMA node(s): 1 NUMA node0 CPU(s): 0-7
  - 110 3y ago
    It should work but not sure how fast. Best way to find out would be to try it. We also have some fixes and perf improvements we plan to release later today in version 0.10.1

IAmNotACellist 3y ago

What's the posthog telemetry used for? Why is there nothing on it in the docs? Why no clear way to opt out?

Kerbonut 3y ago
It’s pretty easy to remove which is what I ended up doing. The project works remarkably well otherwise.
sabaimran 3y ago
Thanks for pointing that out! We use it for understanding usage -- like determining whether people are using markdown or org or more. Everything is collected entirely anonymized, and no identifiable information is ever sent to the telemetry server. To opt-out, you set the `should-log-telemetry` value in `khoj.yml` to false. Updated the docs to include these instructions and what we collect -- https://docs.khoj.dev/#/telemetry https://docs.khoj.dev/#/telemetry.

wanderingmind 3y ago

Two comments 1. If you want better adoption especially among corporations, GPL-3 wont cut it. Maybe think of some business friendly licenses (MIT etc) 2. I understand the excitement about llm's. But how about making something more accessible to people with regular machines and not state of art. I use rip-grep-all (rga) along with fzf [1] that can search all files including pdfs in a specific folders. However, I would like a GUI tool to (a) search across multiple folders, (b) provide priority of results across folders, filetypes and (c) store search histories where I can do a meta-search. This is sufficient for 95% of my usecases to search locally and I don't need LLM. If khoj can enable such search as default without LLM that will be a gamechanger for many people without a heavy compute machine or who dont want to use OpenAI. [1] https://github.com/phiresky/ripgrep-all/wiki/fzf-Integration https://github.com/phiresky/ripgrep-all/wiki/fzf-Integration

pvh 3y ago
Just a note to suggest that giving away your hard work to those who will profit from it in the hope that they will remember you later seems like a pretty dubious exchange. Have a look at how that worked out for the folks who built node and its libraries versus the ones who maintained control of their work (like npm).
- quickthrower2 3y ago
  What happened there. Surely the people who built node (or the people building the most popular fork at least) get to define what the default package manager is etc. and get some BATNA against the likes of a 3rd party package manager profiting from their thing. I don't know what the node/npm relationship story is though.
Zuiii 3y ago
If corporations have no issue with using restrictive proprietary licenses, they should not have any issues with the GPL.
trenchgun 3y ago
That seems like a pretty trivial thing to implement. Why not do it yourself?