simtj

26 ms·

1. 1.▲
   Show HN: Chat with multiple sites at once. Lite and normal crawl modes(twitter.com)

   2 points by simtj 4y ago|1 comments
2. 2.▲
   by simtj 4y ago

   OpenAI python package has a built in distances_from_embeddings function, that can be used to calculate cosine similarity or distance. javascript also has similar libs to do so. From OpenAI Docs "Which distance function should I use? We
3. 3.▲
   by simtj 4y ago

   Locally stored embeds does not get sent to the open ai call. After having all the embeddings stored locally, You can begin asking it questions. Upon asking a question, the question itself is sent to Open Ai for embedding, gets a response,
4. 4.▲
   by simtj 4y ago

   there's no upper limit in theory. App cuts the pdf in chunks and batches ~ 8000 tokens on every call for embedding from open ai, and stores the vector values locally. It takes quite abit of time to finish embedding 100k words, thats th
5. 5.▲
   by simtj 4y ago

   By using embeddings, relevant docs are clustered and then sent as context to query for a response
6. 6.▲
   Show HN: Chat with multiple (100k words) PDFs privately(twitter.com)

   5 points by simtj 4y ago|10 comments