6 ms·

Ok, since is running all private, how can I add my own private data? For example I have a 20+ years of an email archive that I'd like to be ingested.

by ccozan 3y ago

rdedev 3y ago
A simple way would be to do some form of retrieval on those emails and add those back to the original prompt
cromka 3y ago
I imagine this means you’d need to come up with own model, even if based on existing one.
- ravishi 3y ago
  And is that hard? Sorry if this is a newbie question, I'm really out of the loop on this tech. What would be required? Computing power and tagging? Or can you like improve the model without much human intervention? Can it be done incrementally with usage and user feedback? Would a single user even be able to generate enough feedback for this?
  - phillipcarter 3y ago
    Yes, this would be quite hard. Fine-tuning an LLM is no simple task. The tools and guidance around it are very new, and arguably not meant for non-ML Engineers.
    - aryamaan 3y ago
      What are some ways people to get familiar with machine learning engineering who are also working adults
influxmoment 3y ago
That would require custom training. This project only does inference
- alach11 3y ago
  No need to fine tune the model. The model could be augmented with retrieved context (as discussed in my sibling comment).
alach11 3y ago
The simplest way, as rdedev is describing, is to do Retrieval Augmented Generation (RAG) in your prompting. This would require the addition of a vector database and a text embedding model. There are many open source / local / private options for that. The steps would then be: 1. Embed your private data in chunks and store the resulting embeddings in a vector database 2. In your prompting workflow, when a user queries the chat model, embed their query using the embedding model 3. Retrieve the most similar chunks of text from your vector database based on cosine similarity 4. In the chat response, provide it the context of those chunks of text For example, if you asked "who have I discussed Ubuntu with?", it might retrieve emails that have similar content. Then the model will be able to answer informed by that context.
- squarefoot 3y ago
  Looks like this is not easy at all for a non ML expert. And probably the required computing power is still out of reach for mere mortals. I'd have a similar use case to the parent: technical books. I'd love to be able to ask where a certain topic is discussed in my pdf archive and have the AI reply with references and possibly a significant piece of the relevant articles, with images (or local links to them).
  - gzer0 3y ago
    This is already a feature of adobe pdf reader professional (called index mode). There’s also an app on macOS called “pdf search” which does quite a good job. I use it for the exact reasons you describe; I’ve got a repertoire of technical books on AWS and Azure and I reference them all the time via my local search engine via these apps.
  - siquick 3y ago
    It’s not as difficult as you think with libraries like Llamaindex and Langchain. Both have extensive examples in their documentation for almost identical use cases to the above.
  - deet 3y ago
    The computing power is definitely not out of reach of mere mortals. I'm working on software that does this for emails and common documents, generating a hybrid semantic (vector) and keyword search system over all your data, locally. The computing power we're requiring is simply what's available in any M1/M2 Mac, and the resource usage for the indexing and search is negligible. This isn't even a hard requirement, any modern PC could index all your emails and do the local hybrid search part. Running the local LM is what requires more resources, but as this project shows it's absolutely possible. Of course getting it to work *well* for certain use cases is still hard. Simply searching for close sections of papers and injecting them into the prompt as others have mentioned doesn't always provide enough context for the LM to give a good answer. Local LMs aren't great at reasoning over large amounts of data yet, but getting better every week so it's just a matter of time. (If you're curious my email is in my profile)

Ok, since is running all private, how can I add my own private data? For example I have a 20+ years of an email archive that I'd like to be ingested.

by ccozan 3y ago

Ok, since is running all private, how can I add my own private data? For example I have a 20+ years of an email archive that I'd like to be ingested.

rdedev 3y ago

A simple way would be to do some form of retrieval on those emails and add those back to the original prompt

cromka 3y ago

I imagine this means you’d need to come up with own model, even if based on existing one.

ravishi 3y ago
And is that hard? Sorry if this is a newbie question, I'm really out of the loop on this tech. What would be required? Computing power and tagging? Or can you like improve the model without much human intervention? Can it be done incrementally with usage and user feedback? Would a single user even be able to generate enough feedback for this?
- phillipcarter 3y ago
  Yes, this would be quite hard. Fine-tuning an LLM is no simple task. The tools and guidance around it are very new, and arguably not meant for non-ML Engineers.
  - aryamaan 3y ago
    What are some ways people to get familiar with machine learning engineering who are also working adults

influxmoment 3y ago

That would require custom training. This project only does inference

alach11 3y ago
No need to fine tune the model. The model could be augmented with retrieved context (as discussed in my sibling comment).

alach11 3y ago

The simplest way, as rdedev is describing, is to do Retrieval Augmented Generation (RAG) in your prompting. This would require the addition of a vector database and a text embedding model. There are many open source / local / private options for that. The steps would then be: 1. Embed your private data in chunks and store the resulting embeddings in a vector database 2. In your prompting workflow, when a user queries the chat model, embed their query using the embedding model 3. Retrieve the most similar chunks of text from your vector database based on cosine similarity 4. In the chat response, provide it the context of those chunks of text For example, if you asked "who have I discussed Ubuntu with?", it might retrieve emails that have similar content. Then the model will be able to answer informed by that context.

squarefoot 3y ago
Looks like this is not easy at all for a non ML expert. And probably the required computing power is still out of reach for mere mortals. I'd have a similar use case to the parent: technical books. I'd love to be able to ask where a certain topic is discussed in my pdf archive and have the AI reply with references and possibly a significant piece of the relevant articles, with images (or local links to them).
- gzer0 3y ago
  This is already a feature of adobe pdf reader professional (called index mode). There’s also an app on macOS called “pdf search” which does quite a good job. I use it for the exact reasons you describe; I’ve got a repertoire of technical books on AWS and Azure and I reference them all the time via my local search engine via these apps.
- siquick 3y ago
  It’s not as difficult as you think with libraries like Llamaindex and Langchain. Both have extensive examples in their documentation for almost identical use cases to the above.
- deet 3y ago
  The computing power is definitely not out of reach of mere mortals. I'm working on software that does this for emails and common documents, generating a hybrid semantic (vector) and keyword search system over all your data, locally. The computing power we're requiring is simply what's available in any M1/M2 Mac, and the resource usage for the indexing and search is negligible. This isn't even a hard requirement, any modern PC could index all your emails and do the local hybrid search part. Running the local LM is what requires more resources, but as this project shows it's absolutely possible. Of course getting it to work *well* for certain use cases is still hard. Simply searching for close sections of papers and injecting them into the prompt as others have mentioned doesn't always provide enough context for the LM to give a good answer. Local LMs aren't great at reasoning over large amounts of data yet, but getting better every week so it's just a matter of time. (If you're curious my email is in my profile)