3 ms·
Very interesting use of ChatGPT and prompt engineering. For your problem, summarizing a large document, splitting the document into smaller parts is indeed the
by ruxudev 4y ago
Very interesting use of ChatGPT and prompt engineering. For your problem, summarizing a large document, splitting the document into smaller parts is indeed the way to go.
I also had problems myself with operating on large documents. In my case, I had an insurance policy that I wanted to extract information from.
My solution: use the OpenAI API to convert the document to OpenAI's embeddings and saving those embeddings to a vector database. Then, use similarity search on the database to find chunks of the document that might be related to my query and pass only those chunks to GPT for the information extraction prompt.
I plan to create a guide on how to tackle these problems after I consolidate my findings.
- agomez314 4y agoI'm increasingly seeing these kinds of solutions for similar tasks. I wonder if we are seeing the discovery of new abstractions from using LLMs.
- a_bonobo 4y agoVery cool solution! My solution was to write a bit of code that writes a CSV, then I used a langchain-based CSV agent. Since that one calls on pandas it effectively has no token limit, but it also has no overview of the data, only what pandas tells it.
- Beefin 4y agothis is how we enable 10mb+ file ingestion and search /w http://mixpeek.com http://mixpeek.com
- monkeydust 4y agoAlso doing this, works really well. Check out James Briggs on YouTube. Excellent tutorials on how to achieve this.
- FooBarWidget 4y agoCan you refer to a specific video?
- monkeydust 4y agohttps://youtu.be/tBJ-CTKG2dM https://youtu.be/tBJ-CTKG2dM Goes into workings of retrieval augmentation with example