3 ms·
Ask HN: Way to extract relevant parts from a PDF based on a question?
Dear HN,
I am trying to do some semantic search in a given corpus of PDF documents based on a question as input. My goal is to find the relevant parts from the PDF that best answers the input question. I am interested in finding out concepts, frameworks, and methodologies that will help me with this task. If you have any pointers, I would greatly appreciate it!
- anoni2 2y agoTry: https://notebooklm.google/ https://notebooklm.google/
- madhatter999 2y agoI didn't know notebooklm had that feature embedded. Thanks for sharing!
- liampulles 2y agoThis is a key usecase for text embeddings. Essentially it is a process of converting sentences or paragraphs to vectors, where the closeness of vectors then represents a semantic similarity. So you can convert all the paragraphs in your document into vectors, convert your question into a vector, and then find the e.g. 10 closest vectors, or all that fall under a certain maximum distance, etc. You can store the embeddings in a vector database, to search across multiple documents.
- madhatter999 2y agoThank you for the answer! Wouldn't it make sense to compare the potential answers to the questions to the parts of the documents?
- liampulles 2y agoIf the potential answers are given, sure
- verdverm 2y agoLlamaIndex is my tool of choice right now https://docs.llamaindex.ai/en/stable/ https://docs.llamaindex.ai/en/stable/ https://docs.llamaindex.ai/en/stable/examples/citation/pdf_page_reference/?h=pdf https://docs.llamaindex.ai/en/stable/examples/citation/pdf_p... I'm using it with Qdrant and can get the text sections & locations that are tied to the answer & citation as well
- madhatter999 2y agoThank you for the pointer! What makes LlamaIndex different than other similar tools in your opinion?
- verdverm 2y agoThey are focused on RAG / agent systems and have lots of docs / tutorials to that effort