4 ms·
Try looking into vector databases to solve that problem. You can chunk up a book and embed those partitions into a vector database. Then you can take a query a
by 101011 3y ago
Try looking into vector databases to solve that problem.
You can chunk up a book and embed those partitions into a vector database. Then you can take a query and fuzzy match the most relevant documents in your vector database, then feed it back to open AI to resolve an answer.
It's brilliant. Postgres has an extension to support indexing the vectors, and there are some other open source and turnkey solutions in the market as well.
- weinzierl 3y agoBrilliant, but not the same. I think both approaches have their place and are not mutually exclusive.
- 101011 3y agoThey are not the same, but your initial problem of throw a whole book at it and have OpenAI give you an answer is a demo that is solvable in 20 lines of langchain code when leveraging a vector DB. Demo here: https://www.youtube.com/watch?v=h0DHDp1FbmQ https://www.youtube.com/watch?v=h0DHDp1FbmQ (github code is linked within as well)