2 ms·
The hardest part in RAQ is document parsing. If you only consider text then it should be ok, but once you start having tables, tables going multiple pages, char
by Oras 11mo ago
The hardest part in RAQ is document parsing. If you only consider text then it should be ok, but once you start having tables, tables going multiple pages, charts, ignore TOC when available, footnotes … etc, that part becomes really hard and accuracy suffers to get the context regardless of what chunking do you use.
There are some patterns to help such as RAPTOR where you make ingestion content aware and instead of just ingesting content, you start using LLMs to question and summarise the content and save that to the vector database.
But reality is, having one size fits all for RAQ is not an easy task.
- Royce-CMR 11mo agoSuper noob in vector embeddings: I never considered that tables would be a complexifier. (beyond defining in a parseable format for ingestion). Do vector databases do better with long grouped text vs table formats?
- Oras 11mo agoThe issue is the ingestion (extracting the right data in the right format). This is mainly an issue in PDFs and sometimes when there are tables added as images in Docx too. You need a mix of text and OCR extraction to get the data correctly first before start chunking and adding embeddings