mingtianzhang

10 ms·

1. 61.▲
   by mingtianzhang 1y ago

   1. One-sample detection is impossible. These detection methods work at the distributional level—more like a two-sample test in statistics—which means you need to collect a large amount of generated text from the same model to make the test
2. 62.▲
   by mingtianzhang 1y ago

   Yeah, I guess another point is that with ToC, metadata representation is transparent—people know what information is lost. On the other hand, vector-based representation is a black box. Explainability and transparency are also important con
3. 63.▲
   by mingtianzhang 1y ago

   It is also worth mentioning that compression and generative AI are two sides of the same coin. I highly recommend the book "Information Theory, Inference, and Learning Algorithms" by David MacKay, which explores these deep connect
4. 64.▲
   by mingtianzhang 1y ago

   A follow-up question is: what is the lossless way to represent knowledge? That would mean reading all the knowledge at once, which is the most accurate but also the least efficient method. Therefore, for different applications, we need to f
5. 65.▲
   by mingtianzhang 1y ago

   We are always looking for representations that can capture the meaning of information. However, most representations that compress information for retrieval are lossy. For example, embeddings are a form of lossy compression. Similar to the
6. 66.▲
   by mingtianzhang 1y ago

   It would be interesting to add more lossless compression stuff, which has a close connection to generative AI. This PhD thesis gives a very good introduction: https://arxiv.org/abs/2104.10544
7. 67.▲
   by mingtianzhang 1y ago

   Our PageIndex for HTML will be open-sourced next week, we are actually working on that!
8. 68.▲
   by mingtianzhang 1y ago

   LLM-only RAG for small corpora looks super interesting!
9. 69.▲
   by mingtianzhang 1y ago

   I just realized that the whole Hacker News discussion is formalized as a tree, and I am using my eyes to tree search through the tree to retrieve ideas from the insightful comments.
10. 70.▲
    by mingtianzhang 1y ago

    You could also try PageIndex OCR, the first long-context OCR model. Most current OCR tools process each page independently, which causes them to lose the document’s structure and produce markdown with incorrect heading levels. PageIndex OCR
11. 71.▲
    by mingtianzhang 1y ago

    Yeah, thanks for pointing out the OCR! We also found that for complex PDFs, you first need to use OCR to convert them into Markdown and then run PageIndex. However, most OCR tools process each page independently, which causes them to lose t
12. 72.▲
    by mingtianzhang 1y ago

    Thanks, that’s a great point! That’s why we use the tree structure, which can search layer by layer without putting the whole tree into the context (to compromise the summary quality). We’ll update with more examples and experiments on this
13. 73.▲
    by mingtianzhang 1y ago

    In this approach, the documents need to be pre-processed once to generate a tree structure, which is slower than the current vector-based method. However, during retrieval, this approach only requires conditioning on the context for the LLM
14. 74.▲
    by mingtianzhang 1y ago

    But different people may have different ways. For example, I use command+f in macbook.
15. 75.▲
    by mingtianzhang 1y ago

    A good thing about tree representation compared to a 'list' representation is that you can search hierarchically, layer by layer, in a large tree. For example, AlphaGo performs search in a large tree. Since the scale of retrieval
16. 76.▲
    by mingtianzhang 1y ago

    Strongly agree! It is basically the Mone-Carlo tree search method used in Alpha Go! This is also mentioned in one of their toturials: PageIndex/blob/main/tutorials/doc-search/semantics.md. I believe it will make the
17. 77.▲
    by mingtianzhang 1y ago

    I think it only needs to generate the tree once before retrieval, and it doesn’t require any external model at query time. The indexing may take some time upfront, but retrieval is then very fast and cost-free.
18. 78.▲
    by mingtianzhang 1y ago

    In their gitHub repo’s readme, they show a benchmark on FinanceBench and found that PageIndex-based retrieval significantly outperforms vector-based methods. I’ve noticed that in domain-specific documents, where all the text has similar “se
19. 79.▲
    by mingtianzhang 1y ago

    A good thing about tree representation compared to a 'list' representation is that you can search hierarchically, layer by layer, in a large tree. For example, AlphaGo performs search in a large tree. Since the scale of retrieval
20. 80.▲
    by mingtianzhang 1y ago

    A good thing about tree representation compared to a 'list' representation is that you can search hierarchically, layer by layer, in a large tree. For example, AlphaGo performs search in a large tree. Since the scale of retrieval
21. 81.▲
    by mingtianzhang 1y ago

    Yeah, they share a similar intuition. I found that the difference is that PageIndex is more of a learning-free approach, more like how a human would do retrieval?
22. 82.▲
    by mingtianzhang 1y ago

    Strongly agree, I also found txtai is super interesting! Thank you for your open-source effort!
23. 83.▲
    Vectorless RAG with a Table of Contents(docs.pageindex.ai)

    2 points by mingtianzhang 1y ago|1 comments
24. 84.▲
    by mingtianzhang 1y ago

    We wrote a short tutorial on vectorless RAG with a Table of Contents (ToC). In this tutorial, we use PageIndex to create an LLM-friendly ToC for retrieval instead of using chunking/vectors. Since a ToC is the human way of doing retriev
25. 85.▲
    by mingtianzhang 1y ago

    Hey, we use PageIndex to generate "Table of Contents" to do retreival without Vector DBs. Github version can be found: https://github.com/VectifyAI/PageIndex/blob/main/cookbook/pa... Any f
26. 86.▲
    by mingtianzhang 1y ago

    Very interesting idea! I also replaced vector databases with "Table of Content" for retrieval: https://colab.research.google.com/github/VectifyAI/PageIndex...
27. 87.▲
    by mingtianzhang 1y ago

    Thanks, I like the name lol.
28. 88.▲
    by mingtianzhang 1y ago

    Thanks, that’s a good point. Yeah, it makes retrieval look like function calling or tool selection, which I guess makes the idea more generic and better suited to current AI systems like MCP. For the ToC generation quality, you can try our
29. 89.▲
    by mingtianzhang 1y ago

    Yeah, that’s a good point. I’m starting to like this term. It would also be great to read the paper you mentioned—could you share it with me?
30. 90.▲
    by mingtianzhang 1y ago

    But keyword search is at least a white-box method, it gives you what you want. But semantic similarity is an approximate search and unexplainable.