7 ms·

Do you plan to offer content aware chunking?

• magundu 3y ago
  What is content aware chunking?
  • Art9681 3y ago
    Using content appropriate delimiters to create the chunks. If its a Python document, split it at the proper delimiters. If its an HTML document, JSON, etc. We wouldn't want to split a chunk in the middle of a function or paragraph. Or maybe we would. But that is what OP is refering to I believe.
    • chiccomagnus 3y ago
      That's a good point, and documents too needs different chunking techniques. You don't want to split a word file the same way you split an excel...
• ocolegro 3y ago
  Yes, this is on the shortlist. Do you have any preferred frameworks?
  • gardnr 3y ago
    I haven't found any frameworks that offer it. The best explanation of an implementation that can take a stream of unformatted text and map over it to determine when a topic changes is explained in this video: https://youtu.be/8OJC21T2SL4?t=1932 https://youtu.be/8OJC21T2SL4?t=1932 They compute embeddings using a window of three sentences and then compute distance to find the largest deltas to break up the text into "topics". It is computationally expensive.
    • chiccomagnus 3y ago
      check out this https://preprocess.co https://preprocess.co
    • gardnr 3y ago
      I just noticed this has been added to langchain: https://python.langchain.com/docs/modules/data_connection/document_transformers/semantic-chunker https://python.langchain.com/docs/modules/data_connection/do...