3 ms·

How does this work when there is a limited context window. You do some pre-chunking?

• CuriouslyC 3y ago
  Phi can ingest 2k tokens and the optimal chunk size is between 512-1024 depending on the model/application, so you just give it a big chunk and tell it to break it down into smaller chunks that are semantically related, leaving enough room for book-end sentences to enrich the context of the chunk. Then you start the next big chunk with the remnants of the previous one that the model couldn't group.
  • drittich 3y ago
    Isn't "give it a big chunk" just the same problem at a higher level? How do you handle, say, a book?
    • CuriouslyC 3y ago
      You don't need to handle a whole book, the goal is to chunk the book into chunks of the correct size, which is less than the context size of the model you're using to chunk it semantically. When you're ingesting data, you fill up the chunker model's context, and it breaks that up into smaller, self relevant chunks and a remainder. You then start from the remainder and slurp up as much additional text as you can to fill the context and repeat the process.