7 ms·

What is the use-case here? What is a "chunk"? It looks like it's just an arbitrary group of files, where "more important" files get put at the end. Why is that

What is the use-case here? What is a "chunk"? It looks like it's just an arbitrary group of files, where "more important" files get put at the end. Why is that useful for LLMs? Also, I see it can chunk based on token count but... what's a token? ChatGPT? Llama? Note, I understand why code context is important for LLMs. I don't understand what this chunking is or how it helps me get better code context.

mohsen1 2y ago
token counting is done by crate that I'm using. I agree that not all LLMs use the same tokenizer but they are mostly similar. Chunking is useful because in chat mode you can feed more than context max size if you feed in multiple USER messages LLMs pay more attention to the last part of conversation/message. This is why sorting is very important. Your last sentence in a very long prompt is much more important the first. Use case: I use this to run an "AI Loop" with Deepseek to fix bugs or implement features. The loop steers the LLM by not letting it go stray in various rabbit holes. Every prompt reiterates what the objective is. By loop I mean: Serialize repo, run test, feed test failure and repo to LLM, get a diff, apply the diff and repeat until the objective is achieved.
- CGamesPlay 2y ago
  Got it, thanks. > in chat mode you can feed more than context max size if you feed in multiple USER messages Just so you know, this is false. You might be using a system that automatically deletes or summarizes older messages, which would make you feel like that, and would also indicate why you feel that the sorting is so important (It is important! But possibly not critically important). For future work, you might be interested in seeing how tools like Aider do their "repo serializing" (they call it a repomap), which tries to be more intelligent by only including "important lines" (like function definitions but not bodies).
- kruxigt 2y ago
  [dead]

mohsen1 2y ago

token counting is done by crate that I'm using. I agree that not all LLMs use the same tokenizer but they are mostly similar. Chunking is useful because in chat mode you can feed more than context max size if you feed in multiple USER messages LLMs pay more attention to the last part of conversation/message. This is why sorting is very important. Your last sentence in a very long prompt is much more important the first. Use case: I use this to run an "AI Loop" with Deepseek to fix bugs or implement features. The loop steers the LLM by not letting it go stray in various rabbit holes. Every prompt reiterates what the objective is. By loop I mean: Serialize repo, run test, feed test failure and repo to LLM, get a diff, apply the diff and repeat until the objective is achieved.

CGamesPlay 2y ago
Got it, thanks. > in chat mode you can feed more than context max size if you feed in multiple USER messages Just so you know, this is false. You might be using a system that automatically deletes or summarizes older messages, which would make you feel like that, and would also indicate why you feel that the sorting is so important (It is important! But possibly not critically important). For future work, you might be interested in seeing how tools like Aider do their "repo serializing" (they call it a repomap), which tries to be more intelligent by only including "important lines" (like function definitions but not bodies).
kruxigt 2y ago
[dead]