4 ms·
token counting is done by crate that I'm using. I agree that not all LLMs use the same tokenizer but they are mostly similar. Chunking is useful because in cha
by mohsen1 2y ago
token counting is done by crate that I'm using. I agree that not all LLMs use the same tokenizer but they are mostly similar.
Chunking is useful because in chat mode you can feed more than context max size if you feed in multiple USER messages
LLMs pay more attention to the last part of conversation/message. This is why sorting is very important. Your last sentence in a very long prompt is much more important the first.
Use case: I use this to run an "AI Loop" with Deepseek to fix bugs or implement features. The loop steers the LLM by not letting it go stray in various rabbit holes. Every prompt reiterates what the objective is. By loop I mean: Serialize repo, run test, feed test failure and repo to LLM, get a diff, apply the diff and repeat until the objective is achieved.
- CGamesPlay 2y agoGot it, thanks. > in chat mode you can feed more than context max size if you feed in multiple USER messages Just so you know, this is false. You might be using a system that automatically deletes or summarizes older messages, which would make you feel like that, and would also indicate why you feel that the sorting is so important (It is important! But possibly not critically important). For future work, you might be interested in seeing how tools like Aider do their "repo serializing" (they call it a repomap), which tries to be more intelligent by only including "important lines" (like function definitions but not bodies).
- kruxigt 2y ago[dead]