4 ms·
Tokenization is a probable issue here, the longer the context the longer the initial processing with llama I think. Possible tokenizer is not optimized.
by ericlewis 3y ago
Tokenization is a probable issue here, the longer the context the longer the initial processing with llama I think. Possible tokenizer is not optimized.
- brrrrrm 3y agoThat’s very unlikely, tokenization is really simple and usually quite fast (scales with input size). Unexpected slowness with a larger context window might point to a non-existent or unoptimized KV cache.