3 ms·
Yeah, but presumably the main point of compressing a prompt is not to consume your token quota. If the model has to decompress the prompt first, then it will us
by mistercow 4y ago
Yeah, but presumably the main point of compressing a prompt is not to consume your token quota. If the model has to decompress the prompt first, then it will use up just as many tokens, plus the length of the compressed version.