3 ms·
The prompts in the example on that page are pretty short... hardly taking advantage of the longer context window. I'm actually not sure if longer responses can
by arbuge 3y ago
The prompts in the example on that page are pretty short... hardly taking advantage of the longer context window.
I'm actually not sure if longer responses can be expected with the 32k vs 8k models. Anyone from OpenAI care to comment on that?
- vorticalbox 3y agoThe token limit is for the whole conversation, system, assistant and user messages all count. Once the limit is hit it will stop, sometimes mid sentence. I have a slack bot at work with a system document which is a little over 1k tokens, meaning there is around 3k tokens left for questions and replies. Trick I am currently doing is to prune older messages to keep it under the limit.
- dmix 3y ago> Once the limit is hit it will stop, sometimes mid sentence. Oh that’s why GPT does that in a long thread. It makes sense in retrospect it’s the whole conversation not the individual messages. ChatGPT UX has much to be desired, there should be error messages that communicate this stuff better.
- heliophobicdude 3y agoCan you please elaborate on how you prune?
- vorticalbox 3y agoyou can just use the API, if you set completions to 0 it will return the token count. Then you can just remove the oldest message until it's under any number. I picked 3k to allow 1k for the reply
- akiselev 3y agoYou can also use the `tiktoken` python library: import tiktoken len(tiktoken.encoding_for_model("gpt-4").encode(contents))
- mlyle 3y agoThere's the token limit-- the maximum number of tokens to respond. There's also the token context: how many words into the "past" it considers when formulating the next word. They're different things. You can generate very, very long responses with a model with a short context window; it just will have amnesia about what it said earlier-- though OpenAI often seems to restrict you / prevent you from having context scroll off in this way.