4 ms·
For understanding: What would be a suitable input length in your oppinion? And why isnt this a good one: Are real-life queries shorter? Or longer? If i count
by ta12653421 2y ago
For understanding:
What would be a suitable input length in your oppinion?
And why isnt this a good one: Are real-life queries shorter? Or longer?
If i count one word as a token, then in my case most of the queries are less than 128 words.
- spacecadet 2y agoIn most cases thats not enough
- Gasp0de 2y agoIncluding the initialization prompt and your history if you have one? I use ChatGPT for a very simple task, to map chat messages to one of 5 supported function calls, and the function definitions alone already take up 200 tokens I think
- stefs 2y agoIt's not just the current prompt, but the whole conversation, if possible. Or, if you want the AI to summarise an article, the article has to fit in. If I understood that correctly, context length is something like session storage or short term memory. If it's too small the AI starts to forget what it's talking about.
- qeternity 2y agoI think today 512 tokens is a minimum. It's not just the query (if you're running a chatbot, which many of us are not). It's the entire context window. It's not uncommon to have a system prompt that is > 512 tokens alone. I would like to see benchmarks for 512, 1024, 4096 and 8192 token inputs.
- rfoo 2y agoIMO the relevant benchmark for now is a mixed stream of requests with 50 (20%), 500 (50%), 2000 (10%) and 50k (20%) input tokens, ignore EOS and decode until you get around 300 output tokens.
- SuchAnonMuchWow 2y agoI'm really interested, do you have a source for those percentages ? I tried to look for some service provider to publish this kind of metrics, but haven't found any.
- rfoo 2y agoSorry, I can't. My employer doesn't publish this kind of metrics, either. What I posted was definitely just some very rough number off my brain.