I think today 512 tokens is a minimum. It's not just the query (if you're running a chatbot, which many of us are not). It's the entire context window. It's no

I think today 512 tokens is a minimum. It's not just the query (if you're running a chatbot, which many of us are not). It's the entire context window. It's not uncommon to have a system prompt that is > 512 tokens alone. I would like to see benchmarks for 512, 1024, 4096 and 8192 token inputs.