3 ms·
So with 128K context window, if you actually input 100K it would cost you: Input: $0.01 per 1K tokens * 100 = $1.00 $1.00 per query? Given that each query us
by raylad 3y ago
So with 128K context window, if you actually input 100K it would cost you:
Input: $0.01 per 1K tokens * 100 = $1.00
$1.00 per query?
Given that each query uses the entire context window, the session would start at $1 for the first query and go up from there? Or do I have it wrong?
- minimaxir 3y agoIt would be $1 for each individual API call, if you were continuing the conversation based on the same 100K input. ChatGPT is stateless.
- raylad 3y agoRight, so that adds up very fast.
- Der_Einzige 3y agoThis is a sad fact, and one which they should have implemented a fix for. We know medium term memory works. Sentence transformers and everyone playing with pooled embeddings knows what it is because they're using it. I should be able to map my previous history to a smaller number of tokens using embedding pooling to give a notion of a lossy "medium term" memory independent of RAG.
- 0xDEF 3y agoIf it truly is GPT-4+ with a 128K context window it's still absolutely worth the high price. That is literally 300 pages. However if they are cheating like everyone else who has promised gigantic context windows then we are better off with RAG and a vector database.