14 ms·
Why doesn't Pro include longer context windows? I'm a Plus member, and the biggest limitation I am running into by far is the maximum length of a context windo
by Someone1234 2y ago
Why doesn't Pro include longer context windows?
I'm a Plus member, and the biggest limitation I am running into by far is the maximum length of a context window. I'm having context fall out of scope throughout the conversion or not being able to give it a large document that I can then interrogate.
So if I go from paying $20/month for 32,000 tokens, to $200/month for Pro, I expect something more akin to Enterprise's 128,000 tokens or MORE. But they don't even discuss the context window AT ALL.
For anyone else out there looking to build a competitor I STRONGLY recommend you consider the context window as a major differentiator. Let me give you an example of a usage which ChatGPT just simply cannot do very well today: Dump a XML file into it, then ask it questions about that file. You can attach files to ChatGPT, but it is basically pointless because it isn't able to view the entire file at once due to, again, limited context windows.
- dudus 2y agoThe longer the context the more backtracking it needs to do. It gets exponentially more expensive. You can increase it a little, but not enough to solve the problem. Instead you need to chunk your data and store it in a vector database so you can do semantic search and include only the bits that are most relevant in the context. LLM is a cool tool. You need to build around it. OpenAI should start shipping these other components so people can build their solutions and make their money selling shovels. Instead they want end user to pay them to use the LLM without any custom tooling around. I don't think that's a winning strategy.
- tom1337 2y ago> Instead you need to chunk your data and store it in a vector database so you can do semantic search and include only the bits that are most relevant in the context. Isn't that kind of what Anthropic is offering with projects? Where you can upload information and PDF files and stuff which are then always available in the chat?
- cma 2y agoThey put all the project in the context, works much better than RAG when it fits. 200k context for their pro plan, and 500K for enterprise.
- gcr 2y agoThis isn't true. Transformer architectures generally take quadratic time wrt sequence length, not exponential. Architectural innovations like flash attention also mitigate this somewhat. Backtracking isn't involved, transformers are feedforward. Google advertises support for 128k tokens, with 2M-token sequences available to folks who pay the big bucks: https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024/ https://blog.google/technology/ai/google-gemini-next-generat...
- dartos 2y agoDuring inference time, yes, but training time does scale exponentially as backpropagation still has to happen. You can’t use fancy flash attention tricks either.
- thunderbird120 2y agoNo, additional context does not cause exponential slowdowns and you absolutely can use FlashAttention tricks during training, I'm doing it right now. Transformers are not RNNs, they are not unrolled across timesteps, the backpropagation path for a 1,000,000 context LLM is not any longer than a 100 context LLM of the same size. The only thing which is larger is the self attention calculation which is quadratic wrt compute and linear wrt memory if you use FlashAttention or similar fused self attention calculations. These calculations can be further parallelized using tricks like ring attention to distribute very large attention calculations over many nodes. This is how google trained their 10M context version of Gemini.
- dartos 2y agoI may be missing something, but I thought that each context token would result in an 3 additional parameters per context token for self attention to build its map, since each attention must calculate a value considering all existing context
- upghost 2y agoSo why are the context windows so "small", then? It would seem that if the cost was not so great, then having a larger context window would give an advantage over the competition.
- Melatonic 2y agoSeems like a good candidate for a "dumb" AI you can run locally to grab data you need and filter it down before giving to OpenAI
- solarkraft 2y ago> you need to chunk your data and store it in a vector database so you can do semantic search and include only the bits that are most relevant in the context Be aware that this tends to give bad results. Once RAG is involved you essentially only do slightly better than a traditional search, a lot of nuance gets lost.
- rahimnathwani 2y agoThis depends on the amount of context you provide, and the quality of your retrieval step.
- hackernewds 2y agoI don't know whether using exponential in the general English language usage of the word, but it does not get exponentially more expensive
- danpalmer 2y agoBecause they can't do long context windows. That's the only explanation. What you can do with a 1m token context window is quite a substantial improvement, particularly as you said for enterprise usage.
- frakt0x90 2y agoHave you considered RAG instead of using the entire document? It's more complex but would at least allow you to query the document with your API of choice.
- itissid 2y agoI've been concatenating my source code of ~3300 lines and 123979 bytes(so likely < 128K context window) into the chat to get better answers. Uploading files is hopeless in the web interface.
- fragmede 2y agowhy not use aider/similar and upload via API?
- j45 2y agoIf a $200/month pro level is successful it could open the door to a $2000/month segment, and the $20,000/month segment will appear and the segregation of getting ahead with AI will begin.
- dr_kiszonka 2y agoThis is a significant concern for me too.
- j45 2y agoIt's important to become early users of everything while AI is heavily subsidized. Over time, using open source model as well will get more done per dollar of compute and hopefully the gap will remain close.
- fragmede 2y agoQuestion is if OpenAI is actually making money at $200/month.
- j45 2y agoFor funded startups, losing less can be a form of runway and capacity especially at the numbers they are spending.
- vbezhenar 2y agoWith o1-preview and $20 subscription my queries typically were answered in 10-20 seconds. I've tried $200 subscription with some queries and got 5-10 minutes answer time. Unless the load is substantially increased and I was just waiting in queue for computing resources, I'd assume that they throw a lot more hardware for o1-pro. So it's entirely possible that $200/month is still at loss.
- johnisgood 2y agoAgreed. Where may I read about how to set up an LLM similar to that of Claude, which has the minimum length of Claude's context window, and what are the hardware requirements? I found Claude incredibly useful.
- KTibow 2y agoIn my experience OpenAI models perform worse on long contexts than Anthropic/Google's, even when using the cheaper ones.
- kranke155 2y agoClaude is clearly the superior product id say. The only reason I open Chat now is because Claude will refuse to answer questions on a variety of topics including for example medication side effects.
- visarga 2y agoWhen I tested o1 a few hours ago, it seemed like it was losing context. After I asked it to use a specific writing style, and pasting a large reference text, it forgot my demand. I reminded it, and it kept the rule for a few more messages, and after another long paste it forgot again.
- mark_l_watson 2y agoSwitch to Gemini Pro just when you need huge context size. That is what I do.
- 8n4vidtmkvmk 2y agoJust? You don't think the model is as capable when the context does fit?
- mark_l_watson 2y agoI tend to use OpenAI, Gemini, and Claude. All are excellent, but when I am not happy with results I hit all!three.
- carbocation 2y agoPro does have longer context windows, specifically 128k. Take a look at the pricing page for this information: https://openai.com/chatgpt/pricing/ https://openai.com/chatgpt/pricing/
- katamari-damacy 2y agoChatGPT and GPT4o APIs have 128K window as well. The 32K is from the days of GPT4.
- carbocation 2y agoAccording to the pricing page, 32K context is for Plus users and 128K context is for Pro users. Not disagreeing with you, just adding context for readers that while you are explaining that the 4o API has 128K window, the 4o ChatGPT agent appears to have varying context depending on account type.
- mattwallace 2y agoo1 pro failed to accept 121903 tokens input into the chat (claude took it just fine)
- carbocation 2y agoSeems like something that would be worth pinging OpenAI about because it's a pretty important claim that they are making on their pricing page! Unless it's a matter of counting tokens differently.
- nstj 2y agoThanks for this. I’m surprised they haven’t made this more obvious in their release and other documentation
- thomasahle 2y agoIt's disappointing because the o1-preview had 128k context length. At least on the API. So they nerfed it and made the original product $200/month.
- rmbyrro 2y agoGoogle models have huge contexts, but are terrible...
- arresin 2y agoAgreed. The new 1121 is better but still garbage relatively.
- WhitneyLand 2y agoWhat don’t you like about Claude? I believe the context is larger. Coincidentally I’ve been using it with xml files recently (iOS storyboard files), and it seems to do pretty well manipulating and refactoring elements as I interact with it.
- domysee 2y agoWhen talking about context windows I'm surprised no one mentions https://poe.com/ https://poe.com/. Switched over from ChatGPT about a year ago, and it's amazing. Can use all models and the full context window of them, for the same price as a ChatGPT subscription.
- EVa5I7bHFq9mnYK 2y agoPoe.com goes straight to login page, doesn't want to divulge ANY information to me before I sign up. No About Us or Product description or Pricing - nothing. Strange behavior. But seeing it more and more with modern web sites.
- nilsherzig 2y agoYou can take a look at openrouter, also a pay as you go frontend (or API "proxy") for every single API in existence
- bobnamob 2y agoI wouldn’t bother with Poe, poe2 early access costs $30 and starts on the 6th