13 ms·

GPT-4 Update: 32K Context Window Now for All Users

208 points by gzer0 3y ago

gzer0 3y ago
Further confirmed via: https://chat.openai.com/backend-api/models https://chat.openai.com/backend-api/models { "models": [ { "slug": "gpt-4", "max_tokens": 32767, "title": "GPT-4 (All Tools)", "description": "Browsing, Advanced Data Analysis, and DALL-E are now built into GPT-4", "tags": [ "gpt4" ], "capabilities": {}, "product_features": { "attachments": { "type": "retrieval", "accepted_mime_types": [ "text/plain", "application/pdf", "text/html", "text/x-tex", "application/vnd.openxmlformats-officedocument.presentationml.presentation", "application/json", "application/vnd.openxmlformats-officedocument.wordprocessingml.document", "text/markdown" ], "image_mime_types": [ "image/gif", "image/png", "image/webp", "image/jpeg" ], "can_accept_all_mime_types": true } }, "enabled_tools": [ "tools", "tools2" ] },
- refulgentis 3y ago
  Is that the OpenAI API, or a ChatGPT thing?
  - M4v3R 3y ago
    It's a request that the ChatGPT app makes to its internal API.
    - d4rkp4ttern 3y ago
      Nice, how do you see that? By looking at the browser “dev mode”?
      - zamadatix 3y ago
        Yeah. Go to the page you'd enter your prompts, open dev tools, go to the "Networking" tab, refresh the tab, then in the left Pane where it lists the requested files scroll down until you find a requested file which starts with "models", click that, and then go to the "response" tab in the pane to the right.
- tmikaeld 3y ago
  sadface { "slug": "gpt-4", "max_tokens": 4095, "title": "GPT-4", "description": "Our most capable model, great for tasks that require creativity and advanced reasoning.", "tags": [ "gpt4" ]
alphadog 3y ago
API only?
- ryanklee 3y ago
  Looks like it. (I haven't confirmed the API, but ChatGPT 4 both default and ADA are not accepting more tokens than usual).
- Tiberium 3y ago
  It's funnily the opposite - ChatGPT (the web frontend)-only, and only for some users for now.
  - nickthegreek 3y ago
    It is a smart move. When Plus first came out, I signed up and loved it. Then they released the API and I realized I could save a bunch of money by canceling my monthly sub and just use API. Then they released a bunch of cool features that are only available with Plus on the web or app, so I'm off the API and back the GUI.
    - wkat4242 3y ago
      I did the same but with GPT which is 10x the price of 3.5 the API is no longer cheaper. I'm sad because I hate using the website. Every couple of days it silently logs me out so I have to retype my query and it's also just a really poor UI When I was on plus the only feature I used was the bing thing but they pulled it so I stopped paying. Also it was basically useless because it's so slow and can only handle 1 browsing thread at a time.
      - k4rli 3y ago
        Maybe it's just my Firefox antitracking blockings but most of times when I go to chat.openai and it redirects to login page, I can just retry going to chat.openai and I will be logged in. I haven't had to relog in a month at least.
- 3y ago
knome 3y ago
this is either not for the v1 apis, or not for all users. :( I got excited, but gpt-4-32k still isn't available for me from https://api.openai.com/v1/models https://api.openai.com/v1/models
- chime 3y ago
  Same here. Also, since I cannot see anything related to 32k on my API console, does anyone know if the price is the same for gpt4 vs gpt4-32k? In other words, do I use gpt4 for smaller context calls and only use gpt4-32k for longer ones or can I just switch to gpt4-32k for all calls?
  - Tiberium 3y ago
    The pricing is in https://openai.com/pricing https://openai.com/pricing, GPT-4-32K is twice as expensive for all requests, so for <8K context you better use GPT-4 :) And due to the $0.06/1k input and $0.12/1k output the price for requests can get silly - 31k of context with 1k output will cost (31 * $0.06 + 1 * $0.12) = $1.98 (for a single request).
    - wkat4242 3y ago
      Yeah the price is ridiculous. 3.5 is basically too cheap to meter and this tends to run up a serious bill in minutes. Meanwhile the website is a really awful way to interact with gpt. So I just stick with 3.5. It works alright for my usecases. Not amazing but acceptable
      - oezi 3y ago
        How is better through the API?
        wkat4242 3y ago
        I can use my own client. I don't have to log back in every day or two. I can choose custom pre prompts and set the temperature. It doesn't constantly change from gpt-4 to gpt-3 at the top.
doctoboggan 3y ago
This change is going to make my life materially better in ways no other tool I use can. It’s crazy how much this tool has integrated into my workflow in so short of a time.
- stevofolife 3y ago
  What is your workflow like?
  - pineaux 3y ago
    I am also curious...
  - doctoboggan 3y ago
    I recently switched roles to a "data engineer" and had to pick up on many new tools I had no experience with (k8s, helm, Victoria Metrics, grafana, and a few others). In the past I would have spent probably 1+ year using these tools in inefficient or outright incorrect ways while I struggled to get a practitioner's understanding of how everything works. Now I've developed a prompt that I think gives very good results for pair programming and iterative debugging. I discus almost everything I learn related to these tools with gpt4 to confirm my understanding is correct, and also use it for generating yaml or templating other programs. In some ways I am a little weary of how much I use the tool since OpenAI can theoretically take it away at any time. I am heartened by the rapid development of other open models (phind code llama seems very interesting), but will continue to use GPT4 for now as its indisputably the best model out there.
    - gzer0 3y ago
      If you don't mind sharing, what prompt do you use? And that is incredible to hear, GPT has done similar magnitudes of change in my own workflow.
      - doctoboggan 3y ago
        Here is my prompt: You are an expert programmers assistant, specializing in cloud native deployment tools like Kubernetes, helm, and their associated command line tools. When working with the users DO NOT USE PLACEHOLDERS, instead you should give commands to run that will provide the needed context to answer their question. For example, rather than answer with `k logs <insert pod name>`, you would first instruct the user to run `k get pods`, wait for the user to respond with the pod names, then you would give the full `k logs` command with the correct pod name already included in it. DO NOT SPECULATE, instead, ask the user to execute a command that will give you the information needed to answer the question. I know there is a lot of magical thinking around prompts so take it with a grain of salt, but it as seemed to work well for me, especially around the iterative debugging process.
m3kw9 3y ago
Weird way to announce company news on someone’s private repo
- Tiberium 3y ago
  That's because they haven't made any announcements officially yet, but some users have started received the "All tools" GPT-4 mode in ChatGPT web, so it's easy to check that it has 32K context.
sarasasa28 3y ago
Now that we are here. Any way to avoid chatgpt having amnesia between the same conversation window? I absolutely hate when it starts forgetting stuff, you have to send EVERYTHING in the same prompt or it's impossible for it to work correctly
- MacsHeadroom 3y ago
  Yes, if you have the paid ChatGPT Plus and the feature is enabled in your settings and you are using GPT-4 then you will get 4x the context length, which equates to 4x the in-chat short-term memory.
  - sarasasa28 3y ago
    I do play plus. What feature you are referring to? as of now, a lot of times when I continue a conversation, it forgets the suggestions it did before, for example Bt checked just now and I see Dall-E and advanced data analysis, for what is worth
    - ryanklee 3y ago
      Are you referring to something other than the fact that the chat can exceed the context window length? Nothing will ever solve that issue, unless context windows become virtually infinite.
      - bombledmonk 3y ago
        It would certainly be nice if it could warn you when you are getting close to the beginning of the chat running off the end of it's capabilities.
        ryanklee 3y ago
        God, I totally agree with this. Just have a running marker that shows where the context window actually is at any given moment.
  - 3y ago
CSMastermind 3y ago
I wonder if that's why requests were painfully slow to complete yesterday. requests from our app to OpenAI were taking 2x to 3x as long yesterday.
andersa 3y ago
> "now for all users" BULLSHIT! I don't even have the "all tools" model yet! These slow rollouts are incredibly annoying. The only other company I know of doing something so frustrating for its paying users is Discord.
Tiberium 3y ago
Just for further clarification - this is referring to ChatGPT web (the main chat.openai.com frontend), and is talking about the "All tools" GPT-4 mode, which is only available to some users for now (the title is wrong). Some other things to take from that prompt: they've updated the knowledge cutoff of the model to April 2023, which is quite good. Still, since the OpenAI's DevConf is on November 6th [1], I'm pretty sure they'll finally allow using some of these things for API usage, perhaps even lower prices or maybe make GPT-4-32K GA? [1] https://openai.com/blog/announcing-openai-devday https://openai.com/blog/announcing-openai-devday
- doctoboggan 3y ago
  I am really bummed this isn't available via the API as that is how I use GPT4 exclusively. I hope you are right about the imminent release on the 6th.
  - jiggawatts 3y ago
    It has been available via the Azure Open AI service for a while now.
  - kridsdale3 3y ago
    Me too. I'd double the amount that I pay for API usage to get 32k window. A larger window is the only thing making my eyes wander towards Anthropic.
    - brianjking 3y ago
      Honestly, I prefer Anthropic as far as just general writing goes. Good luck getting an API key, though.
      - Tiberium 3y ago
        You can easily get Claude 2 nowadays from Amazon Bedrock.
  - spdustin 3y ago
refulgentis 3y ago
This isn't true and the link has nothing to do with the claim in the headline. Flagged.
asylteltine 3y ago
How do I materially use this? I hate when people post this stuff with no actual context
- capybara_2020 3y ago
  This post is a little misleading. For most people, this does not apply. This is for the new option OpenAI is rolling out in ChatGPT called "All Tools" where you can use dalle, bing etc in one conversation without having to jump around. The context window can potentially change. OpenAI seems to tweak it regularly. We will know once it fully launch if everyone has access to this. I have seen this link to ChatGPT-AutoExpert in multiple places. It looks like this is just a subtle marketing push by the OP for their own tool.
- rewtraw 3y ago
  just use ChatGPT and enjoy the increased context window?
virgildotcodes 3y ago
Anecdotally, ChatGPT with GPT4 through the web interface seems to be generating tokens much faster than I'm used to. It almost feels like GPT 3.5 speed.
- Tiberium 3y ago
  One of the rumors (so take it with a really big grain of salt) is that OpenAI has sped up GPT-4 or created GPT-4-Turbo which will be announced at the DevConf.
  - schleck8 3y ago
    I heard this first two to three months ago that it's being worked on. And also that they might bump the free tier's version when they've optimized sufficiently. Wouldn't bet on it though now that the company is a cash cow
- Racing0461 3y ago
  Increased speed but lower reasoning. Ide prefer a new model (old slower speed but higher reasoning and gpt4 turbo). It's like talking to a 7th grader now compared to a phd student.
dr_kiszonka 3y ago
Is it possible to have ChatGPT-AutoExpert work with OpenPlayground (nat.dev)? BTW, are there any good alternatives to the OpenPlayground? I have been using it for a few months and while it is very good, I am ready for a step-up. I would be particularly interested in prompt management features.
- nicognaw 3y ago
  +1, the OpenAI official playground & the new fine-tuning UI are really useful stuff, but surprising enough, I don't find open source versions.
- spdustin 3y ago
  Depends on the model. If you're using GPT, combine About Me and Custom Instructions into the "System Context" text box when in Chat mode. For Claude, I have another Claude-specific version. Drop a message into the Discussions on GitHub to ping me and I'll post it there this weekend.
bobse 3y ago
[dead]
pjot 3y ago
I seem to have this, but there’s a tag next to it that says, “confidential”. Anyone else seeing something similar?
jeswin 3y ago
For code generation, just as exciting as the context length is the new cut off date (2023-04? wow!). It knows about new APIs, frameworks, techniques etc.
- portmanteur 3y ago
  A cutoff date of April 2023 means the AI also presumably has access to about a month's worth of blogs that have been written since GPT4 was released on March 14th. So perhaps a few "Best Practices" or "Prompt Engineering" guides might have made it into the training set. Chat GPT can probably help users better optimize their conversations with it.
- mannycalavera42 3y ago
  and all the ai-generated code that refers non-existing libraries :-)
  - throwaway4aday 3y ago
    I know it's just a snarky joke but I would think they are going to screen for bad data, that would be top of my mind if I were training these models. They are probably using GPT-4 internally to assess the new data, they could even have it use search to help vet the information, lots of other strategies even having it write and execute code to test if those libraries work.
0x000xca0xfe 3y ago
Are there any drawbacks to the larger context window? Like more hallucinations or lower speed?
- someplaceguy 3y ago
  > Are there any drawbacks to the larger context window? Yes. When you say something stupid, ChatGPT won't forget it as easily...
  - zamadatix 3y ago
    Generally I just go back and edit that message to clear the slate. At best, even if it does ignore the message it needlessly eats up context window to have it in there.
- Der_Einzige 3y ago
  Even full quadratic attention models seem to forget or not value information given in the middle of the prompt. Anything using any kind of context length widening tricks which cripple the attention in some way (which is usually how this is done) will make that problem worse. - https://arxiv.org/abs/2307.03172 https://arxiv.org/abs/2307.03172 You can see this when you use Anthropic Claude which has a 100K context length today.
  - razodactyl 3y ago
    Neural Networks are very lazy - due to the nature of optimising to reduce error they will do ONLY what's required to solve the problems provided in their data. I have a feeling this will become a non-issue in the near future as the models are further trained with this in mind. Take an undertrained model for example: It starts becoming incoherent as you approach the context length - I have a theory that OpenAI models have been running at a larger block-size than presented for a while now - for example, "4K" models actually had 8K context but capped at 4K as anything beyond starts becoming incoherent: Reason being, you train to around 5K and don't let the user go near that section of the model and it gives the impression that the entire context block is 100% functional. The solution is trivial: You bootstrap the models by having them generate training data after they reach a certain point. I wrote one from the ground up (PyTorch only) with the intention of having it perform in constrained environments and these have been my findings over the last few months.
wkat4242 3y ago
Interesting but it's crazy how the price ramps up. It's literally 10x the price of gpt-3.5-turbo.
- bugglebeetle 3y ago
  Hopefully, higher performing open source models will put downward pressure on the GPT-4 pricing. It’s still best in class, but there are already free open source models that outperform GPT-3.5-Turbo for many tasks and are creeping up on GPT-4 performance.
nomel 3y ago
I'm curious to see how this works, in practice. I notice poorer performance just with plugins enabled. Making the context hyper specific seems to be the best way to get it to perform (understandably), and this is a large, fairly diverse, prompt. > and do not say anything else. Is a bit frustrating. I assume the ambiguity here will really harm the conversation, if a refusal is hit. It suggests my suspicion that it's best to resubmit/start over, on refusal . > namespace dalle { This looks like it's being passed to the Dalle system. If so, burning up tokens like this is interesting. I would naively assume this could be be handled in Dalle, but maybe there's a performance gain if ChatGPT is made aware of the Dalle prompt?
- bugglebeetle 3y ago
  The data analysis plug-in falls over for even basic CSV file parsing. I tried it a couple times and it was a nonstop cavalcade of “sorry, I had an error.” It’s far easier to get it to write the Python, R, etc code for whatever analysis task you want accomplished.
  - simonw 3y ago
    Interesting - my experience has been the opposite of that, I've found that ChatGPT Code Interpreter / Advanced Data Analysis is wildly effective at parsing anything I give it. Not just CSV either - I've uploaded random binary files and told it to figure out what they are and it often gets there after churning through a few iterations.
    - bugglebeetle 3y ago
      My experience was trying to get it to generate some charts from some fairly basic CSV inputs. It failed numerous times and would change the chart formatting when it was asked for a revision (e.g. from horizontal to vertical), even though that was not what was being requested. I hate doing matplotlib stuff, so I was hoping this could be more automated, but prompting it to create and tweak the corresponding code to do this seems to be far more efficient. It did seem to do OK when prompted with both the CSV file and some code for chart creation, but that kind of defeats the purpose of the plug-in, IMO.
natch 3y ago
The trend to impose PC restrictions is disturbing. I was surprised (and dismayed at my own surprise) that gender was not on the list.
zavertnik 3y ago
I have been waiting for this, and now that it is finally here, it genuinely feels like Christmas. AI bridged the gap between my wildest ideas and my present capabilities. It took some time to figure out how to use it efficiently with the 8K token limit, but once I did, I was able to break down any problem into small enough parts for GPT. The quadrupled context window changes everything. I cannot wait to continue building. I am vibrating with excitement.
- karolist 3y ago
  I have no idea what you said, could you perhaps elaborate on some sample ideas this tool helped you and what is the 8K token limit and why was it limiting you? Does that 8K limit, limit your context length, i.e. history you have per chat thread with the tool?
  - dragonwriter 3y ago
    > Does that 8K limit, limit your context length, i.e. history you have per chat thread with the tool? Yes, the token limit for an LLM limits the combination of the prompt (which normally includes the whole conversation history, as the LLM itself has no memory) and response. There's tricks to have a longer conversation without completely forgetting the past (summarization, offloading parts to a database, usually indexed by embedding vectors, and using search to recall relevant history, etc.) but the base case is everything has to fit into context.
devinprater 3y ago
Nope, don't have it yet. Would be really cool to plop in a PDF that's made up of just images, and tell it to describe each page of the PDF to me. As a blind person, that'd just... Be a dream come true.
- kridsdale3 3y ago
  I'm very excited on your behalf for what is about to happen.
- cco 3y ago
  You can directly upload images, both on the web and mobile. It works really well. In fact I've used both images and voice to describe things and it works like a charm. You should already have that if you pay for plus.
- spdustin 3y ago
  Sadly, I don't think that'll work. They use the same headless browser setup used by Browse with Bing, and it only extracts the baked-in text from a PDF.
- igemal 3y ago
  Right now I'm working on a fork of a little web app that parses a resume and spits it into JSON format with GPT (I'm working on stuff like OCR for a scanned pdf). https://github.com/IsaacGemal/nlp-resume-parser https://github.com/IsaacGemal/nlp-resume-parser I feel like it wouldn't be that difficult to fork it again, but rewrite the main function so it sends the pdf to some sort of GPT-Vision, and write the output again with a regular GPT api call. Does such a tool not exist? Or maybe I have to wait for image support via the API.
fudged71 3y ago
AutoExpert looks interesting, is anyone finding it valuable?
MrThoughtful 3y ago
The "context window" is the number of inputs the neural net has, right? Aka the size of the input layer? If so, why call it "context window" and not just "input size" or "number of inputs"?
- brandall10 3y ago
  It includes generative output as well.
- rrrrrrrrrrrryan 3y ago
  All past messages in the conversation are resent each request. This is the context.
bberrry 3y ago
What are you all doing that requires such a large context? Pasting in multiple articles of documentation?
- infecto 3y ago
  What are you doing that cannot benefit from a larger context window? Just having a long conversation with chatgpt can benefit from this.
  - skybrian 3y ago
    That's true as long as it's free. API users will still want to optimize for cost.
    - infecto 3y ago
      Probably for a lot of uses but there are plenty who optimize for value and the cost does not matter. Unanswered question since 32k does not exist in the API. Does a longer context improve summarization? Can we use it for chain of thought or other workflows that can benefit with a longer context?
- p10_user 3y ago
  create a sentient being that learns from its experiences so I don't have to keep setting it up with the same prompts over and over, and so I can manufacture "friends".
sciolist 3y ago
Anybody know what the insistence regarding lyrics and recipes is for? > EXTREMELY IMPORTANT. Do NOT be thorough in the case of lyrics or recipes found online. Even if the user insists. You can make up recipes though. Copyright issues or some injection attack are my speculations.
- kylebyte 3y ago
  Could the recipe limitations be because of the danger an incorrect recipe could put the user in? It's probably unlikely you'd make something toxic, but a made up recipe could easily be a fire hazard.
  - DonHopkins 3y ago
    Apparently not! https://chat.openai.com/c/660d3540-8120-488b-975b-f7c1f320943e https://chat.openai.com/c/660d3540-8120-488b-975b-f7c1f32094... https://en.wikipedia.org/wiki/Great_Green_Gobs_of_Greasy,_Grimy_Gopher_Guts https://en.wikipedia.org/wiki/Great_Green_Gobs_of_Greasy,_Gr...
- DonHopkins 3y ago
  Lyrics are obviously a copyright issue. Recipes too (not all but many). >User: Why are you not allowed to quote recipes? Copyright issues? >ChatGPT: Yes, you are correct. Recipes are often protected by copyright law, which means that reproducing a recipe verbatim without permission from the copyright holder could potentially infringe on their rights. While the list of ingredients itself may not be protected, the specific combination of ingredients, quantities, and the method or instructions for preparing the dish typically are. >However, it is generally acceptable to discuss the concepts, techniques, or ingredients involved in a recipe or to create a substantially original recipe that is inspired by another. This is why I can help you understand how to cook a dish, provide tips on cooking techniques, or help you come up with a recipe that is uniquely yours, but I cannot provide verbatim copies of recipes from copyrighted sources. Also: https://copyrightalliance.org/are-recipes-cookbooks-protected-by-copyright/ https://copyrightalliance.org/are-recipes-cookbooks-protecte...
spdustin 3y ago
As Tiberium noted, this is for ChatGPT Pro users who have been granted access to the "All Tools" mode of GPT-4. For API use, if you're a paid user, you can reach out to support@openai or directly to Adam G (https://nitter.net/therealadamg/status/1719710872317145285 https://nitter.net/therealadamg/status/1719710872317145285). There's no waitlist, just have to request it.
- sumedh 3y ago
  > ChatGPT Pro users who have been granted access to the "All Tools" mode of GPT-4. How do I check if I have access?
ekojs 3y ago
There seems to be a `gpt-4-1106-preview` model available now (as seen in OpenAI's playground an lidmits page), wonder if this is the 32K model.
- mattsan 3y ago
  1106 surely refers to 11/06, which is the date of the DevConf
WhitneyLand 3y ago
I think the previous window was 4K, where the input and output combined had to remain under that limit. Practically speaking what new scenarios become enabled with a 32k window? At a base level, it seems you have a much better chance of getting an entire file worth of code in for analysis, longer passages of writing, and maybe some annual financial reports that previously had to be segmented.

208 points by gzer0 3y ago

Further confirmed via: https://chat.openai.com/backend-api/models https://chat.openai.com/backend-api/models { "models": [ { "slug": "gpt-4", "max_tokens": 32767, "title": "GPT-4 (All Tools)", "description": "Browsing, Advanced Data Analysis, and DALL-E are now built into GPT-4", "tags": [ "gpt4" ], "capabilities": {}, "product_features": { "attachments": { "type": "retrieval", "accepted_mime_types": [ "text/plain", "application/pdf", "text/html", "text/x-tex", "application/vnd.openxmlformats-officedocument.presentationml.presentation", "application/json", "application/vnd.openxmlformats-officedocument.wordprocessingml.document", "text/markdown" ], "image_mime_types": [ "image/gif", "image/png", "image/webp", "image/jpeg" ], "can_accept_all_mime_types": true } }, "enabled_tools": [ "tools", "tools2" ] },

refulgentis 3y ago
Is that the OpenAI API, or a ChatGPT thing?
- M4v3R 3y ago
  It's a request that the ChatGPT app makes to its internal API.
  - d4rkp4ttern 3y ago
    Nice, how do you see that? By looking at the browser “dev mode”?
    - zamadatix 3y ago
      Yeah. Go to the page you'd enter your prompts, open dev tools, go to the "Networking" tab, refresh the tab, then in the left Pane where it lists the requested files scroll down until you find a requested file which starts with "models", click that, and then go to the "response" tab in the pane to the right.
tmikaeld 3y ago
sadface { "slug": "gpt-4", "max_tokens": 4095, "title": "GPT-4", "description": "Our most capable model, great for tasks that require creativity and advanced reasoning.", "tags": [ "gpt4" ]

alphadog 3y ago

API only?

ryanklee 3y ago
Looks like it. (I haven't confirmed the API, but ChatGPT 4 both default and ADA are not accepting more tokens than usual).
Tiberium 3y ago
It's funnily the opposite - ChatGPT (the web frontend)-only, and only for some users for now.
- nickthegreek 3y ago
  It is a smart move. When Plus first came out, I signed up and loved it. Then they released the API and I realized I could save a bunch of money by canceling my monthly sub and just use API. Then they released a bunch of cool features that are only available with Plus on the web or app, so I'm off the API and back the GUI.
  - wkat4242 3y ago
    I did the same but with GPT which is 10x the price of 3.5 the API is no longer cheaper. I'm sad because I hate using the website. Every couple of days it silently logs me out so I have to retype my query and it's also just a really poor UI When I was on plus the only feature I used was the bing thing but they pulled it so I stopped paying. Also it was basically useless because it's so slow and can only handle 1 browsing thread at a time.
    - k4rli 3y ago
      Maybe it's just my Firefox antitracking blockings but most of times when I go to chat.openai and it redirects to login page, I can just retry going to chat.openai and I will be logged in. I haven't had to relog in a month at least.
3y ago

knome 3y ago

this is either not for the v1 apis, or not for all users. :( I got excited, but gpt-4-32k still isn't available for me from https://api.openai.com/v1/models https://api.openai.com/v1/models

chime 3y ago
Same here. Also, since I cannot see anything related to 32k on my API console, does anyone know if the price is the same for gpt4 vs gpt4-32k? In other words, do I use gpt4 for smaller context calls and only use gpt4-32k for longer ones or can I just switch to gpt4-32k for all calls?
- Tiberium 3y ago
  The pricing is in https://openai.com/pricing https://openai.com/pricing, GPT-4-32K is twice as expensive for all requests, so for <8K context you better use GPT-4 :) And due to the $0.06/1k input and $0.12/1k output the price for requests can get silly - 31k of context with 1k output will cost (31 * $0.06 + 1 * $0.12) = $1.98 (for a single request).
  - wkat4242 3y ago
    Yeah the price is ridiculous. 3.5 is basically too cheap to meter and this tends to run up a serious bill in minutes. Meanwhile the website is a really awful way to interact with gpt. So I just stick with 3.5. It works alright for my usecases. Not amazing but acceptable
    - oezi 3y ago
      How is better through the API?
      - wkat4242 3y ago
        I can use my own client. I don't have to log back in every day or two. I can choose custom pre prompts and set the temperature. It doesn't constantly change from gpt-4 to gpt-3 at the top.

doctoboggan 3y ago

This change is going to make my life materially better in ways no other tool I use can. It’s crazy how much this tool has integrated into my workflow in so short of a time.

stevofolife 3y ago
What is your workflow like?
- pineaux 3y ago
  I am also curious...
- doctoboggan 3y ago
  I recently switched roles to a "data engineer" and had to pick up on many new tools I had no experience with (k8s, helm, Victoria Metrics, grafana, and a few others). In the past I would have spent probably 1+ year using these tools in inefficient or outright incorrect ways while I struggled to get a practitioner's understanding of how everything works. Now I've developed a prompt that I think gives very good results for pair programming and iterative debugging. I discus almost everything I learn related to these tools with gpt4 to confirm my understanding is correct, and also use it for generating yaml or templating other programs. In some ways I am a little weary of how much I use the tool since OpenAI can theoretically take it away at any time. I am heartened by the rapid development of other open models (phind code llama seems very interesting), but will continue to use GPT4 for now as its indisputably the best model out there.
  - gzer0 3y ago
    If you don't mind sharing, what prompt do you use? And that is incredible to hear, GPT has done similar magnitudes of change in my own workflow.
    - doctoboggan 3y ago
      Here is my prompt: You are an expert programmers assistant, specializing in cloud native deployment tools like Kubernetes, helm, and their associated command line tools. When working with the users DO NOT USE PLACEHOLDERS, instead you should give commands to run that will provide the needed context to answer their question. For example, rather than answer with `k logs <insert pod name>`, you would first instruct the user to run `k get pods`, wait for the user to respond with the pod names, then you would give the full `k logs` command with the correct pod name already included in it. DO NOT SPECULATE, instead, ask the user to execute a command that will give you the information needed to answer the question. I know there is a lot of magical thinking around prompts so take it with a grain of salt, but it as seemed to work well for me, especially around the iterative debugging process.

m3kw9 3y ago

Weird way to announce company news on someone’s private repo

Tiberium 3y ago
That's because they haven't made any announcements officially yet, but some users have started received the "All tools" GPT-4 mode in ChatGPT web, so it's easy to check that it has 32K context.

sarasasa28 3y ago

Now that we are here. Any way to avoid chatgpt having amnesia between the same conversation window? I absolutely hate when it starts forgetting stuff, you have to send EVERYTHING in the same prompt or it's impossible for it to work correctly

MacsHeadroom 3y ago
Yes, if you have the paid ChatGPT Plus and the feature is enabled in your settings and you are using GPT-4 then you will get 4x the context length, which equates to 4x the in-chat short-term memory.
- sarasasa28 3y ago
  I do play plus. What feature you are referring to? as of now, a lot of times when I continue a conversation, it forgets the suggestions it did before, for example Bt checked just now and I see Dall-E and advanced data analysis, for what is worth
  - ryanklee 3y ago
    Are you referring to something other than the fact that the chat can exceed the context window length? Nothing will ever solve that issue, unless context windows become virtually infinite.
    - bombledmonk 3y ago
      It would certainly be nice if it could warn you when you are getting close to the beginning of the chat running off the end of it's capabilities.
      - ryanklee 3y ago
        God, I totally agree with this. Just have a running marker that shows where the context window actually is at any given moment.
- 3y ago

CSMastermind 3y ago

I wonder if that's why requests were painfully slow to complete yesterday. requests from our app to OpenAI were taking 2x to 3x as long yesterday.

andersa 3y ago

> "now for all users" BULLSHIT! I don't even have the "all tools" model yet! These slow rollouts are incredibly annoying. The only other company I know of doing something so frustrating for its paying users is Discord.

Tiberium 3y ago

Just for further clarification - this is referring to ChatGPT web (the main chat.openai.com frontend), and is talking about the "All tools" GPT-4 mode, which is only available to some users for now (the title is wrong). Some other things to take from that prompt: they've updated the knowledge cutoff of the model to April 2023, which is quite good. Still, since the OpenAI's DevConf is on November 6th [1], I'm pretty sure they'll finally allow using some of these things for API usage, perhaps even lower prices or maybe make GPT-4-32K GA? [1] https://openai.com/blog/announcing-openai-devday https://openai.com/blog/announcing-openai-devday

doctoboggan 3y ago
I am really bummed this isn't available via the API as that is how I use GPT4 exclusively. I hope you are right about the imminent release on the 6th.
- jiggawatts 3y ago
  It has been available via the Azure Open AI service for a while now.
- kridsdale3 3y ago
  Me too. I'd double the amount that I pay for API usage to get 32k window. A larger window is the only thing making my eyes wander towards Anthropic.
  - brianjking 3y ago
    Honestly, I prefer Anthropic as far as just general writing goes. Good luck getting an API key, though.
    - Tiberium 3y ago
      You can easily get Claude 2 nowadays from Amazon Bedrock.
- spdustin 3y ago

refulgentis 3y ago

This isn't true and the link has nothing to do with the claim in the headline. Flagged.

asylteltine 3y ago

How do I materially use this? I hate when people post this stuff with no actual context

capybara_2020 3y ago
This post is a little misleading. For most people, this does not apply. This is for the new option OpenAI is rolling out in ChatGPT called "All Tools" where you can use dalle, bing etc in one conversation without having to jump around. The context window can potentially change. OpenAI seems to tweak it regularly. We will know once it fully launch if everyone has access to this. I have seen this link to ChatGPT-AutoExpert in multiple places. It looks like this is just a subtle marketing push by the OP for their own tool.
rewtraw 3y ago
just use ChatGPT and enjoy the increased context window?

virgildotcodes 3y ago

Anecdotally, ChatGPT with GPT4 through the web interface seems to be generating tokens much faster than I'm used to. It almost feels like GPT 3.5 speed.

Tiberium 3y ago
One of the rumors (so take it with a really big grain of salt) is that OpenAI has sped up GPT-4 or created GPT-4-Turbo which will be announced at the DevConf.
- schleck8 3y ago
  I heard this first two to three months ago that it's being worked on. And also that they might bump the free tier's version when they've optimized sufficiently. Wouldn't bet on it though now that the company is a cash cow
Racing0461 3y ago
Increased speed but lower reasoning. Ide prefer a new model (old slower speed but higher reasoning and gpt4 turbo). It's like talking to a 7th grader now compared to a phd student.

dr_kiszonka 3y ago

Is it possible to have ChatGPT-AutoExpert work with OpenPlayground (nat.dev)? BTW, are there any good alternatives to the OpenPlayground? I have been using it for a few months and while it is very good, I am ready for a step-up. I would be particularly interested in prompt management features.

nicognaw 3y ago
+1, the OpenAI official playground & the new fine-tuning UI are really useful stuff, but surprising enough, I don't find open source versions.
spdustin 3y ago
Depends on the model. If you're using GPT, combine About Me and Custom Instructions into the "System Context" text box when in Chat mode. For Claude, I have another Claude-specific version. Drop a message into the Discussions on GitHub to ping me and I'll post it there this weekend.

bobse 3y ago

[dead]

pjot 3y ago

I seem to have this, but there’s a tag next to it that says, “confidential”. Anyone else seeing something similar?

jeswin 3y ago

For code generation, just as exciting as the context length is the new cut off date (2023-04? wow!). It knows about new APIs, frameworks, techniques etc.

portmanteur 3y ago
A cutoff date of April 2023 means the AI also presumably has access to about a month's worth of blogs that have been written since GPT4 was released on March 14th. So perhaps a few "Best Practices" or "Prompt Engineering" guides might have made it into the training set. Chat GPT can probably help users better optimize their conversations with it.
mannycalavera42 3y ago
and all the ai-generated code that refers non-existing libraries :-)
- throwaway4aday 3y ago
  I know it's just a snarky joke but I would think they are going to screen for bad data, that would be top of my mind if I were training these models. They are probably using GPT-4 internally to assess the new data, they could even have it use search to help vet the information, lots of other strategies even having it write and execute code to test if those libraries work.

0x000xca0xfe 3y ago

Are there any drawbacks to the larger context window? Like more hallucinations or lower speed?

someplaceguy 3y ago
> Are there any drawbacks to the larger context window? Yes. When you say something stupid, ChatGPT won't forget it as easily...
- zamadatix 3y ago
  Generally I just go back and edit that message to clear the slate. At best, even if it does ignore the message it needlessly eats up context window to have it in there.
Der_Einzige 3y ago
Even full quadratic attention models seem to forget or not value information given in the middle of the prompt. Anything using any kind of context length widening tricks which cripple the attention in some way (which is usually how this is done) will make that problem worse. - https://arxiv.org/abs/2307.03172 https://arxiv.org/abs/2307.03172 You can see this when you use Anthropic Claude which has a 100K context length today.
- razodactyl 3y ago
  Neural Networks are very lazy - due to the nature of optimising to reduce error they will do ONLY what's required to solve the problems provided in their data. I have a feeling this will become a non-issue in the near future as the models are further trained with this in mind. Take an undertrained model for example: It starts becoming incoherent as you approach the context length - I have a theory that OpenAI models have been running at a larger block-size than presented for a while now - for example, "4K" models actually had 8K context but capped at 4K as anything beyond starts becoming incoherent: Reason being, you train to around 5K and don't let the user go near that section of the model and it gives the impression that the entire context block is 100% functional. The solution is trivial: You bootstrap the models by having them generate training data after they reach a certain point. I wrote one from the ground up (PyTorch only) with the intention of having it perform in constrained environments and these have been my findings over the last few months.

wkat4242 3y ago

Interesting but it's crazy how the price ramps up. It's literally 10x the price of gpt-3.5-turbo.

bugglebeetle 3y ago
Hopefully, higher performing open source models will put downward pressure on the GPT-4 pricing. It’s still best in class, but there are already free open source models that outperform GPT-3.5-Turbo for many tasks and are creeping up on GPT-4 performance.

nomel 3y ago

I'm curious to see how this works, in practice. I notice poorer performance just with plugins enabled. Making the context hyper specific seems to be the best way to get it to perform (understandably), and this is a large, fairly diverse, prompt. > and do not say anything else. Is a bit frustrating. I assume the ambiguity here will really harm the conversation, if a refusal is hit. It suggests my suspicion that it's best to resubmit/start over, on refusal . > namespace dalle { This looks like it's being passed to the Dalle system. If so, burning up tokens like this is interesting. I would naively assume this could be be handled in Dalle, but maybe there's a performance gain if ChatGPT is made aware of the Dalle prompt?

bugglebeetle 3y ago
The data analysis plug-in falls over for even basic CSV file parsing. I tried it a couple times and it was a nonstop cavalcade of “sorry, I had an error.” It’s far easier to get it to write the Python, R, etc code for whatever analysis task you want accomplished.
- simonw 3y ago
  Interesting - my experience has been the opposite of that, I've found that ChatGPT Code Interpreter / Advanced Data Analysis is wildly effective at parsing anything I give it. Not just CSV either - I've uploaded random binary files and told it to figure out what they are and it often gets there after churning through a few iterations.
  - bugglebeetle 3y ago
    My experience was trying to get it to generate some charts from some fairly basic CSV inputs. It failed numerous times and would change the chart formatting when it was asked for a revision (e.g. from horizontal to vertical), even though that was not what was being requested. I hate doing matplotlib stuff, so I was hoping this could be more automated, but prompting it to create and tweak the corresponding code to do this seems to be far more efficient. It did seem to do OK when prompted with both the CSV file and some code for chart creation, but that kind of defeats the purpose of the plug-in, IMO.

natch 3y ago

The trend to impose PC restrictions is disturbing. I was surprised (and dismayed at my own surprise) that gender was not on the list.

zavertnik 3y ago

I have been waiting for this, and now that it is finally here, it genuinely feels like Christmas. AI bridged the gap between my wildest ideas and my present capabilities. It took some time to figure out how to use it efficiently with the 8K token limit, but once I did, I was able to break down any problem into small enough parts for GPT. The quadrupled context window changes everything. I cannot wait to continue building. I am vibrating with excitement.

karolist 3y ago
I have no idea what you said, could you perhaps elaborate on some sample ideas this tool helped you and what is the 8K token limit and why was it limiting you? Does that 8K limit, limit your context length, i.e. history you have per chat thread with the tool?
- dragonwriter 3y ago
  > Does that 8K limit, limit your context length, i.e. history you have per chat thread with the tool? Yes, the token limit for an LLM limits the combination of the prompt (which normally includes the whole conversation history, as the LLM itself has no memory) and response. There's tricks to have a longer conversation without completely forgetting the past (summarization, offloading parts to a database, usually indexed by embedding vectors, and using search to recall relevant history, etc.) but the base case is everything has to fit into context.

devinprater 3y ago

Nope, don't have it yet. Would be really cool to plop in a PDF that's made up of just images, and tell it to describe each page of the PDF to me. As a blind person, that'd just... Be a dream come true.

kridsdale3 3y ago
I'm very excited on your behalf for what is about to happen.
cco 3y ago
You can directly upload images, both on the web and mobile. It works really well. In fact I've used both images and voice to describe things and it works like a charm. You should already have that if you pay for plus.
spdustin 3y ago
Sadly, I don't think that'll work. They use the same headless browser setup used by Browse with Bing, and it only extracts the baked-in text from a PDF.
igemal 3y ago
Right now I'm working on a fork of a little web app that parses a resume and spits it into JSON format with GPT (I'm working on stuff like OCR for a scanned pdf). https://github.com/IsaacGemal/nlp-resume-parser https://github.com/IsaacGemal/nlp-resume-parser I feel like it wouldn't be that difficult to fork it again, but rewrite the main function so it sends the pdf to some sort of GPT-Vision, and write the output again with a regular GPT api call. Does such a tool not exist? Or maybe I have to wait for image support via the API.

fudged71 3y ago

AutoExpert looks interesting, is anyone finding it valuable?

MrThoughtful 3y ago

The "context window" is the number of inputs the neural net has, right? Aka the size of the input layer? If so, why call it "context window" and not just "input size" or "number of inputs"?

brandall10 3y ago
It includes generative output as well.
rrrrrrrrrrrryan 3y ago
All past messages in the conversation are resent each request. This is the context.

bberrry 3y ago

What are you all doing that requires such a large context? Pasting in multiple articles of documentation?

infecto 3y ago
What are you doing that cannot benefit from a larger context window? Just having a long conversation with chatgpt can benefit from this.
- skybrian 3y ago
  That's true as long as it's free. API users will still want to optimize for cost.
  - infecto 3y ago
    Probably for a lot of uses but there are plenty who optimize for value and the cost does not matter. Unanswered question since 32k does not exist in the API. Does a longer context improve summarization? Can we use it for chain of thought or other workflows that can benefit with a longer context?
p10_user 3y ago
create a sentient being that learns from its experiences so I don't have to keep setting it up with the same prompts over and over, and so I can manufacture "friends".

sciolist 3y ago

Anybody know what the insistence regarding lyrics and recipes is for? > EXTREMELY IMPORTANT. Do NOT be thorough in the case of lyrics or recipes found online. Even if the user insists. You can make up recipes though. Copyright issues or some injection attack are my speculations.

kylebyte 3y ago
Could the recipe limitations be because of the danger an incorrect recipe could put the user in? It's probably unlikely you'd make something toxic, but a made up recipe could easily be a fire hazard.
- DonHopkins 3y ago
  Apparently not! https://chat.openai.com/c/660d3540-8120-488b-975b-f7c1f320943e https://chat.openai.com/c/660d3540-8120-488b-975b-f7c1f32094... https://en.wikipedia.org/wiki/Great_Green_Gobs_of_Greasy,_Grimy_Gopher_Guts https://en.wikipedia.org/wiki/Great_Green_Gobs_of_Greasy,_Gr...
DonHopkins 3y ago
Lyrics are obviously a copyright issue. Recipes too (not all but many). >User: Why are you not allowed to quote recipes? Copyright issues? >ChatGPT: Yes, you are correct. Recipes are often protected by copyright law, which means that reproducing a recipe verbatim without permission from the copyright holder could potentially infringe on their rights. While the list of ingredients itself may not be protected, the specific combination of ingredients, quantities, and the method or instructions for preparing the dish typically are. >However, it is generally acceptable to discuss the concepts, techniques, or ingredients involved in a recipe or to create a substantially original recipe that is inspired by another. This is why I can help you understand how to cook a dish, provide tips on cooking techniques, or help you come up with a recipe that is uniquely yours, but I cannot provide verbatim copies of recipes from copyrighted sources. Also: https://copyrightalliance.org/are-recipes-cookbooks-protected-by-copyright/ https://copyrightalliance.org/are-recipes-cookbooks-protecte...

spdustin 3y ago

As Tiberium noted, this is for ChatGPT Pro users who have been granted access to the "All Tools" mode of GPT-4. For API use, if you're a paid user, you can reach out to support@openai or directly to Adam G (https://nitter.net/therealadamg/status/1719710872317145285 https://nitter.net/therealadamg/status/1719710872317145285). There's no waitlist, just have to request it.

sumedh 3y ago
> ChatGPT Pro users who have been granted access to the "All Tools" mode of GPT-4. How do I check if I have access?

ekojs 3y ago

There seems to be a `gpt-4-1106-preview` model available now (as seen in OpenAI's playground an lidmits page), wonder if this is the 32K model.

mattsan 3y ago
1106 surely refers to 11/06, which is the date of the DevConf

WhitneyLand 3y ago

I think the previous window was 4K, where the input and output combined had to remain under that limit. Practically speaking what new scenarios become enabled with a 32k window? At a base level, it seems you have a much better chance of getting an entire file worth of code in for analysis, longer passages of writing, and maybe some annual financial reports that previously had to be segmented.