10 ms·
It looks like GPT-4-32k is rolling out
- fieididuxue 4y agoWill this be better than LoRA? 32k seems like a lot.
- deleted 4y ago[deleted]
- rolisz 4y agoThis is completely separate from LoRA. This is how much stuff you can give it in the prompt. You can give it now whole chapters of books to summarize for example. LoRA is for adapting the model to a certain model. It usually means you need to give it shorter prompts, but for book summarization, it wouldn't help.
- sp332 4y agoLoRA doesn’t change the context length. Also you can’t run LoRAs on GPT-4. So those are not relevant to each other.
- heliophobicdude 4y agoLoRA probably does not affect the models biggest bottle neck. The attention mechanism. Original transformer was O(n^2d) where n is the query length and d was the cardinal of all the tokens
- paulmendoza 4y agoI feel like this just killed a few small startups who were trying to offer more context. Also, I pay for ChatGPT but I have none of the new features except for GPT4. Very frustrating.
- arbuge 4y agoSame here: https://twitter.com/arbuge/status/1654288169397805057 https://twitter.com/arbuge/status/1654288169397805057 The really odd thing is that I was given GPT-4 with browsing alpha enabled - for a single session last week. As soon as I reloaded the page, it was gone. Since then the picture has reverted back to the above. Twitter has become a bit painful to read these days, with all the AI influencers posting about what GPT-4 and plugins, code interpreter etc. can do.
- MichaelZuo 4y agoConsidering that increasing context length is O(n^2), and that current 8k GPT-4 is already restricted to 25 prompts/3 hours, I think they will launch it at substantially higher pricing.
- choeger 4y agoIt will be interesting to see how far this quadratic algorithm carries in practice. Even the longest documents can only have hundreds of thousands of tokens, right?
- sebzim4500 4y agoIdeally you'd be able to put your entire codebase + documentation + jira tickets + etc. into the context. I think there is no practical limit to how many tokens would be useful for users, so the limits imposed by the model (either hard limits or just pricing) will always be a bottleneck.
- jtbayly 4y agoI'm confused by this. Would you want to just include your codebase, documentation, etc. in some last-mile training? That way you don't need the expense of including huge amounts of context in every query. It's baked in.
- virgildotcodes 4y agoDoes this purely affect the amount of tokens that can be fed in and retained in context during a session? The output from that prompt seems spectacular, so I'm wondering if there are any other differences. I just tried the same prompt with GPT-4 and the style was much more GPT-like, what I'm used to, not near the same quality as in the OP, although maybe it's just luck?
- arbuge 4y agoThe prompts in the example on that page are pretty short... hardly taking advantage of the longer context window. I'm actually not sure if longer responses can be expected with the 32k vs 8k models. Anyone from OpenAI care to comment on that?
- vorticalbox 4y agoThe token limit is for the whole conversation, system, assistant and user messages all count. Once the limit is hit it will stop, sometimes mid sentence. I have a slack bot at work with a system document which is a little over 1k tokens, meaning there is around 3k tokens left for questions and replies. Trick I am currently doing is to prune older messages to keep it under the limit.
- dmix 4y ago> Once the limit is hit it will stop, sometimes mid sentence. Oh that’s why GPT does that in a long thread. It makes sense in retrospect it’s the whole conversation not the individual messages. ChatGPT UX has much to be desired, there should be error messages that communicate this stuff better.
- heliophobicdude 4y agoCan you please elaborate on how you prune?
- vorticalbox 4y agoyou can just use the API, if you set completions to 0 it will return the token count. Then you can just remove the oldest message until it's under any number. I picked 3k to allow 1k for the reply
- Mistletoe 4y agoI love the prompt and yes it does very well at mimicking DFW. It’s kind of weirding me out in a Year of the Trial-Size Dove Bar kind of way.
- isoprophlex 4y agoThat example is really shockingly good, indeed. I'm not always convinced that GPTs can be properly artistic, most things lack soul and read like some rambling Dan Brown on amphetamine... this DFW works very well. It gave me the same vague feeling of annoyance and disgust at the "look how smart I am" linguistic obstreperousness I get when reading the real deal.
- hackernewds 4y agothe output is rather fantastic it's mind numbing. would DFW think this is a solid example of prose?
- zb3 4y agoOf course I still don't even have the basic GPT-4 after nearly two months of waiting.
- throwaway50606 4y agoYou pay for Plus and don't have it? Maybe try canceling and subscribing on another account. I got it immediately after I subscribed.
- zb3 4y agoNo, I'm talking about the API waitlist. Still, I'm using https://nat.dev/ https://nat.dev/ to access GPT-4 so I don't care that much anymore.
- gwd 4y agoBummer -- I got access after 2-3 weeks. I haven't actually even hit the 8k limit yet, and even experimenting with 32k is pretty expensive, so I'm not sure what I'd do with it.
- GordonS 4y agoI tried to checkout nat.dev, but it wants me to create an account to see anything at all. What is nat.dev please?
- zb3 4y agoThis is a paid LLM playground site with many models including GPT-4 and Claude-v1.
- itsgrimetime 4y agoI signed up for API use as soon as it was released and just got access yesterday, so they’re still rolling it out it I guess.
- phillipcarter 4y agoWhat's your usage and stated use case? I got access for my company account, but I'm pretty sure that's because we've built and shipped product using their API.
- capableweb 4y ago> A helpful rule of thumb is that one token generally corresponds to ~4 characters of text for common English text. This translates to roughly ¾ of a word (so 100 tokens ~= 75 words) - https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer 32,000 (tokens) * 4 = 128,000 (characters) > While a general guideline is one page is 500 words (single spaced) or 250 words (double spaced), this is a ballpark figure - https://wordcounter.net/words-per-page https://wordcounter.net/words-per-page Assuming (on average) one word = 5 letters, context ends up being ~50 pages (128000 / (500 * 5)). Just to put the number of "32k tokens" into somewhat estimated context.
- dmix 4y agoI wonder how many LoC that is on average. Is there an average for LoC? It’s probably based on the language…
- ethbr0 4y agoFinally, APL's day is here! https://en.m.wikipedia.org/wiki/APL_(programming_language) https://en.m.wikipedia.org/wiki/APL_(programming_language)
- wongarsu 4y agoTesting some random rust code, it's about 15-20 tokens per LoC, so about 1500-2000 LoC in a 32k context. Interestingly, using 2-space indentation, as soon as you are about 3-4 indentation levels deep you spend as many tokens on indentation as on the actual code. For example, "log::LevelFilter::Info" is 6 tokens, same as 6 consecutive spaces. There are probably a lot of easy gains here reformatting your code to use longer lines or maybe no indentation at all.
- ec109685 4y agoMake sure you are testing with the tiktokenizer: https://news.ycombinator.com/item?id=35460648 https://news.ycombinator.com/item?id=35460648
- hanoz 4y agoThe more token capacity that's added the more wasteful it seems to have to use this statelessly. Is there any avoiding this? Wonderous as this new tech is, it seems a bit much to be paying $2 a question in a conversation about a 32k token text.
- jtbayly 4y agoYeah, I can see this being useful for one-off queries, but don't they want to offer some sort of final training ("last-mile" I called it in another comment. I can't remember what the proper term is.) to companies to customize the model so it already has all the context they need baked in to every query?
- notpachet 4y agoThis is available through Azure: https://azure.microsoft.com/en-us/products/cognitive-services/openai-service/ https://azure.microsoft.com/en-us/products/cognitive-service...
- BoorishBears 4y agoAs far as I know it's not.
- sashank_1509 4y agoThey used to offer exactly this for fine tuning models. Never offered it after ChatGPT, I think the difficulty comes with fine tuning RLHF models, not obvious how to correctly do this.
- toxicFork 4y agoThe usage needs to be for high value queries. Using it on a simple conversation is not its intended purpose, that's like using a supercomputer to play pong.
- mlyle 4y agoYou can ask multiple/multipart questions.
- cubefox 4y agoI thought it was already available for a while on Microsoft Azure?
- sherbondy 4y agoYep, if you sign up via Azure OpenAI Service, you might get access sooner. Same exact API, just served directly through Azure, and likely to be maintained for longer. Direct link to the relevant sign up form fwiw: https://customervoice.microsoft.com/Pages/ResponsePage.aspx?id=v4j5cvGGr0GRqy180BHbR7en2Ais5pxKtso_Pz4b1_xUOFA5Qk1UWDRBMjg0WFhPMkIzTzhKQ1dWNyQlQCN0PWcu https://customervoice.microsoft.com/Pages/ResponsePage.aspx?... We got access a few weeks back via Azure.
- ren_engineer 4y agoI've now got access to GPT-4-0314, anybody know the difference between that and the 32k model in this post?
- mirekrusin 4y ago8k vs 32k context.
- ec109685 4y agoYours is a snapshot of the 8k token context taken on March 14th.
- m3kw9 4y agoWaiting for gpt4 turbo
- vl 4y agoWhat is turbo in this context?
- m3kw9 4y agoGpt3.5 turbo it was like 10x cheaper than a similar version. Gpt4 is like 100x more expensive then 3.5 turbo
- moffkalast 4y agoA model optimized for inference speed instead of raw accuracy.
- anticensor 4y agoGPT-4 but pruned down to GPT-3's or GPT-2's size.
- richardanaya 4y agoSame .. gpt-4 is a very harsh trade off of time for products.
- JonAtkinson 4y agoDoes anyone have any examples of promoting to feed such a large amount of tokens? For example, would you use something like “I am going to send you an entire codebase, with the filename and path, followed by the file content. Here is the first of 239 files: …”
- mysterydip 4y agoSounds like a return to the days of download managers or file splitters :)
- fragsworth 4y agoI think what's more important is the end part of your prompt, which explains what was previously described and includes a request/question.
- r3trohack3r 4y agoI already do this with the current context limits. I include a few of my relevant source files before my prompt in ChatGPT. It work’s unreasonably well. Something like the following Here is the Template class: … Here is an example component: … Here is an example Input element: … I need to create another input element that allows me to select a number from a drop down between 1/32 and 24/32 in 1/32 increments
- dested 4y agoHere's someone that passed a 23 page congressional document: https://twitter.com/SullyOmarr/status/1654576774976770048 https://twitter.com/SullyOmarr/status/1654576774976770048
- danielbln 4y agoI've had access to the 32k model for a bit and I've been using this to collect and stuff codebases into the context: https://github.com/mpoon/gpt-repository-loader https://github.com/mpoon/gpt-repository-loader It works really well, you can tell it to implement new features or mutate parts of the code and it having the entire (or a lot of) the code in its context really improves the output. The biggest caveat: shit is expensive! A full 32k token request will run you like $2, if you do dialog back and forth you can rack up quite the bill quickly. If it was 10x cheaper, I would use nothing else, having a large context window is that much of a game changer. As it stands, I _very_ carefully construct the prompt and move the conversation out of the 32k into the 8k model as fast as I can to save cost.
- ChildOfChaos 4y agoI'd just like to see GPT4 more available, even on the free chatGPT, although I wonder if that will ever fully come with ChatGPT getting so much use and GPT 3.5 being cheaper to run. Plus seems expensive to me and it is still rate limited quite a lot. I guess it's going to take further optimisation to make it worthwhile for OpenAI.
- eh9 4y agoDo the output limits change? If I give it an entire codebase, could it potentially spit out an entire codebase? I’m wondering if this is a quick way to get to $5 each round trip (send and receive)
- sva_ 4y agoAlmost $2 if you want to use the full context length (32 * $.06). Yikes.
- teaearlgraycold 4y ago:D