5 ms·
OpenAI and Microsoft Azure to deprecate GPT-4 32K
- antman 2y agoSo much for code migrations! Although most examples are about summarization or needle in a haystack search, applications whose output should be the size of input are probably more important although less advertised. Curious if that is a business decision or a technical decision aka if optimizations for cheap and fast 128k gpt4o work only for small outputs
- xena 2y agoOne of the decent parts about open weights models is that you can't have this happen to you. You get to keep access to any model you want. This is essential for continuity of products.
- MuffinFlavored 2y agowhat is chatgpt as a product more than just “really good weights”?
- BiteCode_dev 2y agoProbably a lot that we don't know about, because ChatGPT has better reasoning skills than most other systems out there, and we know reasoning is not really about the weights.
- Slyfox33 2y agoChatgpt can't reason about anything.
- astromaniak 2y agoit can pretend, which often looks indistinguishable. so that you can't say if it's real or not.
- xena 2y agoReally good weights that you don't have to run yourself
- TeMPOraL 2y agoIt's just really good weights - significantly better weights than anyone else has.
- MuffinFlavored 2y agowhy has nobody else been able to catchup/come close?
- astromaniak 2y agogood weights are the key, but it has python interpreter and bing search. and likely more in development.
- astromaniak 2y agojust wonder, do you know any opens source LLM? that can be reproduced?
- azeemba 2y agoI am having trouble understanding what the complaint is here. The docs still mention bigger models with 128k tokens and smaller models with 8k tokens. It seems reasonable to optimize for big and small use cases differently? I don't see how we are being "robbed".
- xena 2y agoThe main limit is that you can have 128k tokens of input, but only 4k tokens of output per run. gpt4-32k lets you have up to 32k tokens of output per run. Some applications need that much output. Especially for token dense things like code and JSON.
- seeknotfind 2y agoSo it's a price concern? Because you could run for 4k output 8 times to get 32K? Or does the RLHF stuff prevent you from feeding the output back in as more input and still get a decent result? The underlying transformers shouldn't care because they'll be doing that already effectively.
- xena 2y agoI'd say it's less a price concern and more a consistency of output concern. It doesn't make much sense to continue incomplete JSON like that I don't think. I need to do some more research.
- peab 2y agoyou can just feed that output into another call, to have the next call continue it, since you have more than 28k extra context. The output per token is faster anyways right, so speed isn't an issue. It's just slightly more dev work (really only a couple lines of code)
- DelightOne 2y agoHow do you know it will have the same state of mind? And how much does that cost.
- kmeisthax 2y agoI'm confused here. What's an "output context"? My assumption was that the context window was shared across input and (prior) output. You put everything into the model at once, and then at the end the first unused context vector becomes a vector you can decode for a single token of output. With multiple token output meaning you repeatedly inference, decode, sample, append, and repeat until you sample an end token. Is this just a limit of OpenAI APIs or something I'm forgetting?
- asabla 2y agoThis isn't unique for OpenAI models. A lot of the open source ones has similar limitations.
- xena 2y agoIt's the number of tokens the model can output in one pass. There's subtle differences between running it multiple times to get a bigger output and running it once to get a bigger output. These are things that only really show up when you integrate these models into production code.
- ComplexSystems 2y agoNo. Newer models have 128K of input tokens, but only 4096 output tokens.
- arugulum 2y agoThe long story short is you are technically correct but in practice things are a little different. There are 2 factors to consider here: 1. Model Capability You are right that mechanically, input and output tokens in a standard decoder Transformer are "the same". A 32K context should mean you can have 1 input tokens and 32K output tokens (you actually get 1 bonus token), or 32K input tokens and 1 output token, However, if you feed an LM "too much" of its own input (read: have too long an output length), it starts to go off the rails, empirically. The word "too much" is doing some work here: it's a balance of both (1) LLM labs having data that covers that many output tokens in an example and (2) LLMs labs having empirical tests to have confidence that the model won't reasonably go off the rails within some output limit. (Note, this isn't pretraining but the instruction tuning/RLHF after, so you don't just get examples for free) In short, labs will often train a model targeting an output context length, and put out an offering based on that. 2. Infrastructure While mathematically having the model read external input and its own output are the same, the infrastructure is wildly different. This is one of the first things you learn when deploying these models: you basically have a different stack for "encoding" and "decoding" (using those terms loosely. This is after all still a decoder only model). This means you need to set max lengths for both encoding and decoding separately. So, after a long time of optimizing both the implementation and length hyperparameters (or just winging it), the lab will decide "we have a good implementation for up to 31K input and 1k output" and then go from there. If they wanted to change that, there's a bunch of infrastructure work involved. And because of the economies of batching, you want many inputs to have as close to the same lengths as possible, so you want to offer fewer configurations (some of this bucketing may be performed hidden from the user). Anyway, this is why it may become uneconomical to offer a model at a given length configuration (input or output) after some time.
- JoeCortopassi 2y agoA lot of people here haven't integrated GPT into a customer facing production system, and it shows gpt-4, gpt-4-turbo, and gpt-4o are not the same models. They are mostly close enough when you have a human in the loop, and loose constraints. But if you are building systems off of the (already fragile) prompt based output, you will have to go through a very manual process of tuning your prompts to get the same/similar output out of the new model. It will break in weird ways that makes you feel like you are trying to nail Jello to a tree There are software tools/services that help with this, and a ton more that merely promise to, but most of the tooling around LLMs these days gives the illusion of a reliable tool rather than results of one. It's the early days of the gold rush still, and every one wants to be seen as one of the first
- tmpz22 2y agoMaybe we shouldn't be selling products built on such a shaky foundation? Like Health Insurance products for example. [2]: https://insurtechdigital.com/articles/chatgpt-the-risks-and-rewards-for-insurers https://insurtechdigital.com/articles/chatgpt-the-risks-and-... --- please disregard [1] it was a terrible initial source I pulled of Google [1]: https://medium.com/artivatic/use-of-chatgpt-4-in-health-insurance-12548f3ae2c5 https://medium.com/artivatic/use-of-chatgpt-4-in-health-insu...
- bcrl 2y agoMinimum Viable Products are pretty much by definition built on shaky foundations. At least with software written by humans the failure modes are somewhat bounded by the architecture of the system as opposed to the who-knows-what-the-model-will-hallucinate of AI.
- nerdjon 2y agoI think that is the key problem, a traditional MVP is a mostly known entity. It may be missing some features, some bugs, etc. But it is an MVP not because it was necessarily rushed out the door (I mean... it was, but differently) but because it has some rough edges and is likely missing major features. Where what it seems we are getting with a lot of these companies shoving AI into something and calling it a product, is an MVP that is an MVP due to an unknown and untested nature.
- mmastrac 2y agoLooks like they are just cleaning house of lesser-used models? This came via mail last week. Back in June 2023 and November 2023, we announced the following models will be deprecated on June 13th, 2024: gpt-3.5-turbo-0301 gpt-3.5-turbo-0613 gpt-3.5-turbo-16k-0613 We noticed that your organization recently used at least one of these models. To help minimize any disruption, we are extending your access to these models for an additional 3 month grace period until September 13th, 2024. After this date, these models will be fully decommissioned.
- nicce 2y agoProbably more expensive as well.
- aftbit 2y agoAh I see, taking a page from the Google playbook and aggressively culling less popular products. I wonder how many sales Google's reputation for capricious culling has cost them.
- Terretta 2y agoArguably it's rather more like taking a page from versioned builds, meaning not supporting old builds indefinitely, through a process of notice with grace period till deprecation.