4 ms·

This was great! One thing that wasn't addressed that I always assume, is that a marketing name like "Opus 5" is not a single model, but many models, versions an

by ddxv 2mo ago

This was great! One thing that wasn't addressed that I always assume, is that a marketing name like "Opus 5" is not a single model, but many models, versions and gets minor updates over time. I also assumed many questions get routed to simpler models or programs to answer correctly, but it almost surprisingly didn't seem that way from the post. Anyways, great post.

tedsanders 2mo ago
I can explain how we do it at OpenAI. In the API, we keep the models fixed. There are tiny caveats like rare bug fixes or models like `chat-latest`, but this is spiritually true. Suspicions of models changing over time are either human hallucinations or bugs on our end. However, in ChatGPT, we sometimes update models without changing their names. For example, we recently launched an update to GPT-5.6 Sol in ChatGPT (https://openai.com/index/improving-gpt-5-6-sol-in-chatgpt/ https://openai.com/index/improving-gpt-5-6-sol-in-chatgpt/). Our goal isn't to be opaque or sneaky, but just to not exhaust people trying to keep track of little changes. When the changes are big, we give models a new name so that people know to expect something different.
- derefr 2mo ago
  That's an good answer to the question when taken in strict terms of a model version = weights. But, insofar as: 1. "a model" as presented to the user, isn't just its weights, but also anything else happening on the "business layer" (though this maybe applies more to ChatGPT than "direct" model access via the API); and 2. said business layer has any "knowledge base"-type stuff going on in it (i.e. automatic or tool-call-triggered embedding of results from search of some vector-DB into which has been embedded distilled pre-validated trustworthy info — like per-user memory mechanisms, but searching + injecting from global shared data sources); and 3. said "knowledge base" mechanisms are where most of the up-to-date, fast-changing info a model "knows" (without having to do a web search) is actually coming from; ...then do y'all ever update the pinned knowledge-base data snapshot associated with the model version, without updating the weights themselves?
  - tedsanders 2mo ago
    Same spirit as above. API is fixed (though of course things like web search results can change from day to day). ChatGPT and Codex harnesses do change over time, though they never result in name changes. We document the big changes here: https://help.openai.com/en/articles/6825453-chatgpt-release-notes https://help.openai.com/en/articles/6825453-chatgpt-release-...
- discodave 2mo ago
  Is there any scaffolding or harness runnning around the model on the cloud/server side? Seems like there's a lot of opportunity to make changes/improvements without making your customers call a different API or change the model parameter. As an example, S3 team was able to migrate from eventually-consistent, to consistent without making any API changes, a complete re-architecture on the backend with 0 API changes.

by ddxv 2mo ago

tedsanders 2mo ago

I can explain how we do it at OpenAI. In the API, we keep the models fixed. There are tiny caveats like rare bug fixes or models like `chat-latest`, but this is spiritually true. Suspicions of models changing over time are either human hallucinations or bugs on our end. However, in ChatGPT, we sometimes update models without changing their names. For example, we recently launched an update to GPT-5.6 Sol in ChatGPT (https://openai.com/index/improving-gpt-5-6-sol-in-chatgpt/ https://openai.com/index/improving-gpt-5-6-sol-in-chatgpt/). Our goal isn't to be opaque or sneaky, but just to not exhaust people trying to keep track of little changes. When the changes are big, we give models a new name so that people know to expect something different.

derefr 2mo ago
That's an good answer to the question when taken in strict terms of a model version = weights. But, insofar as: 1. "a model" as presented to the user, isn't just its weights, but also anything else happening on the "business layer" (though this maybe applies more to ChatGPT than "direct" model access via the API); and 2. said business layer has any "knowledge base"-type stuff going on in it (i.e. automatic or tool-call-triggered embedding of results from search of some vector-DB into which has been embedded distilled pre-validated trustworthy info — like per-user memory mechanisms, but searching + injecting from global shared data sources); and 3. said "knowledge base" mechanisms are where most of the up-to-date, fast-changing info a model "knows" (without having to do a web search) is actually coming from; ...then do y'all ever update the pinned knowledge-base data snapshot associated with the model version, without updating the weights themselves?
- tedsanders 2mo ago
  Same spirit as above. API is fixed (though of course things like web search results can change from day to day). ChatGPT and Codex harnesses do change over time, though they never result in name changes. We document the big changes here: https://help.openai.com/en/articles/6825453-chatgpt-release-notes https://help.openai.com/en/articles/6825453-chatgpt-release-...
discodave 2mo ago
Is there any scaffolding or harness runnning around the model on the cloud/server side? Seems like there's a lot of opportunity to make changes/improvements without making your customers call a different API or change the model parameter. As an example, S3 team was able to migrate from eventually-consistent, to consistent without making any API changes, a complete re-architecture on the backend with 0 API changes.