Model drift is something that ML practitioners have been accommodating for many years now. Even when the model is entirely under your control you have to handle

Model drift is something that ML practitioners have been accommodating for many years now. Even when the model is entirely under your control you have to handle it. So if you're reading this thinking, "I can't use any 3rd party LLM APIs as they could change" then yes, that is the case, but you can use them as long as you have a system which can detect and react to model drift. OpenAI, at least, has been clear that it doesn't change the behavior of specific named models without warning. ChatGPT UI is not constrained by this, only the APIs, so if you are 'evaluating' the performance of GPT-* with the UI then you really have no control or guarantees. Instead make sure you've developed a robust test set that you can use to evaluate newly released model versions and only upgrade if/when they meet your needs. You'll also need a pipeline to continually update this test set because your user behavior and mix will change over time. Perhaps the most unusual thing about dealing with the APIs is the extent of regressions you need to expect in updated versions. The API surface area of LLMs is effectively infinite, so there is no way for a company to guarantee it won't regress on the parts you care about. If you think about model versions the same way you think about software package versions you are going to be continually surprised and disappointed.