4 ms·

OpenAI have some degree of versioning with the models used by their APIs, but it seems they are perhaps still updating (fine tuning) models without changing the

OpenAI have some degree of versioning with the models used by their APIs, but it seems they are perhaps still updating (fine tuning) models without changing the model name/version. For ChatGPT itself (not the APIs) many people have reported recent regressions in capability, so it seems the model is being changed there too. As people start to use these API's in production, there needs to be stricter version control, especially given how complex (impossible unless you are only using a fixed set of prompts) it is for anyone to test for backwards compatibility. Maybe something like Ubuntu's stable long-term releases vs bleeding edge ones would work. Have some models that are guaranteed not to change for a specified amount of time, and others that will be periodically updated for people who want cutting edge behavior and care less about backwards compatibility.

nico 3y ago
My guess is that they are doing something like: * use some version of GPT, to preprocess the prompts * send preprocessed prompt to “real” model for inference * post process result to filter out undesired output Then they can honestly say they haven’t changed the version of the model, without telling you that they have probably changed a lot of their pipeline to process your prompts and deliver a result for you
- jstarfish 3y ago
  In the last few years I've noticed lying by omission has become the new fun corporate/gen-z internet trend (see also: overfunding/gofundme fraud). Like priests and fund managers, their product is a black box, and there's a lot of mischief you can get into when you're entrusted with one of those. They play a fucked-up game of Rumpelstiltskin, where they mislead by default and only admit the truth if you can guess the right question to ask. You're on the right track, and I too think that's what their actual pipeline looks like, but you're missing a step. I think there's another step where they effectively alter the output of production models by hot-swapping different LORAs (or whatever) to them. This lets them plausibly claim they haven't changed the version of the model, because they haven't messed with the model. They messed with middleware, which nobody knows enough about to press them on. You ask them if anything changed with the model/API, they say no, and leave you to think you're going crazy because shit's just not working like it was last week. Nobody's asking them about changes to middleware though, which genuinely surprises me. I am never the smartest person in the room-- only the most skeptical.
d_watt 3y ago
Regarding your comment on updating models, are you saying you think they're updating the "pinned" models. EGT gpt4-0314? Otherwise, I think effectively they already have what you're describing with LTS models as pinned versions, and the unversioned model is effectively the "nightly." From what I've see in the materials, it seems like if you pay for dedicated compute, you can also have some control over your model versions.
- HarHarVeryFunny 3y ago
  I don't know. There have been a lot of recent complaints about changed behavior, but not sure which model versions people are talking about.
cj 3y ago
I agree with this. Although there are APIs used in production whose output changes/evolves over time. First one that comes to mind is Google Translate. We spend 6-figures with the Google Translate API annually, and recently we went back and checked if Google Translate is improving/changing the translations over time, and found that they indeed are (presumably as they improve their internal models, which aren't versioned and isn't exposed in a changelog anywhere). The majority of translations were different for the same content today compared to 6 months ago. I don't particularly agree with this approach. Speaking as a power user of Google Translate API, it would be nice to be able to pin to specific version/model and then manually upgrade versions (with a changelog to understand what's changing under the hood).
- ladberg 3y ago
  At this point the changelog is surely just stuff along the lines of "retrained with more data" and "slightly tweaked model architecture in a random way that improved performance". And out of curiosity: as someone with a lot of expertise and money on the line, how would you compare Google Translate with LLMs? And also smaller self-hosted models with bigger ones that require API access like OpenAI? Do they perform better or worse and are they cheaper or more expensive?
  - KRAKRISMOTT 3y ago
    I assume they probably switched from BERT to GPT architecture at some point.
    - HarHarVeryFunny 3y ago
      I wouldn't assume that. BERT uses the original encoder-decoder transformer architecture which takes advantage of the entire input sequence being available, so you have the forward context as well as backward context of a word to assist with translation. GPT uses the decoder-only transformer architecture and masked attention so that it only relies on backward context, which allows it to predict word-by-word. It's not a better architecture, just one adapted for different requirements. If you are able to use the forward context, then I'd assume that a BERT-like architecture would do better since it has more information available.
deeviant 3y ago
FYI, OpenAI has denied that there have been any unannounced changes to the models. And it seems more than possible that people see what they want to see in LLM output. So I would be careful to make completely unsupported claims.
- ineedasername 3y ago
  Sure but they could be playing with semantics a bit as well. When they say "models" they might just mean the LLM that it's all based on. But there's a long more going on in the pipeline to turn that into a consumer facing service like ChatGPT. They might have changed any combination of the following: 1) Fine Tuning 2) Embedding 3) The initializing prompt 4) Filtering a prompt prior to ingestion & tokenization of the prompt 5) Filtering the output from the application after it has generated a response. The statement "we have no unannounced changes to the models" can be true while still substantially changing functionality & response quality through any of the 5 above areas, and probably some I missed.
- qumpis 3y ago
  I'd be so pleasantly surprised if the same model has remained the same. Everytime I see speedups in its generation speed I assume they've distilled the model further. The outputs subjectively also feel weaker. Surely someone has compared the outputs from the beginning and now?
deleted 3y ago
[deleted]

nico 3y ago

My guess is that they are doing something like: * use some version of GPT, to preprocess the prompts * send preprocessed prompt to “real” model for inference * post process result to filter out undesired output Then they can honestly say they haven’t changed the version of the model, without telling you that they have probably changed a lot of their pipeline to process your prompts and deliver a result for you

jstarfish 3y ago
In the last few years I've noticed lying by omission has become the new fun corporate/gen-z internet trend (see also: overfunding/gofundme fraud). Like priests and fund managers, their product is a black box, and there's a lot of mischief you can get into when you're entrusted with one of those. They play a fucked-up game of Rumpelstiltskin, where they mislead by default and only admit the truth if you can guess the right question to ask. You're on the right track, and I too think that's what their actual pipeline looks like, but you're missing a step. I think there's another step where they effectively alter the output of production models by hot-swapping different LORAs (or whatever) to them. This lets them plausibly claim they haven't changed the version of the model, because they haven't messed with the model. They messed with middleware, which nobody knows enough about to press them on. You ask them if anything changed with the model/API, they say no, and leave you to think you're going crazy because shit's just not working like it was last week. Nobody's asking them about changes to middleware though, which genuinely surprises me. I am never the smartest person in the room-- only the most skeptical.

d_watt 3y ago

Regarding your comment on updating models, are you saying you think they're updating the "pinned" models. EGT gpt4-0314? Otherwise, I think effectively they already have what you're describing with LTS models as pinned versions, and the unversioned model is effectively the "nightly." From what I've see in the materials, it seems like if you pay for dedicated compute, you can also have some control over your model versions.

HarHarVeryFunny 3y ago
I don't know. There have been a lot of recent complaints about changed behavior, but not sure which model versions people are talking about.

cj 3y ago

I agree with this. Although there are APIs used in production whose output changes/evolves over time. First one that comes to mind is Google Translate. We spend 6-figures with the Google Translate API annually, and recently we went back and checked if Google Translate is improving/changing the translations over time, and found that they indeed are (presumably as they improve their internal models, which aren't versioned and isn't exposed in a changelog anywhere). The majority of translations were different for the same content today compared to 6 months ago. I don't particularly agree with this approach. Speaking as a power user of Google Translate API, it would be nice to be able to pin to specific version/model and then manually upgrade versions (with a changelog to understand what's changing under the hood).

ladberg 3y ago
At this point the changelog is surely just stuff along the lines of "retrained with more data" and "slightly tweaked model architecture in a random way that improved performance". And out of curiosity: as someone with a lot of expertise and money on the line, how would you compare Google Translate with LLMs? And also smaller self-hosted models with bigger ones that require API access like OpenAI? Do they perform better or worse and are they cheaper or more expensive?
- KRAKRISMOTT 3y ago
  I assume they probably switched from BERT to GPT architecture at some point.
  - HarHarVeryFunny 3y ago
    I wouldn't assume that. BERT uses the original encoder-decoder transformer architecture which takes advantage of the entire input sequence being available, so you have the forward context as well as backward context of a word to assist with translation. GPT uses the decoder-only transformer architecture and masked attention so that it only relies on backward context, which allows it to predict word-by-word. It's not a better architecture, just one adapted for different requirements. If you are able to use the forward context, then I'd assume that a BERT-like architecture would do better since it has more information available.

deeviant 3y ago

FYI, OpenAI has denied that there have been any unannounced changes to the models. And it seems more than possible that people see what they want to see in LLM output. So I would be careful to make completely unsupported claims.

ineedasername 3y ago
Sure but they could be playing with semantics a bit as well. When they say "models" they might just mean the LLM that it's all based on. But there's a long more going on in the pipeline to turn that into a consumer facing service like ChatGPT. They might have changed any combination of the following: 1) Fine Tuning 2) Embedding 3) The initializing prompt 4) Filtering a prompt prior to ingestion & tokenization of the prompt 5) Filtering the output from the application after it has generated a response. The statement "we have no unannounced changes to the models" can be true while still substantially changing functionality & response quality through any of the 5 above areas, and probably some I missed.
qumpis 3y ago
I'd be so pleasantly surprised if the same model has remained the same. Everytime I see speedups in its generation speed I assume they've distilled the model further. The outputs subjectively also feel weaker. Surely someone has compared the outputs from the beginning and now?

deleted 3y ago

[deleted]