6 ms·

Previous commentary I know of from OpenAI staff: Logan: The API does not just change without us telling you. The models are static there. https://twitter.com/O

by swyx 3y ago

Previous commentary I know of from OpenAI staff: Logan: The API does not just change without us telling you. The models are static there. https://twitter.com/OfficialLoganK/status/1663934947931897857 https://twitter.com/OfficialLoganK/status/166393494793189785... may 31 Peter: No, we haven't made GPT-4 dumber. Quite the opposite: we make each new version smarter than the previous one. https://twitter.com/jlowin/status/1679660938415177731 https://twitter.com/jlowin/status/1679660938415177731 july 14 either the models are static, or they are being improved continuously and there have been unforeseen regressions. only one can be true at any point in time. was this policy changed in the last 1.5 months?

flangola7 3y ago
You can call a specific version of the model. It's ones of the API values. The latter person is referring to the "gpt-4" which the documentation states will update and change without warning.
airgapstopgap 3y ago
Not really. They have a way of squaring this circle, by changing their inference code. Speculative sampling [1] would still make their first claim a lie – sure, there'd still be the original GPT-4 model, plus a smaller draft worker. But early exit decoding [2] allows you to get almost as good results for much cheaper from exactly the same checkpoint. We know that this line of research for large-scale inference is going strong [3] so it stands to reason that OpenAI, with their wealth of talent focused on GPT-4 throughput&inference [4], large contexts and aggressive pricing policy, would also develop something like that. And of course it's "smarter" that way – in a very deceptive sense of the word. 1. https://arxiv.org/abs/2302.01318 https://arxiv.org/abs/2302.01318 2. https://arxiv.org/abs/2207.07061 https://arxiv.org/abs/2207.07061 3. https://arxiv.org/abs/2307.02628 https://arxiv.org/abs/2307.02628 4. https://openai.com/contributions/gpt-4 https://openai.com/contributions/gpt-4
- BoorishBears 3y ago
  I don't get why you're jumping to cloak and daggers style operations: OpenAI would not kneecap their commercial offering by randomly changing how it works. At the end of the day 99% of the confusion comes from people using the web interface, which undoubtedly does change much more often than the API versions they share. The web app they host isn't a simple API wrapper, it does summarization, has some sort of system prompt, and calls the moderation API. That's undoubtedly being updated all the time.
  - Shank 3y ago
    > OpenAI would not kneecap their commercial offering by randomly changing how it works. > As of July 3, 2023, we’ve disabled the Browse with Bing beta feature out of an abundance of caution while we fix this in order to do right by content owners. We are working to bring the beta back as quickly as possible, and appreciate your understanding! https://help.openai.com/en/articles/8077698-how-do-i-use-chatgpt-browse-with-bing-to-search-the-web https://help.openai.com/en/articles/8077698-how-do-i-use-cha...
    - BoorishBears 3y ago
      Thank you for confirming my point? > At the end of the day 99% of the confusion comes from people using the web interface, which undoubtedly does change much more often than the API versions they share. The API does not offer any browsing features, that's the web app.
      - deleted 3y ago
        [deleted]
  - airgapstopgap 3y ago
    No, it makes sense to secure engagement with the most expensive implementation and then cut costs, this kind of stuff is pervasive in the industry. Besides, we have Brockman on record saying that they do "a lot of quantization"[1][2] so it's not paranoia to suspect other optimization schemes when there's a clear performance drop, which they have also denied a few times. 1. https://chat.openai.com/share/44a0c5b6-c629-470a-992f-8cdbbecd64a2 https://chat.openai.com/share/44a0c5b6-c629-470a-992f-8cdbbe... 2. https://www.youtube.com/watch?v=_hpuPi7YZX8 https://www.youtube.com/watch?v=_hpuPi7YZX8

by swyx 3y ago

flangola7 3y ago

You can call a specific version of the model. It's ones of the API values. The latter person is referring to the "gpt-4" which the documentation states will update and change without warning.

airgapstopgap 3y ago

Not really. They have a way of squaring this circle, by changing their inference code. Speculative sampling [1] would still make their first claim a lie – sure, there'd still be the original GPT-4 model, plus a smaller draft worker. But early exit decoding [2] allows you to get almost as good results for much cheaper from exactly the same checkpoint. We know that this line of research for large-scale inference is going strong [3] so it stands to reason that OpenAI, with their wealth of talent focused on GPT-4 throughput&inference [4], large contexts and aggressive pricing policy, would also develop something like that. And of course it's "smarter" that way – in a very deceptive sense of the word. 1. https://arxiv.org/abs/2302.01318 https://arxiv.org/abs/2302.01318 2. https://arxiv.org/abs/2207.07061 https://arxiv.org/abs/2207.07061 3. https://arxiv.org/abs/2307.02628 https://arxiv.org/abs/2307.02628 4. https://openai.com/contributions/gpt-4 https://openai.com/contributions/gpt-4

BoorishBears 3y ago
I don't get why you're jumping to cloak and daggers style operations: OpenAI would not kneecap their commercial offering by randomly changing how it works. At the end of the day 99% of the confusion comes from people using the web interface, which undoubtedly does change much more often than the API versions they share. The web app they host isn't a simple API wrapper, it does summarization, has some sort of system prompt, and calls the moderation API. That's undoubtedly being updated all the time.
- Shank 3y ago
  > OpenAI would not kneecap their commercial offering by randomly changing how it works. > As of July 3, 2023, we’ve disabled the Browse with Bing beta feature out of an abundance of caution while we fix this in order to do right by content owners. We are working to bring the beta back as quickly as possible, and appreciate your understanding! https://help.openai.com/en/articles/8077698-how-do-i-use-chatgpt-browse-with-bing-to-search-the-web https://help.openai.com/en/articles/8077698-how-do-i-use-cha...
  - BoorishBears 3y ago
    Thank you for confirming my point? > At the end of the day 99% of the confusion comes from people using the web interface, which undoubtedly does change much more often than the API versions they share. The API does not offer any browsing features, that's the web app.
    - deleted 3y ago
      [deleted]
- airgapstopgap 3y ago
  No, it makes sense to secure engagement with the most expensive implementation and then cut costs, this kind of stuff is pervasive in the industry. Besides, we have Brockman on record saying that they do "a lot of quantization"[1][2] so it's not paranoia to suspect other optimization schemes when there's a clear performance drop, which they have also denied a few times. 1. https://chat.openai.com/share/44a0c5b6-c629-470a-992f-8cdbbecd64a2 https://chat.openai.com/share/44a0c5b6-c629-470a-992f-8cdbbe... 2. https://www.youtube.com/watch?v=_hpuPi7YZX8 https://www.youtube.com/watch?v=_hpuPi7YZX8