6 ms·
I imagine the API quality isnt nerfed on a given day like ChatGPT can be. There was no question something happened in January with ChatGPT, weirdly would refus
by hospitalJail 3y ago
I imagine the API quality isnt nerfed on a given day like ChatGPT can be.
There was no question something happened in January with ChatGPT, weirdly would refuse to answer questions that were harmless but difficult(Give me a daily schedule of a stoic hedonist)
Every once in a while, I see redditors complain of it being nerfed.
Sometimes I go back to gpt3.5 and am mind boggled how much worse it is.
Makes me wonder if they keep increasing the version number while dumbing down the previous model.
With an API, being unreliable would be a deal-breaker. Looking forward to people fine-tuning LLMs with GPT4 API. I'd love it for medical purposes, I'm so worried of a future where the US medical cartels ban ChatGPT for medical purposes. At least with local models, we don't have to worry about regression.
- santiagobasulto 3y agoI felt the same thing. The first version of GPT-4 I tried was crazy smart. Scary smart. Something happened afterwards…
- moffkalast 3y agoThe even more interesting part is that none of us got to try the internal version which was allegedly yet another step above that.
- kossTKR 3y agoWould gladly pay more for a none nerfed version if they were actually honest. The current versions is close to the original 3.5 version, while 3.5 has become horribly bad, such a scam to not disclose what's going on, especially for a paid service.
- deleted 3y ago[deleted]
- politician 3y agoOh, it's not too hard to see how the spend that Microsoft put into building the data centers where GPT-4 was trained attracted national security interest even before it went public. The fact that they were even allowed to release it publicly is likely due to its strategic deterrence effect and that they believed the released version was already a dumbed-down version. The fact that rumors about GPT-5 were quickly suppressed and the models were dumbed down even more cannot be entirely explained by excessive demand. I think it's more likely that GPT-3.5 and GPT-4 demonstrated unexpected capabilities in the hands of the public leading to a pull back. Moreover, Sam Altman's behaviors changed dramatically between the initial release and a few weeks afterward -- the extreme optimism of a CEO followed by a more subdued, even cowed, demeanor despite strong enthusiasm from end-users. OpenAI cannot do anything without Microsoft's data center resources, and Microsoft is a critical defense contractor. Anyway, personally, I'm with the crowd that thinks we're about to see a Cambrian explosion of domain-specific expert AIs. I suspect that OpenAI/Microsoft/Gov is still trying to figure out how much to nerf the capability of GPT-3.5 to tutor smaller models (see "Textbooks are all you need") and that's why the API is trash.
- santiagobasulto 3y agoTrue. The one that is referenced in that "ChatGPT AGI" youtube video, right? the one from a MS researchers that has been recommended to all of us probably. Good video btw.
- aeyes 3y agoI was playing with the API and found that it returned better answers than ChatGPT. ChatGPT isn't even able to solve simple Python problems anymore, even if you try to help it. And some time ago it did these same problems with ease. My guess is that they began to restrict ChatGPT because they can't sell that. They probably want to sell you CodeGPT or other products in the future so why would they give that away for free? ChatGPT is just a teaser.
- incogitor 3y agoProbably some combination of quantizing down from original fp16 weights and changes to the system prompt used for chat. Both can cause degraded quality, the former more than the latter.
- PeterStuer 3y ago"ChatGPT isn't even able to solve simple Python problems anymore, even if you try to help it. And some time ago it did these same problems with ease." This is my experience also. I have not formally benchmarked the different releases, but specifically for Python coding ChatGPT 4 got considerably worse with the latest updates.
- tap-snap-or-nap 3y agoI agree. It is difficult to say what happened exactly but I am certain that I got all the answers and very few canned responses. Whatever they did for safety has degraded the product.
- atleastoptimal 3y agoThe capability of the latest model will be like a Shepard tone: always increasing, never improving. Meanwhile their internal version will be 100x better with no filtering.
- sashank_1509 3y agoI feel like it's code generation abilities have also been nerfed. In the past I got almost excellent code from GPT-4, somehow these days I need multiple prompts to get the code I want from GPT-4.
- stuckkeys 3y agoNot nerfed. They will sell a different tier service to assist with coding. Coming soon. Speculating ofc.
- londons_explore 3y agoIn the API, you can select to use the 14th March 2023 version of GPT-4, and then compare them side by side.
- it_citizen 3y agoI keep reading “GPT4 got nerfed” but I have been using from day 1, and while it definitely gives bad answers, I cannot say that it was nerfed for sure. Is there any actual evidences other than some user subjective experiences?
- dr-detroit 3y ago[dead]
- mike_hearn 3y agoChatGPT is definitely more restricted than the API. Example: https://news.ycombinator.com/item?id=36179783 https://news.ycombinator.com/item?id=36179783
- azemetre 3y agoThat's disappointing, I thought ChatGPT WAS using the API. I mean what's the point of paying if you don't get similar levels of quality?
- mike_hearn 3y agoI thought that too. It's certainly how they present it. But, apparently not.
- fredoliveira 3y agoChatGPT doesn't use the API. It uses the same underlying model with a bunch of added prompts (and possibly additional fine-tuning?) to add to make it conversational. One would pay because what they get out of chatGPT provides value, of course. Keep in mind that the users of these 2 products can be (and in fact are) different — chatGPT is a lot friendlier (from a UX perspective) than using the API playground (or using the API itself).
- redox99 3y agoThey are comparing text-davinci-003 with ChatGPT which presumably uses gpt-3.5-turbo, so quite different models. They are killing text-davinci-003 btw.
- merpnderp 3y agoIt's the continued alignment with fine-tuning that's degrading its responses. You can apparently have it be nice or smart, but not both.
- interstice 3y agoCurious as to whether theres a more general rule at play there about filtering interfering with getting good answers. If there is that's a scary thought from an ethics perspective.
- vbezhenar 3y agoWhy would someone care if its nice or not? It's an algorithm. You're using it to get output, not to get some psychology help.
- moffkalast 3y agoOpenAI presumably cares about being sued if it provides the illegal content they trained it on.
- staticman2 3y agoThere was a guy in the news who asked an AI to tell him it was a good idea to commit suicide, then he killed himself. Even on this forum I've seen AI enthusiasts claiming AI will be the best psychologist, best school teacher, etc.
- logicchains 3y agoThat was Eliza, an AI so old that it's included in stock Emacs, not an LLM. It's propaganda, not news.
- staticman2 3y agoThe chatbot app was called "Eliza" but it's not the Eliza you are thinking of. https://news.ycombinator.com/item?id=35402777 https://news.ycombinator.com/item?id=35402777 https://www.businessinsider.com/widow-accuses-ai-chatbot-reason-husband-kill-himself-2023-4 https://www.businessinsider.com/widow-accuses-ai-chatbot-rea...
- jondwillis 3y agoI hit rate limits and “model is busy with other requests” frequently while just developing a highly concurrent agent app. Especially with the dated (e.g. -0613) or now -16k models.
- seizethecheese 3y agoInstead of the model changing, it’s equally likely that this is a cognitive illusion. A new model is initially mind-blowing and enjoys a halo effect. Over time, this fades and we become frustrated with the limitations that were there all along.
- colordrops 3y agoIt's both. OpenAI is obviously tuning the model for both computational resource constraints as well as "alignment". It's not an either-or.
- hungrigekatze 3y agoCheck out this post from a round table dialogue with Greg Brockman from OpenAI. The GPT models that were in existence / in use in early 2023 were not the performance-degraded quantized versions that are in production now: https://www.reddit.com/r/mlscaling/comments/146rgq2/chatgpt_is_running_quantized/jnro7wo/ https://www.reddit.com/r/mlscaling/comments/146rgq2/chatgpt_...
- sroussey 3y agoOh interesting. I thought that’s what turbo was.
- refulgentis 3y agoIt was, that's what the comment says?
- moffkalast 3y agoNo it's definitely changed a lot. The speedups have been massive (GPT 4 runs faster now than 3.5-turbo did at launch) and they can't be explained with just them rolling out H100s since that's just a 2x inference boost. Some unknown in-house optimization method aside, they've probably quantized the models down to a few bits of precision which increases perplexity quite a bit. They've also continued to RHLF tune to make them more in-line with their guidelines and that process has been shown to decrease overall performance before GPT 4 even launched.
- PerryCox 3y ago"Give me a daily schedule of a stoic hedonist" worked for me just now. https://chat.openai.com/share/04c1dbc0-4890-447f-b5a5-7b1bc5972f73 https://chat.openai.com/share/04c1dbc0-4890-447f-b5a5-7b1bc5...
- hospitalJail 3y agoYes, it didn't work in january. It said it was impossible/wrong to do it.
- anotherpaulg 3y agoI recently completed some benchmarks for code editing that compared the Feb (0301) and June (0613) versions of GPT-3.5 and GPT-4. I found indications that the June version of GPT-3.5 is worse than the Feb version. https://aider.chat/docs/benchmarks.html https://aider.chat/docs/benchmarks.html
- refulgentis 3y agoAfter reading, I don't think <5% points is helpful to add to discussion here without pointing it out explicitly, people are asserting much wilder claims, regularly
- anotherpaulg 3y agoI haven't come across any other systematic, quantitative benchmarking of the OpenAI models' performance over time, so I thought I would share my results. I think my results might argue that there has been some degradation, but not nearly the amount that you often hear people's annecdata about. But unfortunately, you have to read a ways into the doc and understand a lot of details about the benchmark. Here's a direct link and excerpt of the relevant portion: https://aider.chat/docs/benchmarks.html#the-0613-models-seem-worse https://aider.chat/docs/benchmarks.html#the-0613-models-seem... The benchmark results have me fairly convinced that the new gpt-3.5-turbo-0613 and gpt-3.5-16k-0613 models are a bit worse at code editing than the older gpt-3.5-turbo-0301 model. This is visible in the “first attempt” portion of each result, before GPT gets a second chance to edit the code. Look at the horizontal white line in the middle of the first three blue bars. Performance with the whole edit format was 46% for the February model and only 39% for the June models. But also note how much the solid green diff bars degrade between the February and June GPT-3.5 models. They drop from 30% down to about 19%. I saw other signs of this degraded performance in earlier versions of the benchmark as well.
- ren_engineer 3y agoRecently people have claimed GPT4 is an ensemble model with 8 different models under the hood. My guess is that the "nerfing"(I've noticed it as well at random times) is when the model directs a question to the wrong underlying model