4 ms·

ChatGPT study suggests its LLMs are getting dumber at some tasks

17 points by TalktoCrystal 3y ago

consumer451 3y ago
As I've said in previous comments, this seems easily testable by any ChatGPT user. I am curious if this is real, or an example of mass hysteria/delusion. > Do you use the web app? > If so, could you please go back in your history and make a new chat with an old prompt which had an excellent response? > I am curious if you could see the degradation and share the example. >> u/derwiki >> To your point, that we haven’t seen widespread examples of prompts that used to work but now don’t, seems telling about the accuracy of claims. https://news.ycombinator.com/item?id=36798176 https://news.ycombinator.com/item?id=36798176
- TalktoCrystal 3y ago
  i posted this just because this provides an interesting angle to see ChatGPT.
  - consumer451 3y ago
    No worries, I keep re-posting my comment because I really hope someone will share a real-world example of the degradation of ChatGPT.
- deleted 3y ago
  [deleted]
jasfi 3y ago
If this is true, I have some theories: - The quality is more random than people would like. - OpenAI made performance tweaks that caused the quality to drop.
- carlmr 3y ago
  - The self-censoring of the LLM is making it unable to think.
CraftingLinks 3y ago
This study was imo completely debunked. https://twitter.com/Si_Boehm/status/1681801371656536068?t=rOSxxLWuSsr5HJ5QEyK0Pw&s=19 https://twitter.com/Si_Boehm/status/1681801371656536068?t=rO... The claims by that paper are very misleading.
Madmallard 3y ago
Pretty sick of the posts defending ChatGPT on this one. I have at least 5 prompts I retried that had drastically reduced quality of response and basically forcing me to query a dozen times to get the code it previously just wrote in the very first response. It tries to force you into making skeleton models of your code now if there is any complexity at all whatsoever instead of taking a stab at the code itself. I've done so many retrials of old prompts I've seen it happen enough to notice the actual pattern as to what is going on. "Where's the proof" Sir I have literally hundreds of conversations, and I am no longer subscribed to ChatGPT4 because of the aforementioned issues, so I cannot really compare anymore. If you were using it in February to write code, try those same prompts now. It's blatant.

17 points by TalktoCrystal 3y ago

As I've said in previous comments, this seems easily testable by any ChatGPT user. I am curious if this is real, or an example of mass hysteria/delusion. > Do you use the web app? > If so, could you please go back in your history and make a new chat with an old prompt which had an excellent response? > I am curious if you could see the degradation and share the example. >> u/derwiki >> To your point, that we haven’t seen widespread examples of prompts that used to work but now don’t, seems telling about the accuracy of claims. https://news.ycombinator.com/item?id=36798176 https://news.ycombinator.com/item?id=36798176

TalktoCrystal 3y ago
i posted this just because this provides an interesting angle to see ChatGPT.
- consumer451 3y ago
  No worries, I keep re-posting my comment because I really hope someone will share a real-world example of the degradation of ChatGPT.
deleted 3y ago
[deleted]

jasfi 3y ago

If this is true, I have some theories: - The quality is more random than people would like. - OpenAI made performance tweaks that caused the quality to drop.

carlmr 3y ago
- The self-censoring of the LLM is making it unable to think.

CraftingLinks 3y ago

This study was imo completely debunked. https://twitter.com/Si_Boehm/status/1681801371656536068?t=rOSxxLWuSsr5HJ5QEyK0Pw&s=19 https://twitter.com/Si_Boehm/status/1681801371656536068?t=rO... The claims by that paper are very misleading.

Madmallard 3y ago

Pretty sick of the posts defending ChatGPT on this one. I have at least 5 prompts I retried that had drastically reduced quality of response and basically forcing me to query a dozen times to get the code it previously just wrote in the very first response. It tries to force you into making skeleton models of your code now if there is any complexity at all whatsoever instead of taking a stab at the code itself. I've done so many retrials of old prompts I've seen it happen enough to notice the actual pattern as to what is going on. "Where's the proof" Sir I have literally hundreds of conversations, and I am no longer subscribed to ChatGPT4 because of the aforementioned issues, so I cannot really compare anymore. If you were using it in February to write code, try those same prompts now. It's blatant.