6 ms·

There's no doubt that it's gotten a lot worse on coding, I've been using this benchmark on each new version of GPT-4 "Write a tiptap extension that toggles clas

There's no doubt that it's gotten a lot worse on coding, I've been using this benchmark on each new version of GPT-4 "Write a tiptap extension that toggles classes" and so far it's gotten it right every time, but not any more, now it hallucinates a simplified solution that don't even use the tiptap api any more. It's also 200% more verbose in explaining it's reasoning, even if that reasoning makes no sense whatsoever - it's like it's gotten more apologetic and generic. The answer is the same on GPT plus and API with GPT-4, even with "developer" role.

avereveard 3y ago
Do you have by any chance tested the same question on the playground? I've noticed a quality decrease iny telegram bot as well that directly uses the API, and it drives me crazy because model versioning was supposedly implemented specifically to avoid response change without notice
- tmikaeld 3y ago
  Yes, using the general assistant role and the default content: "You are ChatGPT, a large language model trained by OpenAI, based on the GPT-4 architecture. Knowledge cutoff: 2021-09 Current date: 2023-05-31" And custom roles with custom content via API.
bugglebeetle 3y ago
It was a great ride while it lasted. My assumption is that efficacy at coding tasks is such a small percent of users, they’ve just sacrificed it on the altar of efficiency and/or scale. That, or they’ve cut some back room deal with Microsoft to make Copilot have access to the only version of the model that can actually code.
- jdiez17 3y ago
  Copilot X (the new version, with a chat interface etc) is significantly worse than GPT-4 (at least before this update). It felt like gpt3.5-turbo to me.
  - sagarpatil 3y ago
    Have they started rolling it out? When did you get access?
    - dbmnt 3y ago
      I've had access since 2023-05-13. You have to use the Insiders build of VS Code, and a nightly version of the Copilot extension.
      - sagarpatil 3y ago
        I take it that you have to subscribe to Copilot in order to get access to Copilot X?
        JanSt 3y ago
        yes you have to subscribe
  - LinearEntropy 3y ago
    I have spent the last couple of days playing with Copilot X Chat, to help me learn Ruby on Rails. I'd have thought that Rails would be something it would be competent with. My experience has been atrocious. It makes up gems and functions. Rails commands it gives are frequently incorrect. Trying to use it to debug issues results in it responding with the same incorrect answer repeatedly, often removing necessary lines.
- silisili 3y ago
  Honestly, why not different versions at this point? People who want it for coding don't care if it knows the history of prerevolution France, and vice versa. Seems they could wow more people if they had specialized versions, rather than the jack of all trades that tries to exist now. Edit: Oh God, I just described our human system of specialty and how the AI could replace us using the same means...
  - htrp 3y ago
    >Edit: Oh God, I just described our human system of specialty and how the AI could replace us using the same means... Welcome to the Future... just like the present, but worse for you. In all seriousness, there has been a lot of work done to show that smaller specialized models are better for their own domains and its entirely possible that GPT4 could become a routing mechanism for individual models (think toolformer).
- TeMPOraL 3y ago
  FWIW, I started to get the same feeling as the OP about GPT-4 model I have access to on Azure, so if there's any deal being cut here, it might involve dumbing down the model for paying Azure customers as well. Now, to be clear: I only started to get a feeling that GPT-4 on Azure is getting worse. I didn't do any specific testing for this so far, as I thought I may just be imagining it. This thread is starting to convince me otherwise.
  - bugglebeetle 3y ago
    I’ve seen degradation in the app and via the API, so if I had to bet, they’ve probably kneecapped the model so that it works passably everywhere they’ve been made it available vs. works well in one place or another.
    - TeMPOraL 3y ago
      Yes. I think 'sirsinsalot is likely right in suggesting[0] that they could be trying "to hoard the capability to out compete any competitor, of any kind, commercially or politically and hide the true extent of your capability to avoid scrutiny and legislation", and that they're currently "dialing back the public expectations", possibly while "deploying the capability in a novel way to exploit it as the largest lever" they can. That view is consistent with GPT-4 getting dumber on both OpenAI proper and Azure OpenAI - even as the companies and corporations using the latter are paying through the nose for the privilege. Alternative take is that they're doing it to slow the development of the whole field down, per all the AI safety letters and manifestos that they've been signing and circulating - but that would be at best a stop-gap before OSS models catch up, and it's more than likely that OpenAI and/or Microsoft would succumb to the temptation of doing what 'sirsinsalot suggested anyway. -- [0] - https://news.ycombinator.com/item?id=36135425 https://news.ycombinator.com/item?id=36135425
      - cma 3y ago
        If it got faster at the same time it could just be bait and switch with a quantized/sparsified replacement.
- dx034 3y ago
  Maybe it had to do with jailbreaks? A lot of the jailbreaks were related to coding, so maybe they put more restrictions in there. Only speculating, but I cannot imagine why it got worse otherwise.
- windex 3y ago
  Also the deal to make the browsing model to only use Bing. That's bait and switch. I paid for browsing, and now it only browses Bing. They even had the gall to update the plugin name to Browsing with Bing.
  - sebzim4500 3y ago
    It can definitely browse websites that aren't Bing, I asked it to look at a page that isn't in the bing cache and it worked.
    - mrtranscendence 3y ago
      Clearly "Browse with Bing" doesn't mean that it will only browse bing.com, but what exactly does it mean? I can't quite figure it out. Is it that it's identifying as a Bing crawler?
      - tough 3y ago
        Marketing ?
bumbledraven 3y ago
Do you have API access? If so, have you tried your tiptap question on the gpt-4-0314 model? That is supposedly the original version released to the public on March 14.
- tmikaeld 3y ago
  I did, but it got it almost the same as GPT-3.5 Turbo, the best version of it where there recently (~2-3 weeks ago), where it would make specific chunks of code-changes and explain the chunk in a concise and correct manner - even making suggestions on improvements. But that's entirely gone now..
theaussiestew 3y ago
Wait so you’ve gotten GPT 4 to successfully write TipTap extensions for you? Are you using Copilot or the ChatGPT app?
- tmikaeld 3y ago
  Not only writing, extending and figuring out quite complicated usage based on the API documentation. I'll open source some of them in the near future. I'm using ChatGPT Plus with GPT-4, that gave the best results. Also worked via API key and custom prompts.
  - theaussiestew 3y ago
    Have you tried Bing? I’m also building a TipTap based app so hearing this is quite eye opening, I didn’t think LLMs were up to doing this kind of specialised library usage. Got any examples you could share?
    - tmikaeld 3y ago
      If you mean Bard, it's not available in the EU so I can't. Of course, this one is almost fully authored by GPT-4: https://hastebin.com/share/juqogogari.typescript https://hastebin.com/share/juqogogari.typescript We also made extensions for: font-weight font-size font-family tailwind-manage With different use-cases, the most interesting one is tailwind manager, which manages classes for different usage. Tiptap is excellent when building a headless site-builder.
      - theaussiestew 3y ago
        Impressive, this'll cut down on my work a lot. When I say Bing, I meant Bing AI which also uses GPT-4. Can you share some of the prompts you've been using? I'm assuming you don't need to paste in context around the library, you simply ask it to use TipTap and it'll do that?
        tmikaeld 3y ago

avereveard 3y ago

Do you have by any chance tested the same question on the playground? I've noticed a quality decrease iny telegram bot as well that directly uses the API, and it drives me crazy because model versioning was supposedly implemented specifically to avoid response change without notice

tmikaeld 3y ago
Yes, using the general assistant role and the default content: "You are ChatGPT, a large language model trained by OpenAI, based on the GPT-4 architecture. Knowledge cutoff: 2021-09 Current date: 2023-05-31" And custom roles with custom content via API.

bugglebeetle 3y ago

It was a great ride while it lasted. My assumption is that efficacy at coding tasks is such a small percent of users, they’ve just sacrificed it on the altar of efficiency and/or scale. That, or they’ve cut some back room deal with Microsoft to make Copilot have access to the only version of the model that can actually code.

jdiez17 3y ago
Copilot X (the new version, with a chat interface etc) is significantly worse than GPT-4 (at least before this update). It felt like gpt3.5-turbo to me.
- sagarpatil 3y ago
  Have they started rolling it out? When did you get access?
  - dbmnt 3y ago
    I've had access since 2023-05-13. You have to use the Insiders build of VS Code, and a nightly version of the Copilot extension.
    - sagarpatil 3y ago
      I take it that you have to subscribe to Copilot in order to get access to Copilot X?
      - JanSt 3y ago
        yes you have to subscribe
- LinearEntropy 3y ago
  I have spent the last couple of days playing with Copilot X Chat, to help me learn Ruby on Rails. I'd have thought that Rails would be something it would be competent with. My experience has been atrocious. It makes up gems and functions. Rails commands it gives are frequently incorrect. Trying to use it to debug issues results in it responding with the same incorrect answer repeatedly, often removing necessary lines.
silisili 3y ago
Honestly, why not different versions at this point? People who want it for coding don't care if it knows the history of prerevolution France, and vice versa. Seems they could wow more people if they had specialized versions, rather than the jack of all trades that tries to exist now. Edit: Oh God, I just described our human system of specialty and how the AI could replace us using the same means...
- htrp 3y ago
  >Edit: Oh God, I just described our human system of specialty and how the AI could replace us using the same means... Welcome to the Future... just like the present, but worse for you. In all seriousness, there has been a lot of work done to show that smaller specialized models are better for their own domains and its entirely possible that GPT4 could become a routing mechanism for individual models (think toolformer).
TeMPOraL 3y ago
FWIW, I started to get the same feeling as the OP about GPT-4 model I have access to on Azure, so if there's any deal being cut here, it might involve dumbing down the model for paying Azure customers as well. Now, to be clear: I only started to get a feeling that GPT-4 on Azure is getting worse. I didn't do any specific testing for this so far, as I thought I may just be imagining it. This thread is starting to convince me otherwise.
- bugglebeetle 3y ago
  I’ve seen degradation in the app and via the API, so if I had to bet, they’ve probably kneecapped the model so that it works passably everywhere they’ve been made it available vs. works well in one place or another.
  - TeMPOraL 3y ago
    Yes. I think 'sirsinsalot is likely right in suggesting[0] that they could be trying "to hoard the capability to out compete any competitor, of any kind, commercially or politically and hide the true extent of your capability to avoid scrutiny and legislation", and that they're currently "dialing back the public expectations", possibly while "deploying the capability in a novel way to exploit it as the largest lever" they can. That view is consistent with GPT-4 getting dumber on both OpenAI proper and Azure OpenAI - even as the companies and corporations using the latter are paying through the nose for the privilege. Alternative take is that they're doing it to slow the development of the whole field down, per all the AI safety letters and manifestos that they've been signing and circulating - but that would be at best a stop-gap before OSS models catch up, and it's more than likely that OpenAI and/or Microsoft would succumb to the temptation of doing what 'sirsinsalot suggested anyway. -- [0] - https://news.ycombinator.com/item?id=36135425 https://news.ycombinator.com/item?id=36135425
    - cma 3y ago
      If it got faster at the same time it could just be bait and switch with a quantized/sparsified replacement.
dx034 3y ago
Maybe it had to do with jailbreaks? A lot of the jailbreaks were related to coding, so maybe they put more restrictions in there. Only speculating, but I cannot imagine why it got worse otherwise.
windex 3y ago
Also the deal to make the browsing model to only use Bing. That's bait and switch. I paid for browsing, and now it only browses Bing. They even had the gall to update the plugin name to Browsing with Bing.
- sebzim4500 3y ago
  It can definitely browse websites that aren't Bing, I asked it to look at a page that isn't in the bing cache and it worked.
  - mrtranscendence 3y ago
    Clearly "Browse with Bing" doesn't mean that it will only browse bing.com, but what exactly does it mean? I can't quite figure it out. Is it that it's identifying as a Bing crawler?
    - tough 3y ago
      Marketing ?

bumbledraven 3y ago

Do you have API access? If so, have you tried your tiptap question on the gpt-4-0314 model? That is supposedly the original version released to the public on March 14.

tmikaeld 3y ago
I did, but it got it almost the same as GPT-3.5 Turbo, the best version of it where there recently (~2-3 weeks ago), where it would make specific chunks of code-changes and explain the chunk in a concise and correct manner - even making suggestions on improvements. But that's entirely gone now..

theaussiestew 3y ago

Wait so you’ve gotten GPT 4 to successfully write TipTap extensions for you? Are you using Copilot or the ChatGPT app?

tmikaeld 3y ago
Not only writing, extending and figuring out quite complicated usage based on the API documentation. I'll open source some of them in the near future. I'm using ChatGPT Plus with GPT-4, that gave the best results. Also worked via API key and custom prompts.
- theaussiestew 3y ago
  Have you tried Bing? I’m also building a TipTap based app so hearing this is quite eye opening, I didn’t think LLMs were up to doing this kind of specialised library usage. Got any examples you could share?
  - tmikaeld 3y ago
    If you mean Bard, it's not available in the EU so I can't. Of course, this one is almost fully authored by GPT-4: https://hastebin.com/share/juqogogari.typescript https://hastebin.com/share/juqogogari.typescript We also made extensions for: font-weight font-size font-family tailwind-manage With different use-cases, the most interesting one is tailwind manager, which manages classes for different usage. Tiptap is excellent when building a headless site-builder.
    - theaussiestew 3y ago
      Impressive, this'll cut down on my work a lot. When I say Bing, I meant Bing AI which also uses GPT-4. Can you share some of the prompts you've been using? I'm assuming you don't need to paste in context around the library, you simply ask it to use TipTap and it'll do that?
      - tmikaeld 3y ago