4 ms·

It was a great ride while it lasted. My assumption is that efficacy at coding tasks is such a small percent of users, they’ve just sacrificed it on the altar of

It was a great ride while it lasted. My assumption is that efficacy at coding tasks is such a small percent of users, they’ve just sacrificed it on the altar of efficiency and/or scale. That, or they’ve cut some back room deal with Microsoft to make Copilot have access to the only version of the model that can actually code.

jdiez17 3y ago
Copilot X (the new version, with a chat interface etc) is significantly worse than GPT-4 (at least before this update). It felt like gpt3.5-turbo to me.
- sagarpatil 3y ago
  Have they started rolling it out? When did you get access?
  - dbmnt 3y ago
    I've had access since 2023-05-13. You have to use the Insiders build of VS Code, and a nightly version of the Copilot extension.
    - sagarpatil 3y ago
      I take it that you have to subscribe to Copilot in order to get access to Copilot X?
      - JanSt 3y ago
        yes you have to subscribe
- LinearEntropy 3y ago
  I have spent the last couple of days playing with Copilot X Chat, to help me learn Ruby on Rails. I'd have thought that Rails would be something it would be competent with. My experience has been atrocious. It makes up gems and functions. Rails commands it gives are frequently incorrect. Trying to use it to debug issues results in it responding with the same incorrect answer repeatedly, often removing necessary lines.
silisili 3y ago
Honestly, why not different versions at this point? People who want it for coding don't care if it knows the history of prerevolution France, and vice versa. Seems they could wow more people if they had specialized versions, rather than the jack of all trades that tries to exist now. Edit: Oh God, I just described our human system of specialty and how the AI could replace us using the same means...
- htrp 3y ago
  >Edit: Oh God, I just described our human system of specialty and how the AI could replace us using the same means... Welcome to the Future... just like the present, but worse for you. In all seriousness, there has been a lot of work done to show that smaller specialized models are better for their own domains and its entirely possible that GPT4 could become a routing mechanism for individual models (think toolformer).
TeMPOraL 3y ago
FWIW, I started to get the same feeling as the OP about GPT-4 model I have access to on Azure, so if there's any deal being cut here, it might involve dumbing down the model for paying Azure customers as well. Now, to be clear: I only started to get a feeling that GPT-4 on Azure is getting worse. I didn't do any specific testing for this so far, as I thought I may just be imagining it. This thread is starting to convince me otherwise.
- bugglebeetle 3y ago
  I’ve seen degradation in the app and via the API, so if I had to bet, they’ve probably kneecapped the model so that it works passably everywhere they’ve been made it available vs. works well in one place or another.
  - TeMPOraL 3y ago
    Yes. I think 'sirsinsalot is likely right in suggesting[0] that they could be trying "to hoard the capability to out compete any competitor, of any kind, commercially or politically and hide the true extent of your capability to avoid scrutiny and legislation", and that they're currently "dialing back the public expectations", possibly while "deploying the capability in a novel way to exploit it as the largest lever" they can. That view is consistent with GPT-4 getting dumber on both OpenAI proper and Azure OpenAI - even as the companies and corporations using the latter are paying through the nose for the privilege. Alternative take is that they're doing it to slow the development of the whole field down, per all the AI safety letters and manifestos that they've been signing and circulating - but that would be at best a stop-gap before OSS models catch up, and it's more than likely that OpenAI and/or Microsoft would succumb to the temptation of doing what 'sirsinsalot suggested anyway. -- [0] - https://news.ycombinator.com/item?id=36135425 https://news.ycombinator.com/item?id=36135425
    - cma 3y ago
      If it got faster at the same time it could just be bait and switch with a quantized/sparsified replacement.
dx034 3y ago
Maybe it had to do with jailbreaks? A lot of the jailbreaks were related to coding, so maybe they put more restrictions in there. Only speculating, but I cannot imagine why it got worse otherwise.
windex 3y ago
Also the deal to make the browsing model to only use Bing. That's bait and switch. I paid for browsing, and now it only browses Bing. They even had the gall to update the plugin name to Browsing with Bing.
- sebzim4500 3y ago
  It can definitely browse websites that aren't Bing, I asked it to look at a page that isn't in the bing cache and it worked.
  - mrtranscendence 3y ago
    Clearly "Browse with Bing" doesn't mean that it will only browse bing.com, but what exactly does it mean? I can't quite figure it out. Is it that it's identifying as a Bing crawler?
    - tough 3y ago
      Marketing ?

jdiez17 3y ago

Copilot X (the new version, with a chat interface etc) is significantly worse than GPT-4 (at least before this update). It felt like gpt3.5-turbo to me.

sagarpatil 3y ago
Have they started rolling it out? When did you get access?
- dbmnt 3y ago
  I've had access since 2023-05-13. You have to use the Insiders build of VS Code, and a nightly version of the Copilot extension.
  - sagarpatil 3y ago
    I take it that you have to subscribe to Copilot in order to get access to Copilot X?
    - JanSt 3y ago
      yes you have to subscribe
LinearEntropy 3y ago
I have spent the last couple of days playing with Copilot X Chat, to help me learn Ruby on Rails. I'd have thought that Rails would be something it would be competent with. My experience has been atrocious. It makes up gems and functions. Rails commands it gives are frequently incorrect. Trying to use it to debug issues results in it responding with the same incorrect answer repeatedly, often removing necessary lines.

silisili 3y ago

Honestly, why not different versions at this point? People who want it for coding don't care if it knows the history of prerevolution France, and vice versa. Seems they could wow more people if they had specialized versions, rather than the jack of all trades that tries to exist now. Edit: Oh God, I just described our human system of specialty and how the AI could replace us using the same means...

htrp 3y ago
>Edit: Oh God, I just described our human system of specialty and how the AI could replace us using the same means... Welcome to the Future... just like the present, but worse for you. In all seriousness, there has been a lot of work done to show that smaller specialized models are better for their own domains and its entirely possible that GPT4 could become a routing mechanism for individual models (think toolformer).

TeMPOraL 3y ago

FWIW, I started to get the same feeling as the OP about GPT-4 model I have access to on Azure, so if there's any deal being cut here, it might involve dumbing down the model for paying Azure customers as well. Now, to be clear: I only started to get a feeling that GPT-4 on Azure is getting worse. I didn't do any specific testing for this so far, as I thought I may just be imagining it. This thread is starting to convince me otherwise.

bugglebeetle 3y ago
I’ve seen degradation in the app and via the API, so if I had to bet, they’ve probably kneecapped the model so that it works passably everywhere they’ve been made it available vs. works well in one place or another.
- TeMPOraL 3y ago
  Yes. I think 'sirsinsalot is likely right in suggesting[0] that they could be trying "to hoard the capability to out compete any competitor, of any kind, commercially or politically and hide the true extent of your capability to avoid scrutiny and legislation", and that they're currently "dialing back the public expectations", possibly while "deploying the capability in a novel way to exploit it as the largest lever" they can. That view is consistent with GPT-4 getting dumber on both OpenAI proper and Azure OpenAI - even as the companies and corporations using the latter are paying through the nose for the privilege. Alternative take is that they're doing it to slow the development of the whole field down, per all the AI safety letters and manifestos that they've been signing and circulating - but that would be at best a stop-gap before OSS models catch up, and it's more than likely that OpenAI and/or Microsoft would succumb to the temptation of doing what 'sirsinsalot suggested anyway. -- [0] - https://news.ycombinator.com/item?id=36135425 https://news.ycombinator.com/item?id=36135425
  - cma 3y ago
    If it got faster at the same time it could just be bait and switch with a quantized/sparsified replacement.

dx034 3y ago

Maybe it had to do with jailbreaks? A lot of the jailbreaks were related to coding, so maybe they put more restrictions in there. Only speculating, but I cannot imagine why it got worse otherwise.

windex 3y ago

Also the deal to make the browsing model to only use Bing. That's bait and switch. I paid for browsing, and now it only browses Bing. They even had the gall to update the plugin name to Browsing with Bing.

sebzim4500 3y ago
It can definitely browse websites that aren't Bing, I asked it to look at a page that isn't in the bing cache and it worked.
- mrtranscendence 3y ago
  Clearly "Browse with Bing" doesn't mean that it will only browse bing.com, but what exactly does it mean? I can't quite figure it out. Is it that it's identifying as a Bing crawler?
  - tough 3y ago
    Marketing ?