5 ms·

I've been paying for GPT-4 since 3 hours after its release. The decrease in quality was noticeable just one week later (on top of the cap changes from 50 messag

I've been paying for GPT-4 since 3 hours after its release. The decrease in quality was noticeable just one week later (on top of the cap changes from 50 messages every 4 hours to 25 messages every 3 hours) I originally assumed that this was due to the increase in demand. It never went back to being as sharp as it was during those first hours of usage

pwillia7 3y ago
totally agree -- Bing GPT4 still works somewhat better in my opinion but has declined in quality alongside chat gpt+ too
- StackOverlord 3y ago
  What about the GPT4 hosted on poe.com. It has the same output pace as it had in march.
  - ssabev 3y ago
    Thought they use Claude instead of gpt
ChatGTP 3y ago
Are you sure it wasn't just that the novelty wore off after a few hours of usage? I never really got into LLMs, but I must say at first it seemed like pretty cool stuff. OpenAI have repeatedly stated the model hasn't changed so how could this happen otherwise?
- devinprater 3y ago
  I mean, they could be lying. Or only talking about the API version and not the front-facing ChatGPT.
  - ChatGTP 3y ago
    I feel like it would be a fairly large conspiracy by the OpenAI team though? In fact, what motive would they have to make the model dumber, really? If it gets out and you're right, I think it will cause major trust issues with the product.
    - majkinetor 3y ago
      They probably do not have a motive to make it dumber, but its a side effect of some other agenda, like higher profits.
    - CuriouslyC 3y ago
      I'm sure they aren't trying to make the model dumber, so much as they're trying to cut compute/infra costs so they can be profitable.
    - JohnFen 3y ago
      > I feel like it would be a fairly large conspiracy by the OpenAI team though? I'm not sure what you mean by "conspiracy", but this sort of thing isn't unknown. All it takes is the employees being bound by an NDA and a marketing team can say anything it likes (within the bounds of legality, anyway) without fear that they will spill the beans.
      - ChatGTP 3y ago
        I'm not a lawyer but I think saying anything about the model would be in beach of the NDA then? Maybe they went and go special permission to say that the model hasn't been changed since launch?
    - londons_explore 3y ago
      They want to free up GPU's for other projects. And to do that, they need to make the model dumber.
  - superq 3y ago
    Not saying it's so in this case (or even in the general case), but "marketing" is sometimes a synonym for lying.
- nemo44x 3y ago
  I think this is overlooked. Is it getting worse or are our expectations getting higher as we become more familiar with it? We aren't blown away any more and as our requests get more complex and we become more critical of its output, we mistake that for it getting worse. This is the part of hype cycle named "Peak of Inflated Expectations", at least when it comes to the potential of the tool. I think we are still in the "Innovation Trigger" phase in terms of applying the technology.
- xtracto 3y ago
  These models have been explicitly nerfed since their first release due to copyright considerations. I've mentioned in two previous cases both for [1] code generation and [2] book summarizing. From my point of view, it is sad that these sort of socio-political constructs (copyright) are hindering innovation. The funny thing is that in say, 10 years, the "pirate" version of LLMs will be way more powerful and useful than the "corporate" versions. Once the RIAA/MPAA has sued all of them preventing them of using their audio/video; after the editorials have sued them to prevent them from using their books and articles, and after every internet site has sued them to prevent them from using their text. LLM models trained on SciHub, Library Genesis and Torrents will be amazing in comparison. I sincerely wish that some country would apply to information copyright a similar approach to what India does for medicine patents. [1] https://news.ycombinator.com/item?id=31852138 https://news.ycombinator.com/item?id=31852138 [2] https://news.ycombinator.com/item?id=36138930 https://news.ycombinator.com/item?id=36138930
  - robterrell 3y ago
    > From my point of view, it is sad that these sort of socio-political constructs (copyright) are hindering innovation Why not pay authors of the data the LLM has ingested?
    - SirMaster 3y ago
      Don't a lot of content creators expect payment per use? Like every time someone streams a song on sportify, the artist gets a few pennies. So should they be paid a few pennies every time the LLM spits out a response that "used" that training data? And I am pretty sure it's not even possible to really link the output back to training data anyways.
      - robterrell 3y ago
        > Like every time someone streams a song on sportify, the artist gets a few pennies You're thinking of broadcast radio. Streaming is a fraction of a penny! > So should they be paid a few pennies every time the LLM spits out a response that "used" that training data? That doesn't sound unreasonable! I understand that current LLMs have no way to report "this token came from this data" but that doesn't mean it's impossible to build. (Ack: this is a full-on proper dunning-kruger, having not looked into it & having zero knowledge of the field.) But I think it's probably more reasonable to simply split a % of the service revenue across everyone whose data was used. Or pay an up-fee for ingesting the data in the first place. Generally, it's crazy to name that some people think it's reasonable for these companies to pay for GPUs and CPUs and electricity to run them but not for the data that's the actual core of their service.
- yodsanklai 3y ago
  It's a reasonable hypothesis. Whenever phenomenons are probabilistic, we're poorly equipped to use our direct experience to assess what they do. We see that all the time with pseudo-medicine for instance. And people became extremely defensive about these questions (vaccines, pseudo-sciences...) - which is why you're downvoted.

pwillia7 3y ago

totally agree -- Bing GPT4 still works somewhat better in my opinion but has declined in quality alongside chat gpt+ too

StackOverlord 3y ago
What about the GPT4 hosted on poe.com. It has the same output pace as it had in march.
- ssabev 3y ago
  Thought they use Claude instead of gpt

ChatGTP 3y ago

Are you sure it wasn't just that the novelty wore off after a few hours of usage? I never really got into LLMs, but I must say at first it seemed like pretty cool stuff. OpenAI have repeatedly stated the model hasn't changed so how could this happen otherwise?

devinprater 3y ago
I mean, they could be lying. Or only talking about the API version and not the front-facing ChatGPT.
- ChatGTP 3y ago
  I feel like it would be a fairly large conspiracy by the OpenAI team though? In fact, what motive would they have to make the model dumber, really? If it gets out and you're right, I think it will cause major trust issues with the product.
  - majkinetor 3y ago
    They probably do not have a motive to make it dumber, but its a side effect of some other agenda, like higher profits.
  - CuriouslyC 3y ago
    I'm sure they aren't trying to make the model dumber, so much as they're trying to cut compute/infra costs so they can be profitable.
  - JohnFen 3y ago
    > I feel like it would be a fairly large conspiracy by the OpenAI team though? I'm not sure what you mean by "conspiracy", but this sort of thing isn't unknown. All it takes is the employees being bound by an NDA and a marketing team can say anything it likes (within the bounds of legality, anyway) without fear that they will spill the beans.
    - ChatGTP 3y ago
      I'm not a lawyer but I think saying anything about the model would be in beach of the NDA then? Maybe they went and go special permission to say that the model hasn't been changed since launch?
  - londons_explore 3y ago
    They want to free up GPU's for other projects. And to do that, they need to make the model dumber.
- superq 3y ago
  Not saying it's so in this case (or even in the general case), but "marketing" is sometimes a synonym for lying.
nemo44x 3y ago
I think this is overlooked. Is it getting worse or are our expectations getting higher as we become more familiar with it? We aren't blown away any more and as our requests get more complex and we become more critical of its output, we mistake that for it getting worse. This is the part of hype cycle named "Peak of Inflated Expectations", at least when it comes to the potential of the tool. I think we are still in the "Innovation Trigger" phase in terms of applying the technology.
xtracto 3y ago
These models have been explicitly nerfed since their first release due to copyright considerations. I've mentioned in two previous cases both for [1] code generation and [2] book summarizing. From my point of view, it is sad that these sort of socio-political constructs (copyright) are hindering innovation. The funny thing is that in say, 10 years, the "pirate" version of LLMs will be way more powerful and useful than the "corporate" versions. Once the RIAA/MPAA has sued all of them preventing them of using their audio/video; after the editorials have sued them to prevent them from using their books and articles, and after every internet site has sued them to prevent them from using their text. LLM models trained on SciHub, Library Genesis and Torrents will be amazing in comparison. I sincerely wish that some country would apply to information copyright a similar approach to what India does for medicine patents. [1] https://news.ycombinator.com/item?id=31852138 https://news.ycombinator.com/item?id=31852138 [2] https://news.ycombinator.com/item?id=36138930 https://news.ycombinator.com/item?id=36138930
- robterrell 3y ago
  > From my point of view, it is sad that these sort of socio-political constructs (copyright) are hindering innovation Why not pay authors of the data the LLM has ingested?
  - SirMaster 3y ago
    Don't a lot of content creators expect payment per use? Like every time someone streams a song on sportify, the artist gets a few pennies. So should they be paid a few pennies every time the LLM spits out a response that "used" that training data? And I am pretty sure it's not even possible to really link the output back to training data anyways.
    - robterrell 3y ago
      > Like every time someone streams a song on sportify, the artist gets a few pennies You're thinking of broadcast radio. Streaming is a fraction of a penny! > So should they be paid a few pennies every time the LLM spits out a response that "used" that training data? That doesn't sound unreasonable! I understand that current LLMs have no way to report "this token came from this data" but that doesn't mean it's impossible to build. (Ack: this is a full-on proper dunning-kruger, having not looked into it & having zero knowledge of the field.) But I think it's probably more reasonable to simply split a % of the service revenue across everyone whose data was used. Or pay an up-fee for ingesting the data in the first place. Generally, it's crazy to name that some people think it's reasonable for these companies to pay for GPUs and CPUs and electricity to run them but not for the data that's the actual core of their service.
yodsanklai 3y ago
It's a reasonable hypothesis. Whenever phenomenons are probabilistic, we're poorly equipped to use our direct experience to assess what they do. We see that all the time with pseudo-medicine for instance. And people became extremely defensive about these questions (vaccines, pseudo-sciences...) - which is why you're downvoted.