3 ms·

All of the tests are one-shot questions and answers. Where I have found GPT-4 to be degrading significantly is with sustained discussion about technical topics.

All of the tests are one-shot questions and answers. Where I have found GPT-4 to be degrading significantly is with sustained discussion about technical topics. It starts forgetting important parts of the discussion almost straight away, long before the size of the context window becomes a factor. This wasn’t the case when it was new.

JoshTko 3y ago
I definitely experienced this as well. Like conversation context is soo quickly forgotten these days.
cloudking 3y ago
If your discussions involve code, you should try including the latest code on each prompt. The token window will start forgetting earlier parts of the conversation. For example: Now improve this feature with xyz Current code: -- {code} --
- JimDabell 3y ago
  As I said, it starts forgetting important parts of the discussion almost straight away, long before the size of the context window becomes a factor.
TheHumanist 3y ago
It's wild how sometimes it seems to completely lose context after just 2 or 3 prompts. Most of the time it doesn't but just yesterday I gave a prompt, pasted some code in... didn't really like the first response but wanted to build off it. So, I asked a follow up in the next prompt and in the next response it was wildly off-base and it's response made no sense and it had hallucinated all these functions into my code that had no business there. 2 prompts from me, 1 response from gpt and then it's second response it is completely lost.
purplecats 3y ago
> It starts forgetting important parts of the discussion almost straight away, long before the size of the context window becomes a factor. This wasn’t the case when it was new. Exactly! I remember vividly I was enthralled by GPT-4 because for the first time in my life, I met an entity that was able to take every character of what I wrote into account. It understood the nuances and reason behind each word. It was a breath of fresh air, because even at top companies, written communication is weak as it seems to be interpreted in a lossy manner. It is now almost on the other end of the spectrum where it can't even remember things that I have repeated in effort to help it stick. They had something beautiful, and diluted the shit out of it after they baited you -- which I suppose was the point. I wonder who gets to use the those high quality models now.

JoshTko 3y ago

I definitely experienced this as well. Like conversation context is soo quickly forgotten these days.

cloudking 3y ago

If your discussions involve code, you should try including the latest code on each prompt. The token window will start forgetting earlier parts of the conversation. For example: Now improve this feature with xyz Current code: -- {code} --

JimDabell 3y ago
As I said, it starts forgetting important parts of the discussion almost straight away, long before the size of the context window becomes a factor.

TheHumanist 3y ago

It's wild how sometimes it seems to completely lose context after just 2 or 3 prompts. Most of the time it doesn't but just yesterday I gave a prompt, pasted some code in... didn't really like the first response but wanted to build off it. So, I asked a follow up in the next prompt and in the next response it was wildly off-base and it's response made no sense and it had hallucinated all these functions into my code that had no business there. 2 prompts from me, 1 response from gpt and then it's second response it is completely lost.

purplecats 3y ago

> It starts forgetting important parts of the discussion almost straight away, long before the size of the context window becomes a factor. This wasn’t the case when it was new. Exactly! I remember vividly I was enthralled by GPT-4 because for the first time in my life, I met an entity that was able to take every character of what I wrote into account. It understood the nuances and reason behind each word. It was a breath of fresh air, because even at top companies, written communication is weak as it seems to be interpreted in a lossy manner. It is now almost on the other end of the spectrum where it can't even remember things that I have repeated in effort to help it stick. They had something beautiful, and diluted the shit out of it after they baited you -- which I suppose was the point. I wonder who gets to use the those high quality models now.