4 ms·
From Study: "The prompts were presented sequentially in a single chat session and were also tested in an isolated chat session to view context dependency". So
by FrustratedMonky 2y ago
From Study: "The prompts were presented sequentially in a single chat session and were also tested in an isolated chat session to view context dependency".
So both ways.
Are you saying they took both methods and intermixed results to skew a narrative? That might be a bit of a leap, but I didn't go find the raw data to disprove that.
It looks like they asked questions within a context window, and also isolated in separate context windows. And compared results.
It seems like this was actually part of the study. How much does the context window skew results, versus if questions were independent? How is that a bad study?
You are saying the study is bad for doing what the study said it was doing. How can using the same context window be bad if studying the context window is what they were looking at. It sounds like you wanted a different study done where the data gathered would be different.
"context windows could influence their results"
How much and in what way is useful to study. And, as windows get longer, many common users are just going with one long context and not starting a new window.