3 ms·
Maybe I'm not understanding what the original user was testing here, but it seems like the finding is that different inputs result in a statistically significan
by tqi 3y ago
Maybe I'm not understanding what the original user was testing here, but it seems like the finding is that different inputs result in a statistically significant difference in output length? Seems pretty unsurprising?
- potatopatch 3y agoThe difference seems too small not to rule out all sorts of things, but the general idea would be that you can't predict shorter/longer for one minor change, so the average of samplings of each should be similar unless the thing modified relates to terseness in the training set.
- tqi 3y agoI think you'd expect them to be similar, but not exactly the same? This feels like the original author is over anthropomorphizing LLMs, and expecting them to interpret prompts the way humans would, but it seems obvious that changing the prompt results in a slightly different context window, which results in a slightly different response distribution? Similarly, if you changed whether the bit about time of year was at the beginning or the end of the prompt, I would expect a statistically different distribution of response lengths.