3 ms·
Your full quote is "LLMs are able to often _preciesely_ recreate in contrast to MP3 at best being approximate." Unless you are using the word "often" in a very
by ben_w 2y ago
Your full quote is "LLMs are able to often _preciesely_ recreate in contrast to MP3 at best being approximate."
Unless you are using the word "often" in a very different way to me, for this claim to be correct it would need apply to at least a substantial part of the training set — for example of the usage, if I were to say that "when I go to work, I *often* pass through Landbeach", the fact that this genuinely happened hundreds of times does not make it "often" due to the other fact that the most recent time this occurred was 2014.
It's impossible for LLMs to be able to "often _preciesely_ recreate" because there's not enough parameters to do that.
> 1st shot out of 1 is a strong sign I wasn't lucky unless you accuse me of lying.
Consider the converse: my test failed two out of two.
The second example in particular is a failure mode that LLMs are often criticised for: hallucination.
You picked a document which I anticipate would be a biased example; but even then, consider all the people who did a one-shot, saw it fail, and therefore didn't think it was reproducing content accurately, and therefore didn't post a comment.
To call your claim a "lie" would be to presume that was deliberate, which I would not do without much stronger evidence — as the saying goes, "lies damned lies and statistics", this is statistics.
A fair test isn't one document like you gave, nor two like I gave, it's hundreds or thousands of documents, chosen with careful consideration to the distribution to make sure there's no bias towards e.g. mass-copied newspaper articles (or, for image generators, the Mona Lisa).
This is just as important for determining limits to the quality of the models as to the question of if they are memorising things.