4 ms·
I'm not sure either way. While my naïve thought is that factual knowledge ought to be distinct from knowledge of language comprehension, I can't think how to ac
by ben_w 3y ago
I'm not sure either way. While my naïve thought is that factual knowledge ought to be distinct from knowledge of language comprehension, I can't think how to actually do that — how do you riff on a quotation without knowing the quotation? How can you read between the lines if you are given half a phrase where the point is in the unspoken other half?
I don't know enough about amnesia to know if humans even separate these things or not.
If they are separable, LLMs can be made much smaller without loss of comprehension, as "mere facts" can easily be punted off the network and into a pool of searchable documents.
Without that capability, we'd have to teach the models that verbatim quotes are not allowed, just as we have to tell each other about copyright law because humans can recall things sometimes — and in mere conversation it's considered unremarkable (or even good!) if our mouths or hands reproduce perfect reproductions. Poems, pledges, anthems, quotes…
…but for anyone who has memorised, say, Star Wars, you're forbidden from rather than incapable of writing down the entire script.