4 ms·
> I don't think LLMs can reason about the prevalence of ideas in their training set Good point. But isn't there a similar issue with things like 'now'? If you
by galaxyLogic 3y ago
> I don't think LLMs can reason about the prevalence of ideas in their training set
Good point. But isn't there a similar issue with things like 'now'? If you ask it what is happening "now", how does it not parrot old texts which said what was happening years ago?
- triyambakam 3y agoProbably the training data included something like "the current year is 2023" which semantically maps to "now"
- galaxyLogic 3y agoWhen you say "semantically maps" do you mean that somebody somewhere coded such a "fact" into the training set? Or how is the mapping specified? If the training texts say "Current year is 2023" it would be wrong already :-)
- feanaro 3y agoMore likely the model has a system prompt authoritatively saying what "now" is, and it can reason about other times specified in other resources in the training set because those resources specified their own time reference. So even though a training resource said "It is DATE today. An IMPORTANT THING happened.", it knows that IMPORTANT THING happened in the past, because it knows CURRENT DATE from the system prompt, and it also knows that DATE < CURRENT DATE.
- galaxyLogic 3y agoI see, So there is like a hi8gher level of facts/sources which are marked as such by the creators of the system. Higher level "facts" override the ones acquired from the public. I didn't know about "system prompts".