3 ms·
Humans don't just try to imitate text from some group of humans, like language models. But if we wanted to imitate the text of, say, 18th century novelists, the
by cubefox 4y ago
Humans don't just try to imitate text from some group of humans, like language models. But if we wanted to imitate the text of, say, 18th century novelists, then only those 18th century novels seem to matter. If we also started to imitate our own imitations of 18th century novels (synthetic data) the result would probably get worse, since our imitations are imperfect.
- cguess 4y agoTo consider your analogy a bit further: Imagine you had access to only... five 18th century novels, that's it, no other knowledge about the period, the people or anything. Then you try to write your own. You can make up your own stories that are a merging of various parts of those five novels, but they'll always only contain what's in those, or some combination. If all the books deal with France then Germans or English will never make an appearance because it's literally impossible to guess that they exist since they're not in the original books (the books here are training data if that's not obvious).