3 ms·
The n-grams aren't wrong, but it is a real problem that the underlying corpus distribution changes massively over time (in this case, proportion of academic vs.
by crazygringo 3y ago
The n-grams aren't wrong, but it is a real problem that the underlying corpus distribution changes massively over time (in this case, proportion of academic vs. non-academic works).
This is a really devilish problem with no easy answer.
Because on the one hand, it's certainly easy enough to normalize by genre -- e.g. fix academic works at 20%, popular magazines at 20%, fiction books at 40%, and so forth.
But the problem is that the popularity of genres changes over time separately in terms of supply and demand, as well as consumption of printed material overall. Fiction written might increase while fiction consumed might decrease. Or the consumption of books might decrease as television consumption increases.
So there isn't any objectively "right" answer at all.
But it would be nice if Google allowed you to plot popularity by genre -- I think that would help a lot in terms of determining where and how words become more or less common.