Note that sentiment tends to also jump at the ends of grammatical phrases. For example, "Seriously, the screenplay AND the directing were horrendous" [sudden dr

Note that sentiment tends to also jump at the ends of grammatical phrases. For example, "Seriously, the screenplay AND the directing were horrendous" [sudden drop in sentiment without punctuation] "and clearly done by people who could not fathom what was good about the novel." This seems to have to do with a pretty deep understanding of grammar; the model waits until it the low-level neurons have something to pass up (decoding of a complete unit of meaning) before using that to update its sentiment neuron. A lot of next-character or next-word prediction ends up working like this - internally, the model keeps some state and makes big changes to its understanding at points that have to do with the structure of the stream.