In principle, adding more depth to the network can take care of that. If you are keeping track of higher and higher level abstractions, you can remember what yo

In principle, adding more depth to the network can take care of that. If you are keeping track of higher and higher level abstractions, you can remember what you were saying. But you can't remember everything, so ideally you'd also have an attention model that's capable of looking back at what you've written, and make edits. When I balance parentheses, I don't maintain a counter of how many parentheses are open, I look back at what I've written and count them.