> As I understand programs like BERT, they largely look at the words adjacent to a given word, without paying attention to any structure. That's completely opp

> As I understand programs like BERT, they largely look at the words adjacent to a given word, without paying attention to any structure. That's completely opposite to what happens. In BERT all words are related to all words so information circulates in one step between all pairs. This combinatorial interaction happens on multiple parallel "heads" and sequential "layers", so it can express complex symbol manipulation tasks. In GPT information circulates between all pairs of words but only from past to the future, not the other way around (it is autoregressive). What you are talking about is the RNN or LSTM. They only see the adjacent tokens. This causes an informational bottleneck which limits their expressive power. > Is there a way to force a neural net to construct hierarchical models? And of course the hierarchies are not different for every word all nouns behave more or less the same, all intransitive verbs behave in a different way, etc. Here are a few types of attention maps. When you stack them up you net a hierarchical model. https://media.arxiv-vanity.com/render-output/5515593/x5.png https://media.arxiv-vanity.com/render-output/5515593/x5.png