3 ms·
Exactly. I think of them as Markov Chains in grammar space or in Abstract Syntax Tree space instead of n-gram chain-of-words space. The attention mechanism like
by BenoitEssiambre 1y ago
Exactly. I think of them as Markov Chains in grammar space or in Abstract Syntax Tree space instead of n-gram chain-of-words space. The attention mechanism likely plays a role in identifying the parent in the grammar tree or identifying other types of back references like pronouns or if it's for programming languages, variable back references.
- benob 1y agoThere was a time when people would estimate n-gram probabilities with feed-forward neural networks [1,2]. We just improved that with the (multilayer) attention mechanism which allows for better factoring over individual tokens. It also allowed for much larger n. [1] https://jmlr.org/papers/volume3/bengio03a/bengio03a.pdf https://jmlr.org/papers/volume3/bengio03a/bengio03a.pdf [2] https://www.sciencedirect.com/science/article/abs/pii/S0885230806000325 https://www.sciencedirect.com/science/article/abs/pii/S08852...