differentiable relative to the model parameters. The attention mechanism does store(key(x), value(x)), followed by lookup(query(y)), where key(), value(), looku

differentiable relative to the model parameters. The attention mechanism does store(key(x), value(x)), followed by lookup(query(y)), where key(), value(), lookup(), query() are all designed to be differentiable.