It is a run-time executed reweighting of predictions. The scalar is a multiplication between the outputs of the Key and Query kernels passed through softmax, wh

It is a run-time executed reweighting of predictions. The scalar is a multiplication between the outputs of the Key and Query kernels passed through softmax, which then multiply the result of the value kernel V(x). Now, you may argue that because it's a multiplication with a linear or affine kernel, you might as well use commutative property of scalar multiplication and multiply the factor with the weights first, and then multiply with the input to the kernel. But this only holds for very few kernels.