4 ms·
This adds connections between the parameters of q, k, and v whereas the original doesn't, unless my very tired brain is missing something.
by jszymborski 1y ago
This adds connections between the parameters of q, k, and v whereas the original doesn't, unless my very tired brain is missing something.
- smus 1y agoNope, they all depend on x and the same is true in this scenario
- godelski 1y agoIt is actually really common practice. It is a single linear layer so there's no connection intranodes. The reason to do this is because it is a bit less computationally intensive. tldr: linear layers have an associative property