Whereas vanilla scaled dot-product attention forces outputs to fall in the convex hull of V, your proposal allows them to drift into the convex hull of V+{0}. I

Whereas vanilla scaled dot-product attention forces outputs to fall in the convex hull of V, your proposal allows them to drift into the convex hull of V+{0}. It's pretty unlikely that the origin is already contained in the convex hull of V especially for long inputs, so your proposal genuinely changes the quality of outputs.