Hmm not that I know of, but that would be neat! A lot of the frameworks and model code treat this sort of thing as ‘implementation details’. Which is disappoint

Hmm not that I know of, but that would be neat! A lot of the frameworks and model code treat this sort of thing as ‘implementation details’. Which is disappointing because I think it adds perspective and intuition. One other example would be how multi-head attention is implemented with a single matrix. You don’t actually create matrices for each of the N ‘heads’ separately. It’s a logical distinction