3 ms·

Looks like it's just a matrix decomposition in the paper. I'm guessing anyway. These attention papers are always a painful mix of mathematical, quasi-mathematic

Looks like it's just a matrix decomposition in the paper. I'm guessing anyway. These attention papers are always a painful mix of mathematical, quasi-mathematical, and information retrieval jargon. There is something in the github repo about higher-order decompositions. Don't find where the method for factoring is given.

verdverm 2y ago
I chuckled when I read, in S-3.1 > Specifically, for each token t, with a small abuse of notation, we define:
- jamessb 2y ago
  "Abuse of notation" is a commonly used term: https://en.wikipedia.org/wiki/Abuse_of_notation https://en.wikipedia.org/wiki/Abuse_of_notation