3 ms·

Apparently, you're missing the part where they're all defined :) It does take some getting used to, communicating math in English (and visa-versa). For exampl

Apparently, you're missing the part where they're all defined :) It does take some getting used to, communicating math in English (and visa-versa). For example: "two learnable matrices Q and K ... are d_k by d_f". What more do you need to know about them? Also, "a fixed number of features ... call it d_f", I don't know how to more clearly define what d_f is? I could have called the matrices Joe and Curly, and the feature dimension could have been called Sarah, I suppose. But it's not clear that that's any more helpful... and to anyone with any passing familiarity with math (such as my grad students), giving instances of mathematical objects concise and symbolic names is generally easier to parse, especially when it comes to translating the words into actual math or code.

fireworkcrayon 3y ago
I wasn’t trying to be critical - sorry if it came across that way! > Also, "a fixed number of features ... call it d_f", I don't know how to more clearly define what d_f is? Perfect example. I would expect a fixed number to be represented by a single letter, like x. But here, a the fixed number seems to need two letters to represent it: d and f. When I see this, I wonder “what’s d?” “What’s f?” So I scan ahead: I see there is a d_k, so I think there must be some sibling relationship between f and k, but I can’t see what it is. As far as I can tell, Q/q and K/k are the ones that belong in a pair. It gets very confusing trying to decipher the meaning and relationships.
- cshimmin 3y ago
  No offense taken, I have twice un-deaded your comments because you seem to be posting in good faith :) Again, you get used to it. Note, the underscore would represent a subscript (in the standard math-writing tool of LaTeX, but also in many variants of Markdown). Here, d stands for dimension. If you talk about dimensionality a lot, this is one of a few common letters to use to describe it (others popular contenders, especially when speaking about pairs of dimensions would would be n and m, p and q, r and s. But N is already used in a different context [weirdly n/N seems to be an exception to the following paragraph]). And since there are a few different dimensions floating around, we label them all with d, and give them subscripts to indicate which one we're talking about. d_f symbolically means "dimension of the feature space", d_k means "dimension of the key space" (also the same as the query space. in my description I called them question/answer instead of query/key). d_v means "dimension of the value space." In many contexts, the use of capital letters implicitly denotes matrices. So if you have a matrix that turns x (your input variable), into q (a query vector), naming it Q makes a lot of sense. Especially when we have three matrices floating around, it would have been much more confusing to keep track of if we named the matrices M, N, O. Which one projects to which vector? Much easier to understand that Q makes the q vectors, K makes the k vectors, V makes the v vectors.
  - fireworkcrayon 3y ago
    Thanks for stepping me through that. I was able to reread and understand your original explanation. To be honest, though, I was really just trying to understand what my roadblock is in general. As I mentioned, I experience this kind of friction with most math-related writing. And, sadly, I do have at least a "passing familiarity" with math. I don't really struggle with the concepts. For me it's the communication layer: the variety of conventions that exist never seem to jell into a coherent system in the way that natural languages do - even with all of their idiomatic and arbitrary features. Anyway, thanks again for the discussion.

fireworkcrayon 3y ago

I wasn’t trying to be critical - sorry if it came across that way! > Also, "a fixed number of features ... call it d_f", I don't know how to more clearly define what d_f is? Perfect example. I would expect a fixed number to be represented by a single letter, like x. But here, a the fixed number seems to need two letters to represent it: d and f. When I see this, I wonder “what’s d?” “What’s f?” So I scan ahead: I see there is a d_k, so I think there must be some sibling relationship between f and k, but I can’t see what it is. As far as I can tell, Q/q and K/k are the ones that belong in a pair. It gets very confusing trying to decipher the meaning and relationships.

cshimmin 3y ago
No offense taken, I have twice un-deaded your comments because you seem to be posting in good faith :) Again, you get used to it. Note, the underscore would represent a subscript (in the standard math-writing tool of LaTeX, but also in many variants of Markdown). Here, d stands for dimension. If you talk about dimensionality a lot, this is one of a few common letters to use to describe it (others popular contenders, especially when speaking about pairs of dimensions would would be n and m, p and q, r and s. But N is already used in a different context [weirdly n/N seems to be an exception to the following paragraph]). And since there are a few different dimensions floating around, we label them all with d, and give them subscripts to indicate which one we're talking about. d_f symbolically means "dimension of the feature space", d_k means "dimension of the key space" (also the same as the query space. in my description I called them question/answer instead of query/key). d_v means "dimension of the value space." In many contexts, the use of capital letters implicitly denotes matrices. So if you have a matrix that turns x (your input variable), into q (a query vector), naming it Q makes a lot of sense. Especially when we have three matrices floating around, it would have been much more confusing to keep track of if we named the matrices M, N, O. Which one projects to which vector? Much easier to understand that Q makes the q vectors, K makes the k vectors, V makes the v vectors.
- fireworkcrayon 3y ago
  Thanks for stepping me through that. I was able to reread and understand your original explanation. To be honest, though, I was really just trying to understand what my roadblock is in general. As I mentioned, I experience this kind of friction with most math-related writing. And, sadly, I do have at least a "passing familiarity" with math. I don't really struggle with the concepts. For me it's the communication layer: the variety of conventions that exist never seem to jell into a coherent system in the way that natural languages do - even with all of their idiomatic and arbitrary features. Anyway, thanks again for the discussion.