3 ms·
My intuition for that, and you can tell me if its wrong. The normal explanation for Word2Vec is 2 weight matrices, so the formula looks like this: (One_hot_inp
by MichaelStaniek 7y ago
My intuition for that, and you can tell me if its wrong.
The normal explanation for Word2Vec is 2 weight matrices, so the formula looks like this: (One_hot_input x W1) x W2, which is then softmaxed.
W1 then is the matrix that contain our focus embedding from, but if we only evaluate specific words on the target side, then W2 are actually our context embeddings, and the normal multiplication then is focus_w x context_w.
Am I wrong?
- bollu 7y agoNow it's `one_hot_focus x W1 x (one_hot_context x W2)^T`. So we still pick one row of the matrix from the focus and context embeddings, but they're separate embeddings.
- MichaelStaniek 7y agoYes, but thats also what happens in the normal formulation, no? So the second weight matrix actually are our context embeddings?
- avn2109 7y agoNew user "MichaelStaniek" has a grammatical, relevant, good-faith sibling comment to this which is (imho) inexplicably banned, almost certainly due to a mistake, and I hope somebody will unban him.