4 ms·
They're not really that different. There's only a second vector for a word in the (common, default) negative-sampling case, where each predictable word has a d
by gojomo 7y ago
They're not really that different.
There's only a second vector for a word in the (common, default) negative-sampling case, where each predictable word has a distinct "output node" of the neural network, and the second vector is the in-weights to that one node. Still, most implementations don't emphasize this vector – the classic "word-vector" is a word's representation when it's a neural-network input. And in the hierarchical-softmax training mode, there's no clear second vector.
I suspect the original word2vec authors left out a clearer description of the initialization as they were following some oft-assumed practices implied by their other descriptions.
Another minor difference between the literal descriptions, and original C implementation, was a slightly different looping order in skip-gram training: holding a target-word, and then looping over all context-words, rather than holding a context-word, then looping over all neighboring target-words. One of the authors once mentioned that the shipped approach was slightly more efficient – maybe it was due to CPU cache issues? In any case all the same context->target pairs get trained either way, just in a slightly different order.