3 ms·
hmm. after my engineer degree put all of the vector math in the form k = Wx seeing k = xW is jarring. Is there a reason for using horizontal vectors? Common
by 8note 2y ago
hmm. after my engineer degree put all of the vector math in the form
k = Wx
seeing
k = xW
is jarring. Is there a reason for using horizontal vectors? Common for data science docs?
- t55 2y agoIt’s mostly a convention. In many deep learning frameworks (PyTorch, TensorFlow, etc.), inputs are stored with the “batch × length × hidden-dim” shape, effectively making the token embeddings row vectors. Multiplying “xW” is then the natural shape-wise operation. On the other hand, classical linear algebra references often treat vectors as column vectors and write “Wx.”
- quanto 2y agoYou are in the right here. Horizontal vectors are common for (some) deep learning docs, but column factors are the literature standard elsewhere.
- sifar 2y agoIt is more efficient to compute k = xW with the weights transposed than k = Wx.