4 ms·
The "trick" is that word vectors are just projections of sparse tensor. For example the word "queen" can be represented by its attributes. Representing in trip
by GistNoesis 7y ago
The "trick" is that word vectors are just projections of sparse tensor.
For example the word "queen" can be represented by its attributes.
Representing in triplet form from wikidata Q16511993
(queen, subclass ,monarch),(queen, subclass, female),...
A king would be
(king, subclass ,monarch),(king, subclass, male),
Transitioning from queen to king is just changing an attribute in this triplet space, i.e removing a triplet and adding another. All those operations are linear. And then you take the projections.
This is kind of similar to the kernel trick of SVMs. You do the calculations in a lower dimensional space (word vectors) but implicitly you are doing them in the higher dimensional space (graphs of relations, aka triplets lists).
- joycian 7y agoThe sparse tensor perspective intrigues me, could you perhaps provide me with some references?
- GistNoesis 7y agoNot specifically. You can look at the literature for ontology, relation networks, semantic databases, reasoning. The video that made me clicked, was https://www.ted.com/talks/ajit_narayanan_a_word_game_to_communicate_in_any_language https://www.ted.com/talks/ajit_narayanan_a_word_game_to_comm... It depicts in a game to help autistic children better communicate, a simple order-independent language representation, which is isomorphic to graphs. If you do the math, you see that some triplets query like (queen, subclass, ?) is just a point-wise multiplication with the knowledge database sparse tensor in this sparse boolean tensor representation (the ? is a column of 1). Then it's just chaining requests. A graph of triplet is isomorphic to a boolean tensor of dimension (nbSubjects, nbRelations, nbObjects).
- mlevental 7y agothat person's description is unnecessarily jargony. sparse tensor just means vector that's mostly zero and has some ones. take your corpus of concepts and number them 0 to n-1. then each word you want to turn into a vec turns into a vector that has ones for concepts it's associated with and 0 elsewhere. learning that association is what word2vec does.
- rytill 7y agoCan you explain attention in this same way?
- mlevental 7y agonah i don't know much about LSTM but this looks like a fine tutorial https://towardsdatascience.com/intuitive-understanding-of-attention-mechanism-in-deep-learning-6c9482aecf4f https://towardsdatascience.com/intuitive-understanding-of-at...
- zwaps 7y agoOne contention I have with this description (if I understand it at all), is that the projection into the linear space does not preserve the properties of changes in a relation graph. I mean I "think" I said pretty much the same in my reply, but one big issue with word2vec is that the relations, which are graphical, can be projected, but not all changes in the graph can be. Therefore, I think it is not correct that these operations are linear in the sense that they can be equivalent to linear operations in the embedding space. But maybe I just don't understand your description.