4 ms·

Hi! Can you clarify? This is a fairly standard use of "context" (http://www.socher.org/index.php/Main/ImprovingWordRepresentationsViaGlobalContextAndMultipleWor

Hi! Can you clarify? This is a fairly standard use of "context" (http://www.socher.org/index.php/Main/ImprovingWordRepresentationsViaGlobalContextAndMultipleWordPrototypes http://www.socher.org/index.php/Main/ImprovingWordRepresenta...). While the algorithm can be used with just the covariance of a corpus's word vectors, accounting for the distribution of a context when setting up the distance metric acts to improve performance substantially (https://i.imgur.com/Dnx9c9G.jpg https://i.imgur.com/Dnx9c9G.jpg)

syllogism 8y ago
I just think that titling a paper about a weighted bag-of-words approach "Context is everything" really starts the reader off on the wrong foot. In fact there's almost no context in your model, in contrast to most other sentence learning methods!
- ContextArxiv 8y ago
  To be clear, the weighted bag-of-words is a powerful application of the contextual salience measurement, CoSal, but is certainly not the only one. (In practice, this can also be applied outside of language, for the importance of other semantic vectors in context)
SteveJS 8y ago
‘Attention is all you need’, was the first thought i had when i read the title of your paper. It might be worthwhile to have a small bit explaining the distinction between attention and context as you view them. If i understand correctly you are describing how on to build a highly useful context representation. The application of that context in a neural net would be to focus the attention of the neural network on the parts of the input that are more important. From my point of view your paper here helps answer the question: what directs the attention? Apologies if all of this is blindingly obvious. I’m a fascinated amateur.
- ContextArxiv 8y ago
  Yep, exactly. The current paper presents the contextual model and how to interact with it, and then shows that even simple models (such as the weighted bag of words in the first parent comment) based on it do quite well. I really appreciate the interest and please let me know if you have any questions! :)

syllogism 8y ago

I just think that titling a paper about a weighted bag-of-words approach "Context is everything" really starts the reader off on the wrong foot. In fact there's almost no context in your model, in contrast to most other sentence learning methods!

ContextArxiv 8y ago
To be clear, the weighted bag-of-words is a powerful application of the contextual salience measurement, CoSal, but is certainly not the only one. (In practice, this can also be applied outside of language, for the importance of other semantic vectors in context)

SteveJS 8y ago

‘Attention is all you need’, was the first thought i had when i read the title of your paper. It might be worthwhile to have a small bit explaining the distinction between attention and context as you view them. If i understand correctly you are describing how on to build a highly useful context representation. The application of that context in a neural net would be to focus the attention of the neural network on the parts of the input that are more important. From my point of view your paper here helps answer the question: what directs the attention? Apologies if all of this is blindingly obvious. I’m a fascinated amateur.

ContextArxiv 8y ago
Yep, exactly. The current paper presents the contextual model and how to interact with it, and then shows that even simple models (such as the weighted bag of words in the first parent comment) based on it do quite well. I really appreciate the interest and please let me know if you have any questions! :)