7 ms·

Hi, author here! I can confirm everything you said. A conference version is in the works, though this one is essentially complete

braindead_in 8y ago
Do you plan to release an implementation? I would definitely like to play around with it.
- ContextArxiv 8y ago
  Absolutely. The current priority is conference submission, but releasing a simple interface is definitely the plan, as appropriate
  - ContextArxiv 8y ago
    https://github.com/ezelikman/Context-Is-Everything https://github.com/ezelikman/Context-Is-Everything The code will be posted here when available!
- auvrw 8y ago
  here's an impl of something like (but actually probably nothing like) 1. in the above tl;dr (disclaimer: i dr) https://github.com/ransomw/botiful-soup/blob/master/mibot/src/clj/mibot/learn/metrics.clj#L17 https://github.com/ransomw/botiful-soup/blob/master/mibot/sr... as i recall, there's a lot of neato REPL-y graphvis to try in https://github.com/ransomw/botiful-soup/tree/master/mibot/dev https://github.com/ransomw/botiful-soup/tree/master/mibot/de... ---- while i'm here, let me mention: Gärdenfors books like "Geometry of Meaning" are hearty food for thought about natural language semantics
syllogism 8y ago
A small suggestion: I think the way you're describing this as "context" might be misleading. You'll probably want to try the same thing on the output of a BiLSTM or CNN model instead of just the word vectors. I think this is more like an attention mechanism --- like tf-idf, you're learning to downweight words which are less informative. Your method is also very similar in effect to the LexRank line of work, which is more computationally expensive and complicated.
- ContextArxiv 8y ago
  Hi! Can you clarify? This is a fairly standard use of "context" (http://www.socher.org/index.php/Main/ImprovingWordRepresentationsViaGlobalContextAndMultipleWordPrototypes http://www.socher.org/index.php/Main/ImprovingWordRepresenta...). While the algorithm can be used with just the covariance of a corpus's word vectors, accounting for the distribution of a context when setting up the distance metric acts to improve performance substantially (https://i.imgur.com/Dnx9c9G.jpg https://i.imgur.com/Dnx9c9G.jpg)
  - syllogism 8y ago
    I just think that titling a paper about a weighted bag-of-words approach "Context is everything" really starts the reader off on the wrong foot. In fact there's almost no context in your model, in contrast to most other sentence learning methods!
    - ContextArxiv 8y ago
      To be clear, the weighted bag-of-words is a powerful application of the contextual salience measurement, CoSal, but is certainly not the only one. (In practice, this can also be applied outside of language, for the importance of other semantic vectors in context)
  - SteveJS 8y ago
    ‘Attention is all you need’, was the first thought i had when i read the title of your paper. It might be worthwhile to have a small bit explaining the distinction between attention and context as you view them. If i understand correctly you are describing how on to build a highly useful context representation. The application of that context in a neural net would be to focus the attention of the neural network on the parts of the input that are more important. From my point of view your paper here helps answer the question: what directs the attention? Apologies if all of this is blindingly obvious. I’m a fascinated amateur.
    - ContextArxiv 8y ago
whymauri 8y ago
Was this a 224n project? I'm hoping to try doing the class online on my own. Do you think the 2017 videos are still pretty in-line with the topics on the current website?
- ContextArxiv 8y ago
  Hi! This started off as a 224n project, but it's gone through quite a bit of work since then. The class slides are from the last class are pretty complete. I think they put a lot of time in to keep it updated, especially towards the end

braindead_in 8y ago

Do you plan to release an implementation? I would definitely like to play around with it.

ContextArxiv 8y ago
Absolutely. The current priority is conference submission, but releasing a simple interface is definitely the plan, as appropriate
- ContextArxiv 8y ago
  https://github.com/ezelikman/Context-Is-Everything https://github.com/ezelikman/Context-Is-Everything The code will be posted here when available!
auvrw 8y ago
here's an impl of something like (but actually probably nothing like) 1. in the above tl;dr (disclaimer: i dr) https://github.com/ransomw/botiful-soup/blob/master/mibot/src/clj/mibot/learn/metrics.clj#L17 https://github.com/ransomw/botiful-soup/blob/master/mibot/sr... as i recall, there's a lot of neato REPL-y graphvis to try in https://github.com/ransomw/botiful-soup/tree/master/mibot/dev https://github.com/ransomw/botiful-soup/tree/master/mibot/de... ---- while i'm here, let me mention: Gärdenfors books like "Geometry of Meaning" are hearty food for thought about natural language semantics

syllogism 8y ago

A small suggestion: I think the way you're describing this as "context" might be misleading. You'll probably want to try the same thing on the output of a BiLSTM or CNN model instead of just the word vectors. I think this is more like an attention mechanism --- like tf-idf, you're learning to downweight words which are less informative. Your method is also very similar in effect to the LexRank line of work, which is more computationally expensive and complicated.

ContextArxiv 8y ago
Hi! Can you clarify? This is a fairly standard use of "context" (http://www.socher.org/index.php/Main/ImprovingWordRepresentationsViaGlobalContextAndMultipleWordPrototypes http://www.socher.org/index.php/Main/ImprovingWordRepresenta...). While the algorithm can be used with just the covariance of a corpus's word vectors, accounting for the distribution of a context when setting up the distance metric acts to improve performance substantially (https://i.imgur.com/Dnx9c9G.jpg https://i.imgur.com/Dnx9c9G.jpg)
- syllogism 8y ago
  I just think that titling a paper about a weighted bag-of-words approach "Context is everything" really starts the reader off on the wrong foot. In fact there's almost no context in your model, in contrast to most other sentence learning methods!
  - ContextArxiv 8y ago
    To be clear, the weighted bag-of-words is a powerful application of the contextual salience measurement, CoSal, but is certainly not the only one. (In practice, this can also be applied outside of language, for the importance of other semantic vectors in context)
- SteveJS 8y ago
  ‘Attention is all you need’, was the first thought i had when i read the title of your paper. It might be worthwhile to have a small bit explaining the distinction between attention and context as you view them. If i understand correctly you are describing how on to build a highly useful context representation. The application of that context in a neural net would be to focus the attention of the neural network on the parts of the input that are more important. From my point of view your paper here helps answer the question: what directs the attention? Apologies if all of this is blindingly obvious. I’m a fascinated amateur.
  - ContextArxiv 8y ago

whymauri 8y ago

Was this a 224n project? I'm hoping to try doing the class online on my own. Do you think the 2017 videos are still pretty in-line with the topics on the current website?

ContextArxiv 8y ago
Hi! This started off as a 224n project, but it's gone through quite a bit of work since then. The class slides are from the last class are pretty complete. I think they put a lot of time in to keep it updated, especially towards the end