16 ms·

Hey, author here. Happy to answer any questions or take any suggestions. Runnable code from the article: https://repl.it/@vzhou842/A-RNN-from-scratch https://r

Hey, author here. Happy to answer any questions or take any suggestions. Runnable code from the article: https://repl.it/@vzhou842/A-RNN-from-scratch https://repl.it/@vzhou842/A-RNN-from-scratch

p1esk 7y ago
Suggestion: do the same for transformer.
- lucidrains 7y ago
  http://nlp.seas.harvard.edu/2018/04/03/attention.html http://nlp.seas.harvard.edu/2018/04/03/attention.html
  - p1esk 7y ago
    Numpy only.
RyEgswuCsn 7y ago
Very nicely written post. I particularly like how you attached a link to your codebase on repl.it so anyone who is interested can tinker with the code. One thing I have been wondering for some time is whether the vanilla RNN can learn negations (i.e. 'not good' == 'bad') and valence shifts (e.g. modifier words like 'very' --- they do not carry sentiment connotations themselves, but may amplify/dampen the sentiment of the words they modify; negations like 'not' can be considered as a special-case valence shifter where it inverts the sentiment of the following word). My suspicion is that vanilla RNNs are not capable of modelling negations and valence shifters since they make inference on the sentiment of a sentence by 'adding up' the sentiment connotations of its constituent words --- negations and valence shifts, however, works more like multiplications than additions. I see you already have such examples in your dataset so I thought I'd do some experiments. I simplified your original dataset to the following: train_data = { 'good': True, 'bad': False, 'not good': False, 'not bad': True, 'very good': True, 'very bad': False, 'not very good': False, 'not very bad': True } test_data = { 'very not bad': True, 'very not good': False } While the test cases do not reflect how people actually speak, the hope is that the model should be able to apply its learning to infer their sentiment. For me, however, it would seem the training failed to converge with the default parameter settings (hidden_size=64). It would be interesting to see how other RNN architectures (e.g. LSTM, Transformers) fare with negations and valence shifters. P.S.: When calculating softmax, it is better to use the built-in functions or at least do the log-sum-exp trick to prevent under-flowing.
- vzhou842 7y ago
  Thanks for the comments! Interesting experiment - I wouldn't be surprised if better RNN architectures were more effective for this example. Appreciate the softmax tip, I'll update soon.
- mruts 7y ago
  I tried a LSTM model of stock twists and it seemed reasonably good at handling negations (single and double negatives at least)
applecrazy 7y ago
Machine translation using RNNs actually uses two of them, one serving as an encoder and the other as a decoder (the Seq2Seq architecture)
stibba 7y ago
In your post you initialize your weights a certain way but you said there are better ways. Do you have any resources for better ways?

Hey, author here. Happy to answer any questions or take any suggestions. Runnable code from the article: https://repl.it/@vzhou842/A-RNN-from-scratch https://repl.it/@vzhou842/A-RNN-from-scratch

p1esk 7y ago

Suggestion: do the same for transformer.

lucidrains 7y ago
http://nlp.seas.harvard.edu/2018/04/03/attention.html http://nlp.seas.harvard.edu/2018/04/03/attention.html
- p1esk 7y ago
  Numpy only.

RyEgswuCsn 7y ago

Very nicely written post. I particularly like how you attached a link to your codebase on repl.it so anyone who is interested can tinker with the code. One thing I have been wondering for some time is whether the vanilla RNN can learn negations (i.e. 'not good' == 'bad') and valence shifts (e.g. modifier words like 'very' --- they do not carry sentiment connotations themselves, but may amplify/dampen the sentiment of the words they modify; negations like 'not' can be considered as a special-case valence shifter where it inverts the sentiment of the following word). My suspicion is that vanilla RNNs are not capable of modelling negations and valence shifters since they make inference on the sentiment of a sentence by 'adding up' the sentiment connotations of its constituent words --- negations and valence shifts, however, works more like multiplications than additions. I see you already have such examples in your dataset so I thought I'd do some experiments. I simplified your original dataset to the following: train_data = { 'good': True, 'bad': False, 'not good': False, 'not bad': True, 'very good': True, 'very bad': False, 'not very good': False, 'not very bad': True } test_data = { 'very not bad': True, 'very not good': False } While the test cases do not reflect how people actually speak, the hope is that the model should be able to apply its learning to infer their sentiment. For me, however, it would seem the training failed to converge with the default parameter settings (hidden_size=64). It would be interesting to see how other RNN architectures (e.g. LSTM, Transformers) fare with negations and valence shifters. P.S.: When calculating softmax, it is better to use the built-in functions or at least do the log-sum-exp trick to prevent under-flowing.

vzhou842 7y ago
Thanks for the comments! Interesting experiment - I wouldn't be surprised if better RNN architectures were more effective for this example. Appreciate the softmax tip, I'll update soon.
mruts 7y ago
I tried a LSTM model of stock twists and it seemed reasonably good at handling negations (single and double negatives at least)

applecrazy 7y ago

Machine translation using RNNs actually uses two of them, one serving as an encoder and the other as a decoder (the Seq2Seq architecture)

stibba 7y ago

In your post you initialize your weights a certain way but you said there are better ways. Do you have any resources for better ways?