jostmey

49 ms·

631.
Recurrent attention efficiently reaches into the deep past(arxiv.org)
1 points by jostmey 10y ago|0 comments
632.
Recurrent neural network using only deep memory(github.com)
1 points by jostmey 10y ago|0 comments
633.
Recurrent attention mechanism using moving average(arxiv.org)
1 points by jostmey 10y ago|0 comments
634.
Recurrent Neural Networks with Deep Memory Access(arxiv.org)
5 points by jostmey 10y ago|0 comments
635.
Trump says goodbye to medical science funding(sciencemag.org)
1 points by jostmey 10y ago|0 comments
636.
by jostmey 10y ago
HTML used to serve as a simple way to format a document. Now HTML is too complex for that purpose. Introduce markdown. In ten years, markdown will be too complex for formatting documents. I am big of markdown in case I didn't make that
637.
by jostmey 10y ago
I would put together a tutorial on how models are saved and restored if I understood it, which is to say it still confuses me
638.
by jostmey 10y ago
+1
639.
Simple example of machine learning in TensorFlow(github.com)
334 points by jostmey 10y ago|47 comments
640.
TensorFlow stripped naked: the bare bottom examples(github.com)
5 points by jostmey 10y ago|0 comments
641.
Deep memory augmented neural networks on natural language processing(github.com)
3 points by jostmey 10y ago|0 comments
642.
by jostmey 10y ago
I don't understand the pain or struggles with such a disease. I am sorry for those who suffer. But research in the field is becoming very exciting. Right now tons of data from patient's antibody and T-cell receptor repertoires are
643.
by jostmey 10y ago
The name "concept model" is borrowed from one of the papers I cited. The concept model is just "a(x_i,h_{i-1})". It decides what is important. You can sort of think of it like a filter or gate. When the concept model ret
644.
How recurrent attention forms short-term memories(github.com)
3 points by jostmey 10y ago|0 comments
645.
Can recurrent attention form short-term memories?(github.com)
2 points by jostmey 10y ago|0 comments
646.
by jostmey 10y ago
First comparison of the RWA model (paper: https://arxiv.org/abs/1703.01253 ) against a LSTM model on a natural language processing problem!
647.
[NLP] Recurrent attention model tested on character prediction(github.com)
1 points by jostmey 10y ago|1 comments
648.
by jostmey 10y ago
Oh no! I think you may be right. It is possible to correct the model for numerical stability in a manner that doesn't impact the output, but my approach fails at that. The correction from the previous step needs to be saved and carried
649.
by jostmey 10y ago
Yes, different models for "z" are worth exploring! You should try different forms and see what works!
650.
by jostmey 10y ago
No, I have not tried the GRU model
651.
by jostmey 10y ago
Good question! In retrospect, I should have explained that section differently. I will probably revise that section of the text (that's what I love about arxiv). We don't want the recurrent terms "h_{t-1}" to dominate th
652.
by jostmey 10y ago
I have not tested GRU cells on these problems. The GRU cells are not too different from LSTM cells--both models make heavy use of gating terms. As such, I don't expect a GRU model to do much better than a LSTM. But I cannot say for cer
653.
by jostmey 10y ago
I've only skimmed it, so what I am about to say could be incorrect. The big difference between the models is how the attention mechanism is computed. The attention mechanism is just a weighted average. The weighted average can be compu
654.
by jostmey 10y ago
Perhaps he meant to say O(N!), which would be horrendous scaling.
655.
by jostmey 10y ago
What do you mean by O(gamma function)?
656.
by jostmey 10y ago
Yes, I do save the numerator and denominator. The model I used is: h_t = f( (numerator_(t-1) + z x e^a) / (denominator_(t-1) + e^a) ) Equations (7) in the arxiv link (here it is again: https://arxiv.org/pdf/1703.01
657.
by jostmey 10y ago
EDIT: The attention mechanism is nothing more than a weighted average. The weighted average is computed as a running average by saving the numerator and denominator terms at each step. I hope the description in the paper is clear. You can f
658.
by jostmey 10y ago
The RWA model is described in equations (7) of this paper: https://arxiv.org/pdf/1703.01253.pdf I think you will see that the cost per training set is just as cheap if not cheaper than a LSTM model. On my machine, the
659.
by jostmey 10y ago
In my opinion, that's the beauty of the approach. The attention mechanism reuses the same weights for each symbol in the sequence. The attention mechanism is essentially a weighted average that filters which pieces of the sequence are
660.
by jostmey 10y ago
The RWA model requires fewer training steps than a LSTM model on the tasks I tested ( https://arxiv.org/pdf/1703.01253.pdf ). Even better, the RWA model uses over 25% fewer parameters (because it doesn't have lots o

More ›