3 ms·
No I haven't tried yet. One of our students has started experimenting with language models (AWD LSTM) and getting encouraging results (about 3x speed improvemen
by jph00 8y ago
No I haven't tried yet. One of our students has started experimenting with language models (AWD LSTM) and getting encouraging results (about 3x speed improvement), but not with attention.
I really hope that these results will help encourage more people to both see where and how super-convergence can be achieved. I suspect that we're only scratching the surface of what's possible.
- cs702 8y agoThank you for the prompt answer -- and for your intellectual generosity. I too hope these results encourage more people to see where/how super convergence can be achieved. FWIW, trying to achieve it with fully attentional models is on my "R&D things to try" list at work.