4 ms·
I do have a question, but first I must say that as I was reading your post I found myself nodding in agreement again and again and again. Yes, the Silicon-Vall
by cs702 9y ago
I do have a question, but first I must say that as I was reading your post I found myself nodding in agreement again and again and again. Yes, the Silicon-Valley-centric AI research community (including UK-based DeepMind and its surrounding satellites) tends to overlook or even ignore important research from elsewhere. Yes, "algorithmic creativity" and focused tinkering often prove more important than computational resources. Yes, innovation in deep learning does not require big data nor massive hardware, but "engineers are drawn to using the biggest datasets they can get, on the biggest machines they can access, like moths flitting around a bright light" (so true!). And yes, "genuine advances [have] consistently come from doings things differently, not doing things bigger." Ah-men!
With that out of the way, here's my question:
Have you guys tried or managed to achieve learning super-convergence with attention or residual attention models?
- jph00 9y agoNo I haven't tried yet. One of our students has started experimenting with language models (AWD LSTM) and getting encouraging results (about 3x speed improvement), but not with attention. I really hope that these results will help encourage more people to both see where and how super-convergence can be achieved. I suspect that we're only scratching the surface of what's possible.
- cs702 9y agoThank you for the prompt answer -- and for your intellectual generosity. I too hope these results encourage more people to see where/how super convergence can be achieved. FWIW, trying to achieve it with fully attentional models is on my "R&D things to try" list at work.