3 ms·
Ironically Schmidhuber's lab also came up with a more general version of bypassing http://arxiv.org/abs/1505.00387 http://arxiv.org/abs/1505.00387 to train deep
by orbifold 8y ago
Ironically Schmidhuber's lab also came up with a more general version of bypassing http://arxiv.org/abs/1505.00387 http://arxiv.org/abs/1505.00387 to train deep networks (with similar performance), of which the popular ResNet is a special case.
- tanilama 8y agoBut highway network doesn't work as well as ResNet, and ResNet is simpler.
- jimfleming 8y agoHighway connections are actually used heavily in NLP in combination with LSTMs.
- slashcom 8y agoAnd replaced by residual connections in transformers, which are absolutely dominating LSTMs now.
- stochastic_monk 8y agoTransformer-XL uses recurrence, and most NLP SOTA is still with LSTMs. I’m not sure I’d expect attention mechanisms to fully replace recurrence.
- deepnotderp 8y agoHighway networks with scalar gating beats ResNet by a small margin.