3 ms·
Sparsely activated multitask model is a kind of Jeff Dean's hobby horse. It was published in 2017: https://arxiv.org/abs/1701.06538 https://arxiv.org/abs/1701.0
by sanxiyn 7y ago
Sparsely activated multitask model is a kind of Jeff Dean's hobby horse. It was published in 2017: https://arxiv.org/abs/1701.06538 https://arxiv.org/abs/1701.06538. My assessment is that it is an intriguing but ultimately failed experiment, like Geoffrey Hinton's capsule network.
- acollins1331 7y agoCapsule networks are not failed experiments! Where is this coming from? They merely haven't been applied as much as CNNs or FCNs but there are lots of papers out there where capsule networks outperform those architectures. Source: my thesis using capsule networks for semantic segmentation of aerial imagery
- XuMiao 7y agoI like the Capsule idea too. In some way, capsule network is very similar to sparse attention network. It's just the attention normalization is different. Attention is normalized on the inputs, the capsule is normalized on the output. Potentially capsule can yield much cleaner patterns, while patterns generated by attention networks can be overlapping. It's just that capsule is much harder to solve.
- jeffshek 7y agoRisky to say, considering deep learning by Hinton was also once called a failed experiment ...