2 ms·
Read it. Worth a read, especially for those not closely following the machine learning world. The last section, focussing on having a single large sparsely ac
by londons_explore 7y ago
Read it. Worth a read, especially for those not closely following the machine learning world.
The last section, focussing on having a single large sparsely activated model which can accomplish thousands of different tasks by using a selection of internal 'experts' interests me the most.
I suspect this type of model isn't used much today simply because each company using ML only typically has a few problems to solve. If someone like Google, with far more different problems to solve, can get this type of model to work and demonstrate its effectiveness, I think it would be a big step towards solving artificial general intelligence.
Jeff Dean has a lot of respect and influence inside Google, and his ideas tend to get implemented. I'm looking forward to it!
- sanxiyn 7y agoSparsely activated multitask model is a kind of Jeff Dean's hobby horse. It was published in 2017: https://arxiv.org/abs/1701.06538 https://arxiv.org/abs/1701.06538. My assessment is that it is an intriguing but ultimately failed experiment, like Geoffrey Hinton's capsule network.
- acollins1331 7y agoCapsule networks are not failed experiments! Where is this coming from? They merely haven't been applied as much as CNNs or FCNs but there are lots of papers out there where capsule networks outperform those architectures. Source: my thesis using capsule networks for semantic segmentation of aerial imagery
- XuMiao 7y agoI like the Capsule idea too. In some way, capsule network is very similar to sparse attention network. It's just the attention normalization is different. Attention is normalized on the inputs, the capsule is normalized on the output. Potentially capsule can yield much cleaner patterns, while patterns generated by attention networks can be overlapping. It's just that capsule is much harder to solve.
- jeffshek 7y agoRisky to say, considering deep learning by Hinton was also once called a failed experiment ...