4 ms·
Ask HN: Are there ML systems that can reason across multiple input modalities?
I'm working on Modular Neural Networks, and I think it may be an interesting architecture. For example, you could train smaller subnets on individual problem spaces across modalities and have them feed to a larger intermediary. Is anyone working on this?
- ArtWomb 8y agoSounds close to what Berkeley AI and Robotics lab is developing with model agnostic meta-learning. pytorch implementation here: https://github.com/tristandeleu/pytorch-maml-rl https://github.com/tristandeleu/pytorch-maml-rl And demo where robotic arm learns to navigate a simple task after watching a single video of human doing same ;) https://bair.berkeley.edu/blog/2018/06/28/daml/ https://bair.berkeley.edu/blog/2018/06/28/daml/
- hsikka 8y agoVery cool, thank you!
- screye 8y agoYes. Multimodal emotion recognition is one such area. There are indepdenent pipelines for each mode, and derived features are then fused and used for final prediction. You may also want to check out casacded models.
- hsikka 8y agoAwesome thank you!